C. D. Manning, K. Clark, J. Hewitt, U. Khandelwal, and O. Levy. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117:30046 - 30054, 2020.
M. Mitzenmacher. A Brief History of Generative Models for Power Law and Lognormal Distributions. _Internet Mathematics_, 1(2):226 - 251, 2003.
M. Newman. Power laws, pareto distributions and zipf's law. Contemporary Physics, 46 (5):323-351, 2005. doi: 10.1080/00107510500052444. URL https://doi.org/10.1080/00107510500052444.
S. V. Norman-Haignere, L. K. Long, O. Devinsky, W. Doyle, I. Irobunda, E. M. Merricks, N. A. Feldstein, G. M. McKhann, C. A. Schevon, A. Flinker, and N. Mesgarani. Multiscale temporal integration organizes hierarchical computation in human auditory cortex. Nat Hum Behav, 6 (3):455-469, 2022. doi: 10.1038/s41562-021-01261-y. URL https://doi.org/10.1038/s41562-021-01261-y. Epub 2022 Feb 10.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2020.
M. Schrimpf, I. A. Blank, G. Tuckute, C. Kauf, E. A. Hosseini, N. Kanwisher, J. B. Tenenbaum, and E. Fedorenko. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45):e2105646118, 2021.
T. O. Sharpee, C. A. Atencio, and C. E. Schreiner. Hierarchical representations in the auditory cortex. Current Opinion in Neurobiology, 21(5):761-767, 2011. ISSN 0959-4388. doi: https://doi.org/10.1016/j.conb.2011.05.027. URL https://www.sciencedirect.com/science/article/pii/S095943881100095X. Networks, circuits and computation.
C. Tallec and Y. Ollivier. Can recurrent neural networks warp time? In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=SJcKhk-Ab.
J. Tang, A. LeBel, S. Jain, and et al. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience, 26:858-866, 2023. doi: 10.1038/s41593-023-01304-9.
I. Tenney, D. Das, and E. Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1452. URL https://aclanthology.org/P19-1452.
F. Theunissen and J. P. Miller. Temporal encoding in nervous systems: A rigorous definition. Journal of Computational Neuroscience, 2(2):149-162, 1995. ISSN 1573-6873. doi: 10.1007/BF00961885. URL https://doi.org/10.1007/BF00961885.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models, 2023.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
J. Vig and Y. Belinkov. Analyzing the structure of attention in a transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63-76, Florence, Italy, Aug. 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-4808. URL https://aclanthology.org/W19-4808.