[25] Yongyi Yang, Zengfeng Huang, and David Wipf. Transformers from an Optimization Perspective, May 2022.
[26] Martin J. Wainwright and Michael I. Jordan. Graphical Models, Exponential Families, and Variational Inference. Foundations and Trends® in Machine Learning, 1(1-2):1-305, November 2008. ISSN 1935-8237, 1935-8245. doi: 10.1561/2200000001.
[27] Yee Teh, David Newman, and Max Welling. A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation. In Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006.
[28] Alan L Yuille and Anand Rangarajan. The Concave-Convex Procedure (CCCP). In Advances in Neural Information Processing Systems, volume 14. MIT Press, 2001.
[29] Rajesh P. N. Rao and Dana H. Ballard. Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2(1):79-87, January 1999. ISSN 1546-1726. doi: 10.1038/4580.
[30] Karl Friston and Stefan Kiebel. Predictive coding under the free-energy principle. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1521):1211-1221, May 2009. ISSN 0962-8436. doi: 10.1098/rstb.2008.0300.
[31] Christopher L. Buckley, Chang Sub Kim, Simon McGregor, and Anil K. Seth. The free energy principle for action and perception: A mathematical review. Journal of Mathematical Psychology, 81:55-79, December 2017. ISSN 0022-2496. doi: 10.1016/j.jmp.2017.09.004.
[32] Beren Millidge, Yuhang Song, Tommaso Salvatori, Thomas Lukasiewicz, and Rafal Bogacz. A Theoretical Framework for Inference and Learning in Predictive Coding Networks, August 2022.
[33] John H. Reynolds and David J. Heeger. The Normalization Model of Attention. Neuron, 61(2):168-185, January 2009. ISSN 08966273. doi: 10.1016/j.neuron.2009.01.002.
[34] Matteo Carandini and David J. Heeger. Normalization as a canonical neural computation. Nature Reviews Neuroscience, 13(1):51-62, January 2012. ISSN 1471-0048. doi: 10.1038/nrn3136.
[35] Grace W. Lindsay. Attention in Psychology, Neuroscience, and Machine Learning. Frontiers in Computational Neuroscience, 14, 2020. ISSN 1662-5188.
[36] Andy Clark. The many faces of precision (Replies to commentaries on "Whatever next? Neural prediction, situated agents, and the future of cognitive science"). Frontiers in Psychology, 4, 2013. ISSN 1664-1078.
[37] Harriet Feldman and Karl Friston. Attention, Uncertainty, and Free-Energy. Frontiers in Human Neuroscience, 4, 2010. ISSN 1662-5161.
[38] M. Berk Mirza, Rick A. Adams, Karl Friston, and Thomas Parr. Introducing a Bayesian model of selective attention based on active inference. Scientific Reports, 9(1):13915, September 2019. ISSN 2045-2322. doi: 10.1038/s41598-019-50138-8.
[39] O. Zobay. Mean field inference for the Dirichlet process mixture model. Electronic Journal of Statistics, 3 (none):507-545, January 2009. ISSN 1935-7524, 1935-7524. doi: 10.1214/08-EJS339.
[40] Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin. Augmenting Self-attention with Persistent Memory, July 2019.
[41] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer Feed-Forward Layers Are Key-Value Memories, September 2021.
[42] Stanisław Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual Connections Encourage Iterative Inference, March 2018.
[43] Joe Marino, Yisong Yue, and Stephan Mandt. Iterative Amortized Inference. In Proceedings of the 35th International Conference on Machine Learning, pages 3403-3412. PMLR, July 2018.
[44] Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting Latent Predictions from Transformers with the Tuned Lens, March 2023.
[45] Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. Generalized Variational Inference: Three arguments for deriving new Posteriors, December 2019.
[46] Emmanuelle Tognoli and J. A. Scott Kelso. The Metastable Brain. Neuron, 81(1):35-48, January 2014. ISSN 0896-6273. doi: 10.1016/j.neuron.2013.12.022.