Venue: ICLR 2022 Track Blog

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The main focus of this blog is explaining a deep connection between attention architectures and associative memory models (aka Hopfield networks), in particular: between new scalable implicit attention architectures based on low-rank decomposition of the attention matrix, called Performers (characterized by linear time and space complexity of the attention computation), and new Hopfield associative memories.  The blog shows in the mathematically elegant way that softmax-based attention is equivalent to continuous modern Hopfield networks and derives several far-reaching consequences regarding its novel extensions. Illustrative examples demonstrate that attention modules used in the class of Transformers called Performers (applying implicit attention strategies) can be successfully applied for memories' retrieval.  Furthermore, the authors of the blog analyze the connection between Performers applying ReLU-deifned kernel and sparse associative memory mechanisms. This blog is, to the best of my knowledge, the first mathematically rigorous attempt to connect two important fields of machine learning: associative memory models and Transformers. It is very well-written, with all the concepts clearly explained so that a reader does not need to be an expert on the subject to understand its content. Experts on the subject will also find there deep mathematical ideas and inspiration for future work.
review_title: An important blog on the deep connection between implicit attention models and associative memories

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Before reading the blog post, I did not know what are Hopfield networks and this blog post taught me this concept. The main contribution of the post is to draw parallels between performers and Hopfield networks. It demonstrates how performers resemble Hopfield networks. I would love to see how these parallels yield new, interesting language models but I guess it is for future work. I'm hopeful that it will happen and I would like to see this post being published. The writing gets somewhat dense quickly, unfortunately, but I'm not sure how much it can be helped.
review_title: A curious connection that hopefully will inspire future architectures

Decision: Accept