Before revision: We show that the transformer attention mechanism is the update rule of a modern Hopfield network with continuous states . This new Hopfield network can store exponentially (with the dimension  ) many patterns, converges with one update, and has exponentially small retrieval errors.
After revision: We show that the transformer attention mechanism is the update rule of a modern Hopfield network with continuous states and a corresponding update rule. The new Hopfield network can store exponentially (with the dimension ) many patterns, converges with one update, and has exponentially small retrieval errors.