Augmenting Self-attention with Persistent Memory
Transformer networks have lead to important progress in language modeling and machine translation.These models include two consecutive modules, a feed-forward layer and a self-attention layer.The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of Transformers.Building upon this intuition, we propose a new model that solely consists of attention layers.More precisely, we augment the self-attention layers with persistent memory vectors that play a similar role as the feed-forward layer.Thanks to these vectors, we can remove the feed-forward layer without degrading the performance of a transformer.Our evaluation shows the benefits brought by our model on standard character and word level language modeling benchmarks.
A novel attention layer that combines self-attention and feed-forward sublayers of Transformer networks.This paper proposes a modification to the Transformer model by incorporating attention over "persistent" memory vectors into the self-attention layer, resulting in performance on par with existing models while using fewer parameters.