Addressing Some Limitations of Transformers with Feedback Memory
Authors: Angela Fan
Thibaut Lavril
Edouard Grave
Armand Joulin
Sainbayar Sukhbaatar
Venue:  (2021)
Keywords: Feedback
Memory
Transformers

Abstract: Transformers have been successfully applied to sequential tasks despite being feedforward networks. Unlike recurrent neural networks, Transformers use attention to capture temporal relations while processing input tokens in parallel. While this parallelization makes them computationally efficient,  it restricts the model from fully exploiting the sequential nature of the input. The representation at a given layer can only access representations from lower layers, rather than the higher level representations already available. In this work, we propose the Feedback Transformer architecture that exposes all previous representations to all future representations, meaning the lowest representation of the current timestep is formed from the highest-level abstract representation of the past. We demonstrate on a variety of benchmarks in language modeling, machine translation, and reinforcement learning that the increased representation capacity can create small, shallow models with much stronger performance than comparable Transformers.

Under review as a conference paper at ICLR 2021
ADDRESSING SOME LIMITATIONS OF TRANSFORMERS WITH FEEDBACK MEMORY
Anonymous authors Paper under double-blind review
ABSTRACT
Transformers have been successfully applied sequential, auto-regressive tasks de- spite being feedforward networks. Unlike recurrent neural networks, Transformers use attention to capture temporal relations while processing input tokens in parallel. While this parallelization makes them computationally efficient, it restricts the model from fully exploiting the sequential nature of the input. The representation at a given layer can only access representations from lower layers, rather than the higher level representations already available. In this work, we propose the Feedback Transformer architecture that exposes all previous representations to all future representations, meaning the lowest representation of the current timestep is formed from the highest-level abstract representation of the past. We demon- strate on a variety of benchmarks in language modeling, machine translation, and reinforcement learning that the increased representation capacity can create small, shallow models with much stronger performance than comparable Transformers.
INTRODUCTION
In recent years, the Transformer architecture (Vaswani et 2017) has brought large improvements to wide range of Natural Language Processing tasks such as machine translation, sentence rep- resentation (Devlin al. 2019), and summarization (Edunov et al. 2019). Transformers are also successfully used as an autoregressive model on sequential tasks such as language modeling (Dai et al.) 2019| Rae et al. 2020) and reinforcement learning (Parisotto al. 2019). Unlike more traditional recurrent architectures such RNNs and LSTMs, the Transformer architecture processes sequence in parallel in an order-invariant way. Techniques such as position embeddings (Sukhbaatar et al., Shaw et al. 2018) and attention masking are required to capture input order information. In this work, we focus on several limitations of the Transformer architecture as an autoregressive model and present straightforward solution Feedback memory. These limitations and proposed solution target sequential token prediction tasks, such as language modeling or other auto-regressive generative tasks.
The feedforward nature of Transformers makes them efficient on modern hardware, but restricts the Transformer from taking full advantage of the input's sequential property. In particular, the current hidden representation of Transformer only accesses the past representations of lower layers, even though higher level representations of the past have already been computed as an autoregressive model. At generation, the Transformer generates only one token at a time, so it could access these representations for better performance, but does not exploit these at training time due to parallelization. However, if these past higher level representations could be used at training time, they would enrich future lower level representations, enabling shallower models to have the same representation power.
Another inherent limitation of Transformers on sequential tasks is the lack of recursive computa- tion (Dehghani et al. 2018), and the number of transformations possible on the input bounded by the model depth. Such disadvantages have impact on tasks that require careful tracking of world state or modeling hicrarchical structures (Tran et al. [0188] Hahn 2020). On the other hand, while RNNs maintain an internal state for unbounded time while accumulating more computations upon it, the size of this internal state is limited by the dimension of the hidden state.
In this work, we propose novel autoregressive model. the Feedback Transformer, that makes all previous hidden representations accessible to the computation of a representation at any depth the model feeds back previous computations itself. The feedback allows the model to perform

---

Under review as a conference paper at ICLR 2021
recursive computation, building stronger representations iteratively upon previous states. To achieve this, we modify self-attention to attend to higher level representations rather than lower ones.
As shown in Figure| D the Feedback Transformer merges the hidden states from all layers into single vector for every time step and stores them in a memory. Instead of self-attention, all subsequent layers attend to this memory, which means every previously computed representation is accessible by all future layers, mediated by the memory. This allows Feedback Transformers to recursively compute and transform an input as many times as the input length, which something Transformers cannot achieve. While RNNs can perform recursive computation, the amount of information that Feedback Transformers can maintain is not limited by the number of layers.
There are computational benefits to this straightforward modification. First, it uses less memory because all the layers share single Feedback memory, thus reducing the memory size by L times, where L is the number of layers. There is also less computation because we share the key and value projections during attention computation, which increases the speed of the attention over the Feedback Memory, Further, the GPU memory usage is reduced due to the memory sharing the overall model is 2x smaller - allowing the batch size to be increased for computational efficiency. During inference, the increased batch size contributes to substantially faster decoding speeds.
In summary, our main contributions are: (1) The Feedback Transformer architecture, which com- pletely changes the way Transformer works to access available higher level representations im- mediately. (2) We show the Feedback Transformer can achieve state of the art results with smaller, shallower models that have faster decoding speed and smaller memory footprint. (3) The Feedback Transformer uses substantially less memory during training and inference time.
2
RELATED WORK
Several previous works have analyzed the limitations of Transformer architectures, such as the inability to process input sequentially (Dehghani al. 2018 or represent hierarchical structure (Tran) et al. 2018). Hahn (2020) demonstrate that Transformers cannot model structures involving bounded recursion, such as closing parentheses. Pérez et al. (2019) study Transformers in the context of Turing machines, where they must produce unbounded numbers of decoding steps. Various work in probing Transformers identified several limitations where Transformers may not have the computational capacity of recurrent architecture like an LSTM (Hahn) 2020).
From the architectural perspective, our work shares similarities with recurrent networks augmented with external shared memories (Graves al.) 2014 Joulin & Mikolov 2015 Sukhbaatar et al.) 2015). For example, the stack augmented RNN Joulin & Mikolov] (2155) adds an external memory to recurrent network to keep long term dependencies. Closer to our work, the Neural Turing Machine Graves et al. (2014) models an unconstrained memory that resembles the self-attention layer of a Transformer. Further improvements to recurrent networks, such as the Gated Feedback RNN (Chung) et al. 2015), are based on better controlling signal from different layers and extended to feedback through multiple pathways (Jin al., 2017). These works are built on recurrent networks with additional components to store long term dependencies.
Other works have studied modifications to the Transformer architecture by enriching its structure with components inspired by recurrent networks. For example, Wang el al. (2019) propose adding a local recurrent sublayer to the Transformer layer to remove the need for position embeddings in the multi-head self-attention layers. Universal Transformer (Debghani et al., (018)) share the parameters between the layers of Transformer, leading a recurrent network depth. HHo et al.) (2019) and Chen et al. (2018) augment Transformers with a second, recurrent encoder. As opposed to our work, these prior investigations do not change the computational path in a Transformer to reduce the discrepancy between the training and inference time. Closer to our work, MMerty| (2019) proposes adding a self-attention layer on top of the past outputs from an LSTM cell. However, this approach keeps the recurrent and the self-attention mechanisms decoupled, the attention mechanism recurrent. In particular, the LSTM layer of Mf (2019) still intrinsically has bottleneck corresponding the dimension of the hidden layer.
opposed to ours which makes
2

---

Under review as a conference paper at ICLR 2021
Memory
Feedback Transformer
Transformer
141
I+
Layers
weighted
H
Time
1-1
I-1
Figure 1: The Feedback Transformer merges past hidden representations from all layers into single vector and stores it in memory.
Figure 2: Difference between Feedback and Transformer. t indicates the timestep and I indicates the layer.
METHOD
In this section, we propose the Feedback Transformer, which provides capacity to build richer representations of each timestep of sequential modeling task.
3.1 TRANSFORMER ARCHITECTURES
We briefly describe the Transformer (Vaswani et al. 2017). Each layer is composed of multi- head self-attention sublayer (Attn) followed by a feedforward sublayer (FF), and each sublayer is followed by add-norm operation that combines skip-connection (He et al. 2016) and layer normalization (Lei Ba et al. 2016). The I-th layer of a Transformer processes an input sequence of vectors X .x....... x,) into sequence of vectors of the same length. First, the self-attention sublayer computes representation for each time step by taking its related input vector x2 along with its past context, (x/-r:-., m.. x}=
2 Attn(x),, (x/_.....
Within the self-attention sublayer, x, is used to form query vectors while its context is used to compute key and value vectors, forming memory of the past information. Then the feedforward sublayer processes each vector z independently. i.e., x!+ FF (z/). The Transformer layer transforms its input sequence into an output sequence X FF(Attn(X'))
In practice, block of steps {2+- (=1: x/} is computed in parallel during training, where M can be seen as the backpropagation through time (BPTT) length. This makes training Transformers efficient hardware such GPUs. However, operate sequences of unbounded length, Transformers require modifications such as caching and relative position embeddings (Dai et al.
2019 Sukhbaatar et al. 2019).
3.2
LIMITATIONS OF TRANSFORMERS
Previous work has analyzed the impact of several limitations of the Transformer architecture, such as the inability to track long sequences and process hierarchical inputs (Hahn 2020). In this work, we focus on two major limitations of Transformer architectures.
Limited Access Higher Level Representations. Layer by layer, Transformers build more abstract, high level representations of the input sequence. At each layer, the representations for the input sequence are treated parallel. As a consequence, a Transformer does not leverage the highest level representations from the past to compute the current representation, even though these highest level representations have already been computed for autoregressive models.
Maintaining a Belief State. Many sequential tasks require models to maintain internal state for two main purposes. First, internal states act as memory for recalling past inputs, where Transformers excel because their internal state of is directly accessible future steps through self-attention.
3

---

Under review as a conference paper at ICLR 2021
The second role of internal state is to act belief state that tracks the world state that is not directly observable in inputs. For example, when inputs are actions taken on a Markov Decision Process. an internal state can apply those changes to the current belief state and correctly predict the outcome. As a feedforward model, Transformer have inherent limitations in this area only fixed number of transformations can be applied to its internal states. Since both Attn and FF sublayers contain a fixed number of transformations and there are L layers of them, the total number of transformations between the input and output is limited by the depth. This means Transformers cannot maintain an internal state for long time if it has to be frequently updated.
3.3 FEEDBACK TRANSFORMER
We propose to change the Transformer architecture by using the most abstract representations from the past directly as inputs for the current timestep. This means that the model does not form its representation in parallel, but sequentially token by token. More precisely, we replace the context inputs to attention modules with memory vectors that are computed over the past, i.e.,
z = Attn (m.-r...... m;-1)). where memory vectors m, are computed by summing the representations of all layers at time step I:
m, Σ Sof Soft (o
where wf are learnable scalar parameters. Note these scalars are the only new parameters introduced by our change, with all else the same as the standard Transformer. Here I= 0 corresponds to token embeddings. The weighting of different layers by softmax output gives the model more flexibility as it can average them or select one of them.
This modification of the self-attention input adapts the computation of the Transformer from parallel to sequential, summarized in Figure Indeed, it provides the ability to formulate the representation x/_,, based on past representations from any layer I', while in standard Transformer this is only true f'r < L. This change can be viewed as exposing all previous computations to all future computations, providing better representations of the input. Such capacity would allow much shallower models to capture the same level of abstraction as deeper architecture. This has several practical advantages, as more shallow models have reduced memory footprint and increased decoding speed.
An alternative view of such an architecture modification is providing the capacity for recursive computation - outputs from sublayer can feed back to the same sublayer through the memory. The model can then maintain an internal state for unbounded time. This is a clear advantage over Transformers, in which submodule never looks at its own output. While an RNN can also repeat its computation on its internal state, its internal state has a limited capacity determined by the number of layers and their hidden dimension. In contrast, the internal state of a Feedback Transformer is its whole memory, which can grow with the input length. This allows the model to keep track of large number of things within its internal state.
While our modification requires sequential computation, significantly improve training speed by sharing the key and value projections W and W across all layers. This sharing reduces computation because need to compute key and value vectors only once instead of computing them per layer
k/ k; Wam, v W.m.
=
For the same reason, the memory footprint is smaller than standard Transformer because only one set of k,, V, needs to be stored. To be more precise, the memory requirement for processing single token is reduced from O(I x 7') to O(T), where 7. is the number of layers and T is the context size. Further, the reduced memory usage allows the batch size to be increased to recover some of the lost parallelism, which improves training speed. Thus, the Feedback Transformer is not much slower compared to the standard Transformer. Note that the same sharing of projections will not make the standard Transformer efficient because those projections are applied different representations at each layer (the key and value vectors will not the same for all layers).
Lastly, note that the sequential nature of the Feedback Transformer does not affect the performance during generation where one needs to compute one step at a time anyway. The same is true for online reinforcement learning where the input must be processed sequentially even during training.
4

---

Under review as a conference paper at ICLR 2021
Task / Model
Accuracy (%)
Copy
Char
Seq
Transformer
59.1
6.2
Feedback Transformer
76.2
23.6
0.9
Reverse
Char
Seq
Transformer
50.2
5.9
Suctesa rale
Feedback Transformer
74.8
29.2
0.7
Counting
Len 50
Len 1K
Transformer
99.6
82.4
Feedback Transformer
99.7
95.3
Transormer
Random Walk
0.5
Feedbock Transomtee
Transformer
68
20
40
80
100
Feedback Transformer
100
Memory size
Algorithmic Task
vars
vars
Figure 3: Results on the Corridor task. The Transformer degrades as the memory size de- creases, but the Feedback Transformer main- tains performance.
Transformer 4L
33.7
37.5
Transformer 8L
47.4
29.1
LSTM
82.8
32.1
Feedback Trans. 4L
99.1
92.6
Table I: Results on toy tasks. Char is char- acter accuracy, Seq is sequence accuracy.
EXPERIMENTS
We explore different sequential input tasks in natural language processing and reinforcement learning. First, we demonstrate the downsides of the standard Transformer architecture tasks where the Transformer performs poorly. We show that the Feedback Transformer is able to overcome challenges and retain long memory. Next, we highlight the strength of the Feedback architecture in building complex, high level representations even with shallow models. We demonstrate that the Feedback model can achieve significantly stronger results than Transformer models, an effect that is exaggerated as models get smaller. Finally, we compare the Feedback architecture to the Transformer architecture with other work on standard long-context language modeling tasks. In experiments large datasets, we use the shared key-value projections to improve training time. Additional experimental details and results can be found in the appendix.
4.1 LIMITATIONS OF TRANSFORMER: ILLUSTRATIVE TASKS
4.1.1 LIMITED ACCESS LONG MEMORY
First, we examine the Transformer's limited access to long memory on several simple, straightforward tasks that illustrate this. Unlike the standard Transformer, the Feedback architecture is able to remember information over many timesteps.
Walking down a Corridor. In this reinforcement learning task, each agent is placed at the start of long corridor with either blue green object. The agent must look at the object's color, walk down the corridor, and go through the corresponding colored door at the end. The only task is to remember the color and not become distracted by walking down the very long hallway. Results shown Figure 3 and show that the performance of the Transformer degrades quickly as the memory size shrinks, but the Feedback Transformer maintains strong performance at all memory sizes.
Copy and Reverse. We experiment next on two algorithmic tasks, copy and reverse (Kaiser & Sutskever 2015). We train on sequences length 40 consisting of integers 0 through 9, and test sequences of length 400. Models read the input and then either copy reverse. which requires memory over the sequence and the ability to track position, as well as generalization capability as the train and test settings are different lengths. We consider variations of copying and reversing:
5

---

Under review as a conference paper at ICLR 2021
either at the character level at the sequence level. Results shown in Table | The Feedback architecture has large improvements in accuracy, indicating improved memory and positional tracking.
Counting. Finally, we experiment on a counting task, where models have a sequence of A's in a row, and must output the corresponding quantity of the letter B. The model must count the number of the A's to output the correct number of B B's. We consider two settings: training on short sequences of lengths up to 50 and training long sequences of lengths up 1000. We show results in Table where we demonstrate the Feedback model is much better at counting over long sequences.
4.1.2 LIMITED STATE UPDATES
The complexity of the representations the Transformer is able to formulate is strictly dependent on the depth, as each layer of the Transformer allows for additional nonlinearity. The Transformer, then, only update its state the same number of times as it has layers. We demonstrate that the Feedback Transformer does not have this limitation - in tasks where the model must carefully track and update its state, the Feedback architecture is able to update its state each timestep.
Random Walk. We consider a random walk in small grid where actions are: go forward 1 step, left turn, and right turn. Given a history of actions and the agent's ss initial position, it is strictly possible calculate the current position. The task is trivial because human could write down the current location and direction and keep updating with each action. However, Transformers cannot do this because they lack a storage that can updated with each input. Its hidden state can store this information, but with each update, that information has to go up one layer.
alternative approach this task is solve it all at once given sequence of actions, which is feasible for Transformers since they can access all inputs with their attention. However, this approach is challenging because the effect of each action depends on the direction that point and whether the agent is on the edges, which itself is not known yet. This can be seen in Table [ where the Transformer struggles and only reaches 68% accuracy. contrast, the Feedback Transformer achieves 100% accuracy, which indicates the ability to track state for a long period of time. Both models are trained on 10K sequences, each containing 100 random actions and positions.
Algorithmic task. A more complex setting where tracking and updating of a state is crucial is code executions. A model needs keep track of all variable values and update them if necessary. To demonstrate this, we create a simple algorithmic task that consists of the following simple statements: assignments (e.g. x=5), increments and decrements (e.g. y-),, conditionals (e.g. if x==4: +++), and print commands (e.g. print (x)). Each task consists of 100 randomly selected statements. We consider settings with 3 and 5 different variables.
Processing of each statement in parallel will not work because conditional statements cannot be executed without knowing the current variable value, which itself can depend on another conditional. As shown Table T Transformers cannot solve this task because every time a variable increment or decrement, its value can only be found one layer up in the model, and eventually will be lost. Doubling their layers does help little, but their accuracy is far from perfect. A recurrent model like LSTM is capable of storing a variable value while updating it, thus perform well on the 3 variables version. However, its performance drop when there are more variables because it has to store all their values in a single vector. The Feedback Transformer does not have this bottleneck, and can access updated variable values from the lowest layer, it gives strong performance this task.
4.2 ADVANTAGES OF FREDBACK ARCHITECTURE
We examined two limitations of standard Transformers that we improve upon: limited memory span and limited ability to update state. In the Feedback model, we improve on these limitations and now analyze performance on practical tasks including translation and reinforcement learning.
4.2. STRONG PERFORMANCE WITH SMALL, SHALLOW MODELS
The Feedback Transformer is able to create higher level, more abstract representations with fewer layers and less capacity, layer all of the most recently created representations of previous

---

Under review as a conference paper at ICLR 2021
12-
29.0
10-
BLEU
Reward
28.0
6-
Transfornne
21.5
Feetback Transformer
Feedback Transformer
2000
3000
4000
5000
Decoding speed (wps)
Training steps
xi0
Figure 4: (left) Machine Translation on WMT14 En-De, test set BLEU and decoding speed in words-per-second for varying decoder depths. (right) Maze Navigation in Gridworld. We display average reward comparing Feedback Transformer standard Transformers.
timesteps. We demonstrate on neural machine translation that the Feedback model performs much better than Transformers at small, shallow sizes. Note that for sequence to sequence, we use Feedback Transformers only in the decoder because the encoder inputs are available simultaneously.
We evaluate the performance of the Feedback Transformer on the WMT1 4 En-De machine trans- lation benchmark of 4.5 million pairs. We follow Vaswani et al. (2017) and train WMT1 using as test. We learn 32K joint byte pair encodings
newstest 2013 as dev and newstest201 (Sennrich al. 2016), generate with beam size tuning length penalty on the dev set. We average the last 10 checkpoints and apply compound splitting and compute tokenized BLEU.
In Figure (left), display results when making the model shallower only layers removed from a Feedback Transformer decoder compared to Transformers. As the decoder becomes shallow, the gap in performance between the architectures widens. While the I-layer Transformer model can only reach 27.3, the Feedback Transformer has 28.3 BLEU. Shallow decoders are critical to fast inference reducing I-layer improves decoding speed by 4.2x, while only losing BLEU with the Feedback architecture. Such results are useful for practical applications, where the speed of producing a translation is very important. We report decoding speed tokens per second on GPU.
We further experiment with large encoder but shallow decoders. The Feedback Transformer achieves 29.0 BLEU with 12 layer encoder and 2 layer decoder. As the encoder is parallelized even during inference, the increased size of the encoder has negligible impact on decoding speed. To stabilize the training of deeper models, we use LayerDrop (Fan al. 2019)
4.2.2 LONG MEMORY TRACKS STATE
We apply Feedback to a reinforcement learning maze task that requires long memory to optimally solve because agents have limited vision. Note that in such reinforcement learning tasks, the models are trained online using A2C, so the input must processed sequentially even during training time. Thus, the non- non-parallelized nature of the Feedback Transformer is not drawback, and training Feedback Transformers is as fast as Transformers.
The goal is to navigate procedurally generated random maze where colored objects placed. One of the colors will be randomly selected as a target, and the agent has to reach it for a reward and a new target. For optimal performance, the agent must remember the maze and object locations. In addition, the agent has turn actions like the Random Walk task, which makes it necessary to keep track of its location and orientation. As shown in Figure (right))) higher average reward, compared to Transformers. Results are shown averaged over 10 trials.
(right), the Feedback Transformer converges reach
4.3 COMPARISON OTHER ARCHITECTURES
In this section, we first, we compare Feedback to recurrent architectures such as LSTM, as well as hybrid RNN-Transformer architectures, and show that the Feedback is more powerful than recurrence
7

---

Under review as a conference paper at ICLR 2021
Model
Test
1,30
Recurrent Architectures
1.25
DenseNMT Shen et al. (2018)
25.5 28.5
dev bpc
RNMT+ Chen et al. 2018
1,20
Hybrid Architectures
1.15
BiARN Hao al. 2019
28.9
SRU Lei
2017)
28.4
1.10
baseline
recurrent
top-only
Transformer Architectures
Transformer (Vaswani et al.
2017
28.4
Transformer (Ott al. 2018
29.3
Feedback Transtormer
29.5
Table 2:
Results on WMT En De compar- ing the Feedback Transformer to Recurrent architectures, hybrid Recurrent-Transfore Transformer models, and standard Transformers.
Figure 5: Comparison of different memory com- position strategies on char-PTR. The recurrent connection alone is not as effective as feedback connections from higher layer.
Next, we compare our construction the Feedback Memory with other possible compositions. Lastly, we compare other Transformer architectures on competitive benchmarks.
alone.
4.3.1 COMPARISON TO RECURRENT ARCHITECTURES
We compare the Feedback Transformer architecture recurrent architectures like LSTMs well as hybrid RNN-Transformer architectures. In Table 2 we display that the Feedback Transformer has stronger performance than the Transformer, RNN. and RNN-Transformer hybrid model. We note that recurrent models address some limitations of Transformer architectures, but the Feedback mechanism goes beyond that. By allowing all past representations to be immediately available for the computation of future representations, Feedback is stronger than Recurrence alone Recurrent models can only see representations from the previous layer (as depicted in Table
4.3.2 MEMORY COMPOSITION
We next investigate the importance the specific memory mechanism of the Feedback architecture on char- The Feedback architecture uses all layers when creating the memory, motivated by providing access the entire past of all computations, but other ways of creating the memory as possible. For example, Recurrent architectures have different memory structure. In multi-layer RNNs, each layer has recurrent connections to the same layer, but not to higher layers. This is an advantage of Feedback architectures even the highest level abstractions immediately available.
In Figure 5 we examine the construction of the Feedback memory, comparing our choice of making all computation accessible with recurrent memory that access all previous layers plus the same layer, and top-only memory that can attend only to the topmost layer. The Feedback Transformer has the best performance, closely matched by top-only memory. This indicates the importance of high level representations (see Appendix [6.4]o for further analysis on this). Note that recurrence alone enough for good performance, and thus the Feedback memory provides richer representations beyond the capacity of recurrent networks.
4.3.3 COMPARISON TO OTHER TRANSFORMER ARCHITECTURES
Finally, we examine the performance of Feedback Transformer on long context language modeling benchmarks. We use caching (Dai et al. 2019) and relative position embeddings. Mechanisms applied at inference time (Khandelwal et al. 2019 Krause et al. 2019) can further improve all models, so we do not focus on these.
We evaluate word-level language modeling Wikitext-103 (Merity et al. 2017). Our Feedback architecture takes 3.5 days to train, compared to the Transformer which takes 1.2 days. We train small Feedback model, about half the size of Transformer-XI. and find
Wikitext-103.
8

---

Under review as a conference paper at ICLR 2021
Model
Params
Test
Model
Params
Test
Best Existing (Roy et al. 2020
15,8
Best Existing |Ree 2020)
277M
0.97
Trans-XI. (Dai 2019
257M
18.3
Trans-XI. (Dai et al. 2019)
277M
0.99
Our Transformer Feedback Transformer
140M
19.9
Feedback Transformer
77M
0.96
126M
18.3
Table 4: Results on Enwiki8. We report bit-
Table 3: Results on WikiText-103. We re- port perplexity on test.
per-byte on test.
that it can match the performance of Transformer-XL, as shown in Table 3 This indicates the additional representational capacity of Feedback memory. we train a standard Transformer that is approximately the same size as Feedback Transformer, (19.9 PPL rather than 18.3). Further, mechanisms like the Routing Transformer can be added to the Feedback Transformer well. We focus starting with Transformer-XI. baseline and showing we can match the performance with a much smaller model.
find it has worse performance
Enwiki8. Finally, we test our model on character-level language modeling in Enwik wiiiii (Mahoney 201T), containing 100M unprocessed bytes from Wikipedia. We train relatively small 12-layer model, that is one third of the size of the Transformer-XL baseline. Since the task requires very long context, we use adaptive attention span (Sukhbaatar Transformer model achieves new SOTA performance of 0.96 bit-per-byte despite its small size.
2019), As shown Table 4 the Feedback
CONCLUSION
We propose a novel reformulation of the Transformer that fully exploits sequential input the increased representation power and recursive computation of the Feedback Transformer allows shallow and small models to have much stronger performance compared a Transformer of the same size. This architecture addresses two fundamental limitations of Transformers as autoregressive model - limited access to long memory and limited ability to update state. We demonstrate on variety of tasks the ad advantages of the Feedback architecture to illustrate the strong performance of this straightforward modification.
REFERENCES
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling, In ICLR, 2019.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, et al. The best of both worlds: Combining recent advances in neural machine translation. arXiv preprint arXiv: 009449, 2018.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 10509, 2019.
Junyoung Chung. Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Gated feedback recurrent neural networks. In International conference on machine learning, pp. 2067-2075, 2015.
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-oe xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv:1901.028 002660, 2019.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Eukasz Kaiser. Universal 2018.
transformens. arXiv preprint
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep
bidirectional transformers for language understanding. In NAACI-HIT (1), 2019.

---

Under review as a conference paper at ICLR 2021
Sergey Edunov, Alexei Baevski, and Michael Auli. Pre-trained language model representations for language generation. arXiv preprint arXiv:1903.09722 2019.
Angela Fan, David Grangier, and Michael Auli. Controllable abstractive summarization. arXiv 2017.
preprint
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint VrXiv:1909.115333 2019.
Edouard Grave, Armand Joulin, Moustapha Cissé, and Hervé Jégou. Efficient softmax approximation for gpus. ICMI., 2017.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401 2014.
Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8:156-171, 2020.
Jie Hao, Xing Wang, Baosong Yang. Longyue Wang, Jinfeng Zhang, and Zhaopeng Tu. Modeling recurrence for transformer. arXiv preprint arXiv: /90,000304 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Karl Moritz Hermann, Tomás Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proc. of NIPS, 2015.
Xiaojie Jin, Yunpeng Chen, Zequn Jie, Jiashi Feng, and Shuicheng Yan. Multi-path feedback recurrent neural networks for scene parsing. Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack- atack-augmented recurrent nets. In Advances in neural information processing systems, pp. 190-198, 2015.
Eukasz Kaiser and Ilya Sutskever. Neural gpus learn algorithms. arXiv preprint arXiv: 2015.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprint a7Xiv:1911.00172. 2019.
Ben Krause, Emmanuel Kahembwe, lain Murray, and Steve Renals. Dynamic evaluation of trans- former language models. arXiv preprint arXiv:1904.088: 1904.08378, 2019.
Tao Lei, Yu Zhang, Sida I Wang, Hui Dai, and Yoav Artzi Simple recurrent units for highly parallelizable recurrence. arXiv preprint arXiv: 170900250,, 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint 1:Xiv:1007.6644 2016.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Matt Mahoney. Large text compression benchmark. URL: http://www. mattnnhonyy.. net/text/text. html, 2011.
Stephen Merity. Single headed attention mn: Stop thinking with your head. arXiv preprint arXiv:1911.11423. 2019.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing Istm language models. arXiv preprint
2017.
Richard GM Morris. Spatial localization does not require the presence of local cues. Learning and motivation, 12(2):239-20) 260, 1981.
10

---

Under review as a conference paper at ICLR 2021
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. arXiv preprint arXiv: 1rXiv11806.00187,, 2018.
Emilio Parisotto, H. Song, Jack W. Rae, Razvan Pascanu, Çaglar Gülçehre, Siddhant M. Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, A. Clark, Seb Noury, M. Botvinick, N. Heess, and Raia Hadsell. Stabilizing transformers for reinforcement learning. ArXiv, 0oss/990.066644 2019.
Jorge Pérez, Javier Marinković, and Pablo Barceló. On the turing completeness of modern neural network architectures. arXiv preprint vrXiv:/901.0342) 2019.
Jack W. Rac, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Com- pressive transformers for long-range sequence modelling. In International Conference Learning Representations, 2020. URL http://pppe//ppeeeeeeeeeeeeeeeeo httppeeeeeeeeeeeeeeeeoooouunnnee
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. arXiv preprint arXiv:2003.05997, 2020.
Abigail See, Peter J Liu, and Christopher Manning. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv: (0Xv::170444366 2017.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL (1), 2016.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In NAACL-IILT (2), 2018.
Yanyao Shen, Xu Tan, Di He, Tao Qin, and Tie- Yan Liu. Dense information flow for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1294-1303, 2018.
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In NIPS, 2015.
Sainbayar Sukhbuatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span transformers. In ACL, 2019.
Ke Tran, Arianna Bisazza, and Christof Monz. The importance of being recurrent for modeling hierarchical structure. arXiv preprint arXiv: 10..:v/8003.338.. 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
Zhiwei Wang, Yao Ma, Zitao Liu, and Jiliang Tang. R-transformer: Recurrent neural network enhanced transformer. arXiv preprint arXiv: 190., 00,00572, 2019.

---

Under review as a conference paper at ICLR 2021
ADDITIONAL RESULTS
6.1 REINFORCEMENT LEARNING
Maze Navigation Easy. We experiment with a slightly different version of the Maze Navigation task. Instead of an agent with forward, turn-left and turnrright actions, the agent has orientation and there are only 4 movement actions corresponding to 4 cardinal directions. This makes navigation easier because the agent do not need to keep track of its orientation. Further, it is much easier compute relative locations given a history of actions. This might explain why standard Transformers are far behind Feedback Transformers in performance compare to LSTMs, which performs much worse. See Section 7.2
shown in Figure 6
(left). We also for more implementation details.
Water Maze. We modify the Morris Water Maze task (Morris 1981 to make it more challenging. The maze is defined by a goal position and mapping of cell ID these remain fixed within an episode but change between episodes. The agent receives as an observation the cell IDs of its current location and the target cell. When the agent finds the target, it receives +1 reward and is randomly teleported. During the same episode, if the agent reaches previously seen cell, it needs to remember how reached the target from there to go back. Results are shown averaged over 10 trials (the reward is reported averaged over the last 500 episodes for each trial). As shown in Figure 6 (right), the Feedback Transformer converges to higher average reward.
20
15-
Reward
Tranforres
Heward
1.0
Peedbaok Transformrr
ISTM
51
Transforrer
Fredback Transformet
0.3
0.5
1.0
15
2.0
Training steps
x10"
Training steps
x10
Figure 6: Averaged cumulative reward during training on (left) Maze Navigation Easy and (right)
Water Maze tasks.
6.2 IWSLT DI-EN
We additionally evaluate the Feedback Transformer on IWSLT De-En, small machine translation dataset. We train small Transformer model with 6 layers. For generation, we use beam size without checkpoint averaging. Model quality is evaluated using tokenized BLEU. Results are shown in Figure 7 (left) and show that for shallower models, the Feedback Transformer has better performance than the standard Transformer.
6.3 SUMMARIZATION ON CNN-DAILYMAIL
We evaluate on the CNNi multi-sentence summarization benchmark of 280K news articles Hermann et al. (2015). modeling the first 400 words of the article See et al.) (2017). We evaluate using ROUGE Lin (2004). and use 3-gram blocking and tune length Fan et al. (2017). Figure (right) displays the performance of the Feedback Transformer as the decoder layers are reduced, making the model shallower only. For all model depths, the Feedback architecture maintains a consistent improvement in ROUGE compared the standard Transformer. Compared to sentence- level tasks such as translation, this summarization benchmark requires multi-sentence generation, and the increased capacity of the Feedback architecture is beneficial.
12

---

Under review as a conference paper at ICLR 2021
35.25
37.5
Tist HOUGEE-
Test 13.33
$1.70
31.50
36.5
34.00
Pransforict
Transforer
33.75
Poollack Traneformer
33.5
x
Foetbock Tranfforme
4000
6000
8000
10000
12000
18000
per
Decoder Depth:
Figure 7:
Results on (left) the IWSLT De-En dataset, and (right) Summarization on CNN-Dailymail, test set ROUGE-L for varying decoder depths.
1.30
1.28
Single
Dev. (bpe)
1.26
Transformer
Feedback
1.24
Average all
1/22
1.20
Layer
Figure 8: Ablation results on char-PTB: instead of weighted sum of all layers Feedback memory, only a single layer is used as memory for all layers. We also include a setting where the average of all layers is used.
6.4 ABLATION STUDIES ON LANGUAGE MODELS
We investigate which layer of model has the best representation to be used as a Feedback memory. In Feedback Transformers, a weighted sum of all layers is used as the memory. and feeds to all layers. An alternative approach is manually select one of the layers as the memory and let all layers attend to it. In Figure(8 we explore this approach, using the same 6-layer char-PTB models Section 4.3.2 (top-only memory there corresponds using the last 6th layer as memory). We can see that representations from higher layers work better as memory, confirming our assumption of the importance of higher level representations. Simply averaging all layers together works reasonably well as well. Interestingly, when all layer attend to the first layer output, it works as good as the standard Transformer. The weighted sum approach matches the best performance because it can adopt to select any of the layers.
Here we study how different techniques affect the model performance on results shown in
The Pre-normalization combined with higher learning rates helps the performance, particularly for the standard Transformer.
Increasing the context size with adaptive span further improves the performance for both models.
The technique of increasing the BPTT length during training for efficiency does not affect the final performance.
The gap between two model is consistent along those variations.
Next, we examine the effect of the model depth on performance on char-Prt and WikiText- This time, keep the total number of parameters constant and only vary the number of layers to
13

---

yes Under review as a conference paper at ICLR 2021
Model
Pre-norm Adapt. higher LR span BPTT ppl
Increase
dev
Transformer
22.9
Transformer
no no
yes
22.9
Transformer
yes
21.0
Transformer yes yes
no
20.6
Feedback no no
no
19.7
Feedback
no no
yes
19.9
Feedback yes no yes
19.6
Feedback yes yes yes
19.0
Table 5: Ablation on WikiText 103 of various modeing choices. Results are shown without finetuning.
2.0
Peedbock Trandforen
Predbock Trazedoraet
Dev. (bpc)
Dev. (pp3)
1.6-
2
Model depth
Model Depth
Figure 9: The performance on (left) char- PTB and (right) Wikitext- 103 as function of the model depth. The number of parameters is kept constant by increasing the width.
isolate the effect of depth. This achieved by proportionally increasing the head dimension and the ReLU layer size when we decrease the number of layers. The results in Figure demonstrate that for the standard Transformer improves as the depth increase. In contrast, the Feedback architecture is much robust reduced depth, even achieving the best performance char with only two layers.
ADDITIONAL IMPLEMENTATION DETAILS
7.1 RANDOM WALK TASK DETAILS
We provide additional details for the random walk toy task we explore. The agent starts at fixed position of 8 grid. Available actions are move one step forward, 2) turn left and 3) turn right. At every time step. the agent randomly picks on of the three actions and executes it. An action would be ignored if can't be executed like going out of the grid. After 100 actions, the agent is reset back to the initial position.
The input to the model a sequence of actions taken by the agent, and a special symbol there was reset. The output is sequence of location symbols corresponding the agent's location after each action. We generate 10k training episodes, totalling IM tokens.
We use the same setup our language modeling experiments, except now the model predicts separate output tokens rather than a next token. We concatenate all the episodes and feed them to the model single sequence. The training is done with the negative- likelihoo loss. See Table g for the hyperparameters used the experiment. The attention span is set to 100, so that the models can attend to all the information they needs to solve the task.
14

---

Under review as a conference paper at ICLR 2021
visien Exrge
go01 VI5.0 range
Figure 10: (left) Maze Navigation task and (right) Water Maze task.
print
print print
print if S
print
print
print
Y +- if
print ------ x print
print Y + print i5
Y
: if < x x 1f 2 2
print y print print x print 2 print print
y x 11
print -
print 2
print if
print priat
END
Table 6: An example program from the algorithmic task with 3 variables.
7.2 MAZE NAVIGATION DETAILS
We generate random x mazes using Kruskal's algorithm. Dead ends are eliminated by randomly removing one of the blocks surrounding them. We randomly place 8 target objects with different colors as shown in Figure 10 (left). The agent is given randomly selected color as a target. the agent manages to reach the correct target, it gets reward of +1 and new target color is sampled. An episode ends after 200 steps. The observation includes the x 3 area around the agent and target color.
We train 2-layer Transformers with hidden size 256 and heads. We set the BPTT to 100 and the batch size to 1024. The reward discount rate is 0.99. The attention span is 200 the agent keep an entire episode in memory. All agents were trained using A2C with Adam with a learning rate of 0.0003 and entropy cost of 0.0005. For the easy version of the task, we RMSprop with a batch size of 128 and a learning rate of 0.0003. The RMSProp epsilon regularization parameter is set to 0.01 The LSTM model is 3-layer LSTM with hidden size of 256.
7.3 WATER MAZE DETAILS
The water maze task we designed is depicted visually in Figure I0 (right). The grid size is 15 x 15. To help exploration, the agent can see if the goal is within 3 x 3 area around it. An episode ends after 200 steps. We train for 500M steps .2.5M episodes). We use 2-layer Transformers with hidden size of 64 and 1 head. The attention span is 200 so the agent can put entire episode in memory.
All agents where trained using A2C with RMSprop with entropy cost 0.0001, RMSProp epsilon regularisation parameter of 0.01, batch size of 64, and BPTT 200. Feedback Transformer and Transformer baseline were trained with a learning rate of 0.0003. LSTT model a 2-layer LSTM with hidden size of 64. For LSTM model we used a learning rate of 0.0004.
7.4 ALGORITHMIC TASK DETAILS
In this task, each program consists of 100 simple statements that should be sequentially executed. The available statement types are:
1. Initialization. Assign an initial value to a variable like x=3. variable can only be initialized once in each program.
15

---

Under review as a conference paper at ICLR 2021
Hyperparameter
Summarization
WMT En-De
IWSLT De-En
Encoder Layers
6
6
Decoder Layers
6
6
6
FFN Size
2048
4096
1024
Attention Heads
8
16
4
Dropout
0.3
0.3
0.3
Hidden Size
512 0.0005
1024
512
Learning Rate
0.001
0.0005
Table Hyperparamers for sequence to sequence experiments.
Hyperparameter
Random Walk
char-PTB
Enwik8
WikiText-10S small
WikiText-103
Algorithmic
large
Layers
4
6
12
8
Hidden size (d)
256
384
512
512
1024
FF size
4d
4d
8d
8d
4d
Head count (h)
4
4
8
8
Head dim
d/h
d/h
2d/h
2d/h
d/h
Attention span
100
512
8192*
512
512, 2048*
Dropout rate
0.2
0.5
0.5
0.1
0.3
Embed. dropout
0.1
0.2
BPTT len (M)
128
128
256
256
Batch size (B)
512
2048
1024
512
512
Learning rate
0.0001
0.0015
0.0015
0.0007
0.0007
Gradient clip
0.1
1.0
0.1
0.1
0.1
LR warm-up steps
1k
1k
8k
8k
8k
Parameters
3.2M
10.7M
77M
44M
139M
Table Hyperparamers for language modeling experiments. Here indicates the adaptive span.
2. Increment and decrement. Increment or decrement a variable value by 1, like x++ or
y--
3. Print. Output the value certain variable like print (y). Only this statement requires model to make prediction.
4. Conditional. Execute the nested statement only variable has a certain value, c.g.,
if x==4: y-- Note that conditional and print statements cannot be nested.
A program is generated by randomly choosing a statement one after another, but with the following conditions: variable must be initialized before being used, and a variable value have between and 10. The training data contains 10k such programs concatenated with special separator keyword. generate two version the data with and different variables in them. An example program shown in Table 6
We used the same hyperparameters the random walk task show in Table g
7.5 MACHINE TRANSLATION AND SUMMARIZATION
We detail the hyperparameters Table Summarization experiments are done with the Transformer base architecture size and WMT En-De experiments done with the Transformer big architecture size. As IWSLT De-En is smaller dataset, we use a smaller model. For all sequence to sequence experiments, only the decoder is modified to have the Feedback Transformer architecture.
7.6 LANGUAGE MODELING
In the language modeling experiments, we added several improvements on top of the original Transformer Vaswani et al. (201)) better adapt unbounded sequences:
16

---

Under review as a conference paper at ICLR 2021
Hidden representation caching Dai et al. (2019): Since the input the model is an bounded sequence and the model needs to process it in small blocks, hidden representations from previous blocks are kept in cache so that any token in the current block will the same context length regardless of its position in the block.
Relative position embedding [Sha et al. (2018): Relative position embeddings allow same way regardless of its absolute position
each token in a block to be processed in the block. We found that adding shared embeddings to key vectors at every layer to be effective.
Adaptive attention span Sukhbaatar et al. 2019) Language modeling requires model have a very long attention span, which is computationally expensive. The adaptive span mechanism allows each attention head to learn different attention spans for efficiency. Pre-normalization Child et al. (2019): We observed that pre-normalization makes train- ing more stable for Transformers, which allowed us to use larger batch sizes for better parallelization.
Dropouts are applied to attention and ReLU activations. In mikiText-103 models, additional dropouts are added to the embedding layer output and the last sublayer output.
In Table we present the hyperparameter values used for our experiments. We use the same hyperparameters for both Transformers and Feedback Transformers, and optimize them with Adam. The final performances are obtained by finetuning the models with 10x smaller learning rate.
Details on the char-PTB experiments We trained the models for 15k updates (or earlier the validation loss stops decreasing), and funetined them for k steps. We varied the depth of the models while keeping the number of parameters constant. This is achieved by changing the FF size and the head dimension inverse proportionally to the depth.
Details on the enwik8 experiments We used an adaptive span limited to 8192 tokens with a loss of 0.0000005. The training done for 100k updates and another 10k steps is used for finetuning. The warming up BPTT length is used for speeding up the training, where the BPTT length is decreased to 64 for the first half of the training.
Details for Training WikiText- 103 We employed the adaptive input Baevski & Auli (2019) and the adaptive softmax Grave et al. (2017) techniques for reducing the number of parameters within word embeddings. The models are trained for 200k steps and the finetuned for additional 10k steps.
While most of the models have a fixed attention span 512, the best performance is achieved by extending the attention span 2048 with adaptive span loss 0.00001.
After training our models, we noticed that our tokenization method differed from others by omitting end-of-line (EOL) symbols. Since our dictionary already contained the EOL token, we were able finetune our trained models on the data with EOL tokens, rather than training them from scratch. This change alone brought about lppl improvement.
17


Reviews:
[{"review_id": "ezLeNQuQQ05", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "> Summary: This paper proposes some changes to the classical Transformer architecture to address its major limitations, such as limited access to higher-level representations. It specifically introduces recurrence to the Transformer architecture by feeding the activations of all previous time steps to a later time step (in the form of self-attention). Empirical results on language modeling and small-scale RL tasks seem to suggest the usefulness of doing do.\n\n--------------------\n\nPost-rebuttal thoughts:\n\nSee the comment block below.\n\n--------------------\n\nOverall:\n\nI found this paper interesting and relatively easy to follow. The idea is simple, and seems useful, although I do find some arguments handwavy and not quite convincing (e.g., the \"maintaining a belief state\" one). It is unclear to me how exactly the efficiency compares, though the authors did report the # of days on WikiText-103 (see my detailed question below). I overall think that this could be a good architectural improvement on the condition that the authors provide more details.\n\nPros:\n\n1. Simple idea and the flow of the paper is easy to follow. \n2. An extensive set of experiments to verify both the usefulness of the Feedback Transformer and the limitations that the authors hypothesize to be true for transformers.\n\nCons:\n\n1. The introduction of sequential-ness to Transformer is good but obviously would slow things down especially as the sequence gets longer. The authors reported on this very briefly, but I think it is an important enough aspect to warrant more analysis.\n2. Lack of certain ablative settings in the experiments (which is unavoidable in a certain sense, given that the paper proposes various changes to the architecture).\n\n----------------------------------------------\n\nAdditional comments and questions:\n\n1. The core of the hypothesis on the value of high-level representation feedback is the autoregressiveness, is this correct? As the paper claims, typical Transformers are restricted from \"taking full advantage of the input's sequential property\" because they can't access the higher-level representations of previous time steps. I have two questions in this respect, and wonder if the authors have verified this (if not, I think you probably should (?)):\n    1) Would you expect a \"feedback LSTM\" to work better than an LSTM as well? In other words, an LSTM that when computing $h_t^{(l)}$ of time $t$ at layer $l$, uses $h_{<t}^{(L)}$ just like in the Feedback Transformer?\n    2) In pixel sequences like CIFAR-10 density modeling (e.g., see Sparse Transformer by Child et al. 2019), where the autoregressiveness is not rather obvious (e.g., you can do column-based or row-based, or even Hilbert curves), does Feedback Transformer still help? If so, then it means higher-level representation is not exactly a \"temporal\" phenomenon, because there's nothing in pixels that's temporal...\n\n2. Regarding the sharing of keys and values in a Feedback Transformer--- is the motivation for this just to speed up the architecture? How well does Feedback Transformer perform without this sharing, and how slow would it be?\n\n3. I'm confused about the \"maintaining a belief state\" paragraph. The authors claim that Transformers are limited by \"only a fixed number of transformations can be applied to its internal states\". But aren't those internal states already aggregated by lower levels? Why might more transformations be better? Can't one simply increase the number of layers of a Transformer? I also don't see the logical connection between this claim and the end of this paragraph: \"This means Transformers cannot maintain an internal state for a long time if it has to be frequently updated\". Can the authors clarify on this part?\n\n4. In cases especially like NMT, where decoders are trained in parallel (because at training time, the decoder is trained just like an LM) and used for inference in sequence (at test time, it generates tokens one by one), wouldn't it make more sense to pre-train a classical Transformer (with no feedback memory) and then directly use, or probably with slight fine-tuning, the feedback version of it at inference? Is this possible?\n\n5. One of the most important thing that I believe the current version is missing is a more comprehensive analysis of the efficiency, which seems to be an important drawback (if any). I noticed that the authors claim that key-value sharing compensated for the loss on parallelism--- but by how much exactly? Specifically, I'd appreciate if the authors can provide an analysis of at least some of the following:\n    1) How many GPUs (and what sort of GPU) did you use to train your models, e.g., for WikiText-103 and for char-PTB? Did you use the same setting for the classical Transformer? (The 1.2 vs. 3.5 days on WikiText-103 is still a large gap, almost 3x slower...)\n    2) How does the efficiency of the training (not in terms of days of training, but ms per batch) scale as you use longer and longer sequences? I'm asking because I noticed in Table 7 that these sequence lengths are still pretty short; e.g., I believe SOTA char-PTB uses length > 256 and WikiText-103 uses length > 1024 at inference. Does Feedback Transformer further improve when you use longer sequences?\n    3) If the \"high-level representation\" of Transformers is indeed a major limitation, does a deeper (but still the same # of parameters, so probably smaller hidden dimensionality) Transformer perform better, because it can have \"more updates\" to its internal state? Or maybe a weight-sharing Transformer? How does the efficiency vs. performance compare in these cases?\n \n6. Did you train all of the Feedback Transformers from scratch (i.e., `train_step`=0), or did you warm-up/pretrain the models?\n\n7. Have you ever tried non-toy-scale RL tasks? I think this proposed architecture would be very useful in these very sequential settings (e.g., in robotics, where the data stream actually has temporal dimensions), and these large-scale RL tasks (e.g., Doom FPS game, etc.) could make the paper even stronger.\n\n8. I'd suggest expanding Tables 3 and 4--- there are plenty of prior works that evaluated on these two datasets and it'd be worth it to cite them to compare. In addition, for Table 3, does increasing parameters further improve the performance? It is impressive that you can achieve the same level of result as Transformer-XL with only half of the parameters, which seems to suggest there's still room for improvement?\n\n9. Section 4.3: \"we first\" ---> \"first\"\n\n---------------------------------\n\nOverall, I think this paper presents relatively solid results, but there are some key ablative settings, efficiency details & analysis, and large-scale RL results that are missing. I'm putting a 5 on this paper for now, but I look forward to the authors' response and am happy to adjust my score positively once my questions are further clarified.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DYdPERtnqpy", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe main topic of this paper is modification and enhancement of Transformers originally proposed in Vaswani’17.\nAs we all know, Transformers are now used as a core technology in a wide range of research communities such as natural language, vision, and speech. \nMany researchers aim to improve such core technology since it might provide a high impact to the communities. Thus, tons of papers propose a wide variety of modifications for Transformers in recent years.\nIn this perspective, this paper can be categorized as one of such papers.\nTherefore, the audience and influence of this paper could be significantly broader.\n\n\nThis paper focuses on the limitations of the Transformer architecture as an autoregressive model.\nThis paper points out two drawbacks.\nOne is the original Transformers do not handle the higher layer representations of the past states that have already been computed as a viewpoint of the autoregressive model.\nThe other is that the model depth bounds the number of transformations possible on the input.\nThis paper then proposes a method called “Feedback transformers“ that can effectively overcome such drawbacks by explicitly incorporating all the past state representations, including higher-layer representations, when using transformers as an auto-regressive sequential generator.\n\nThe top-level concept is rather straightforward and can be easily noticeable by many researchers in some sense, nothing innovative or unique.\nFrom this perspective, it seems that this paper is incremental study rather than an innovative one.\nHowever, the idea of injecting auto-regressive computation is the somewhat totally counter concept for the original Transformers since Transformers try to significantly reduce the computational cost on the specialized computational environment like GPUs by ignoring the auto-regressive nature.\nAlthough the proposed method does not obey the original concept of Transformers, the findings from this paper's experiments are very impressive.\nI think the findings in this paper can help many researchers as a new insight into the community.\nIn my feeling, this is basically an insightful paper.\n\n\nThe following are the questions/concerns of this paper. \n\n1, The implicit explanation for the target situation\nThe discussion in this paper only focuses on the auto-regressive generation or sequentially predicting tokens one-by-one.\nHowever, the Transformer architectures are also popular to be used in many other situations, such as masked language models like BERT.\nUnfortunately, the current version does not explicitly distinguish how the Transformer architectures are used for.\nSome readers might misunderstand that the discussion of this paper could include such a situation.\nEven if not so much, the authors should clearly state the target of their claims and discussions at the very beginning of this paper.\n\n\n2, Calculation cost\nThe calculation cost is one of the main discussion points in the proposed method.\nHowever, there is no clear experimental results shown about this part.\nThis is a clear disadvantage of this paper.\n\n\n3, Intuition of additional parameters w^l appeared in Eq. 1. (Ablation study)\nThe proposed method suggests using the weighted sum of all the hidden vectors in all the layers.\nHowever, there is no reasonable explanation about intuition for this introduction.\nWe have many other possible choices.\nThis paper does not discuss such a possible variant at all.\nTo better understand the proposed method, the authors should provide a certain amount of ablation studies.\n\nFor example, what would happen if we used all the hidden vectors independently, not just weighted summing ups them.\nMoreover, what would happen if just average them (without weighting factor), etc.\n\n\n\n\n\n4,\nI am not totally convinced that MT and LM's results are really significant improvements from the original Transformers or the comparative previous methods.\nRegarding the WMT experiments, the one-point BLEU difference can be often observed by just changing random seeds in the identical method.\nThe authors should somehow provide additional evidence that the proposed method significantly differs from the baseline methods.\n\n\nI am willing to change my score if I got reasonable answers for all the questions and concerns written in the above reviews.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "D37yg-s2MVS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis paper modifies transformers with feedback memory. Specifically, for each timestep, it merges hidden representations of all layers into a high-level single vector and stores it in memory. For the current timestep, it attends past memory vectors. The authors claim that in this way, low layers of the current timestep can utilize high-level representations of past timesteps. The authors show that the proposed models with shallow layers can achieve stronger performance than comparable transformers. However, it seems that the models need a much longer time to train.\n\n\n### Strengths\n* With feedback memory, the modified can speedup decoding (autoregressive generation) as shown in Figure 4.\n* Since the proposed model can directly utilize previous high-level representations, it just needs a small size and shallow layers to achieve comparable performance as shown in Table 3 and Table 4.\n\n### Weaknesses and Questions\n* Training time and inference speed are important for such practical models. It is better to complement these to Table 1/2/3/4. It seems that the proposed needs to take a much longer time to train as the authors mentioned it in a sentence on page 8. The authors can give more results and discussions so that future users can know whether to choose transformers with feedback memory according to their situations.\n* (optional) In Table 3/4, how about feedback transformer that keeps the same number of layers and similar parameters as Trans-XL. Transformers usually can achieve better performance when the number of layers increases.  It is just an optional discussion as feedback transformers seem to need much time to train and the rebuttal time is limited.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8UHbH-Fq4IK", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors try to identify several problems in the Transformer model and modify the model architecture.\n\nMajor:\n\n1. The authors argue that for any position k and layer l, the standard Transformer can only access previous positions (<k) and lower layers (<l). Instead, the authors propose to leverage >l layers for <k positions. First, apparently, compared to standard Transformer, the training of this model (teacher forcing setting) is much slower as the computation of any positions requires the whole forward outputs of all previous positions (Standard Transformer run positions together (in parallel)). The author should demonstrate the training efficiency of the proposed model.\n\n2. As far as I know, for other architectures, such as deep RNN(LSTM) or convseq, as the same as Transformer, the computations of any position k and layer l only access to previous positions (<k) and lower layers (<l). Therefore, I think the authors should discuss the problems in those model architectures and test their proposal in more settings. \n\n3. The strong requirement of a belief state in a model architecture is not convincing evidence to me. Or I can also view the ffn outputs at any (position, layer) as a virtual belief state. I also have difficulty understanding the arguments in the 'Maintaining a Belief State' paragraph for concrete reasons. I hope the author could pay more attention to describing the motivation behind.\n\n4. The authors argue that the proposed model is memory efficient than the standard Transformer, but this seems to be not a fair comparison. They share the key and values across different layers as in Albert, Universal Transformer, and DEQ, but fail to connect to these previous works. This trick cannot be viewed as a contribution to the paper.\n\n5. Regarding experiments and comparisons.\na. If you need to highlight long memory tasks (table 1),  please include the Transformer-XL, sparse Transformer into the comparison, which are very typical baselines in this scenario.\nb. For the experiment in 4.2.1, the single decoding layer setting, what is the difference between your model and Transformer? In such a setting, both model access to all previous states. Where does the benefit come from? \nc. In section 4.2.1, in the main body, you write the performance of your model (12-layer encoder + 12-layer decoder) is 29.0. But in Table 2, you write the performance of your model is 29.5, but the baseline models are only (6-layer encoder + 6-layer decoder)\nd. If you need to highlight the fast decoding, please include the non-autoregressive models and linear transformers as baselines.\n\n\nMinor:\n\n1. In the 2nd paragraph, the 'feedforward nature' is not clear.\n\nOverall:\n\nThe general problem that the authors want to solve is not very clear or very well-motivated. The experimental comparisons and baselines are not adequate. There is much room for better paper writing and presentation.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on the limitations of the transformer architecture as an autoregressive model. The paper is relatively easy to follow. Though most reviewers find the paper interesting, the idea is not very novel. The introduction of sequential-ness to Transformer is good, though it also slow things down especially as the sequence gets longer.

An extensive set of experiments are performed, though the results are not entirely convincing. The authors are encouraged to add more ablative experiments, efficiency analysis, and large-scale results.
Decision: Reject