Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper extends the previous line of work on interpreting linear transformer as fast weight programmers (FWP), and introduce recurrent variants of FWP. 
main_review: Originality:
- The idea is not very novel in my opinion. It is a straightforward extension of the previous work, by changing either the slow network or the fast network from a feedforward neural network to a recurrent neural network.

Clarity:
- The paper is written very nicely. When introducing the background material, it provides a unifying view of the subject. It helps me understand FWP much better and also the connection between FWP and linear transformer.


Quality:
- As the paper introduces a simple variant of the existing work, I would mainly hope that the empirical results to be strong enough to deserve a publication. However, the current results do not convince me at all. Mainly,
   1. The results are not very strong. E.g., in Table 2, LSTM seems to beat all the variants proposed. 
   2. The comparisons of existing works are missing. In the wikiText dataset, it missed the results from [3], which attained better performance than any of the reported variants. For CodeExec task, it missed the results from [4], which also did better than the reported results. For atari, it missed the baselines such as [1] or [2]. Note that these references are already cited in the paper (some tasks are taken from those). So it seems the author knows these results but intentionally choose to not include these as baselines? 
- Over the past several years, many of the impressive results on realistic benchmarks were demonstrated without the use of recurrence. Hence this raises a question -- when is recurrence really needed? Part of the reason we prefer non-recurrence architecture is because of its easiness for scalability, and fast training speed, as one can parallelize computation across time. I would like to see a the necessity of recurrence on some practically relevant task.

Question:
- In the paper line 235-247 the authors discussed the computational efficiency of this model. But I am a bit confused by the measurement there. The computational complexity of vanilla transformer computes the embeddings for each token in the layer in one big matrix multiplication which can be highly parallelized in modern accelerators, but the recurrence would not allow computational complexity any lower than O(N), where N is the sequence length. But somehow the number of tokens per second is roughly the same. I wonder what's the reason here?
  

Significance:
- The work is a straightforward extension of the previous work. The results are not very convincing. Hence I wouldn't think the work is significant at its current stage.

Summary:
- I would encourage the authors to improve the empirical results to make it more convincing. In particular, I think the author should demonstrate on a practically relevant task that recurrence is necessary. 


[1] Mott et.al.: Towards interpretable reinforcement learning using attention augmented agents.

[2] Parisotto et.al.: Stabilizing Transformers for reinforcement learning.

[3] Peng et. al.: Random feature attention.

[4] Fan et.al: Addressing some limitations of Transformers with feedback memory
limitations_and_societal_impact: NA
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes modifications to the linear transformer paradigm that adds recurrent dependencies to past transformer embeddings. The resulting "Recurrent Fast Weight Programmer" model proposed is shown to outperform transformers on multiple sequence learning tasks. Furthermore, the proposed method remains competitive with RNNs in tasks where RNN architectures are still dominant over transformers, implying a best of both worlds feature in various sequence learning settings. 
main_review: The text presents several potential variations of the linear transformer formulation which incorporate various properties such as recurrence and differing types of interactions between fast and slow weights. But the motivation for such architectural modifications is unclear from the presentation of the paper. Why is the DeltaNet formulation necessary as compared to the Linear Transformer? What aspect of the DeltaNet formulation improves model capacity?

Overall, the connections between linear transformers and Fast Weight Programmers are interesting because it may give rise to explanations of where the benefits and limitations exist among both styles of architectures. Furthermore, it may help explain what deviations could be expected in performance from the original/dominant Transformer. These differences are seen in the experiments of this work. But this work has little to no hypotheses and therefore no subsequent analysis of how these modifications improve/don't improve performance. For instance, how is the \beta gating function behaving when given various difficulty levels of the code execution and sequence list tasks?

Post Rebuttal: I have gone over the author responses and discussions from other reviewers and have decided to increase my score. My main concern is still the lack of exploration into the function of the additional modifications proposed in the RDN model. Without knowing what is systematically different between the proposed model after learning a task and other models, a never-ending paper stream of variations of linear transformers that do not necessarily perform any better than the original transformer formulation in the real world can result. But I agree with the author's responses that this may be too much to ask for in a paper submission.
limitations_and_societal_impact: Suggestions for improvement:

-Introduction and specifically lines 36-45 begin discussion of the architectural modifications that are hard to understand without any formulation. (lines 42-43 seem contradictory)

-Explain what are the scaling differences between Delta Delta Net and the size of the backpropagation graph if another meta-level of slow to fast weights are added?

-Lines 186-190 need a better explanation for the formulations of the slow weights. More capacity and additional memory are too vague. Why formulate the fast weights with 1-D states and weights as 2-D states?

-Since some of these models are not limited by the token span as much as vanilla transformers, why not try the models that can support it much longer sequence spans?

-Lines 216-218 like earlier lines are too vague, an intuition for why the slow network requires additional input (and also untested). It seems like the lack of strong reasoning behind these architecture modifications makes conclusions drawn from the results ambiguous.

-The surprisingly good performance of the LSTM on the code execution task is also not explained or analyzed. The lack of analysis here is unfortunate because this deviates from what is expected and is worth understanding.

-The falloff in LSTM performance at higher depth (depth 15) is also an interesting observation that should be expanded upon.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a new series of transformer architectures with novel linear attention mechanisms, all based on the formalism of fast weight programmers (FWPs). The fast weight programming perspective views attention as a "slow" network producing internal weight updates to a "fast" weight hyper-network, where an update to the fast weight matrix is predicted and accumulated at each time step in the input sequence. Therefore the fast weight network changes it's parameters during processing of a single sequence, while the slow network is updated using standard gradient descent on the loss function across sequences. Using this perspective of fast weights, the paper designs several novel attention replacements: (1) recurrent architectures for the fast weight network, (2) deep feedforward networks, and (3) a nested recurrent architecture, the Recurrent Delta Net (RDN), where both the fast and slow network include a recurrence. Results on language modeling, code execution and reinforcement learning domains reveal the benefits of approaching linearized attention under this formalism, with the RDN in particular showing advantages over even standard softmax-attention transformers. 
main_review: The paper is well-written and clear to follow. The formalism of fast weight programmers makes understanding the attention variants intuitive, and the design choices of the network architectures presented using this perspective follow naturally.  The experimental results seem to support the viability of the architectures, with similar or better results to both standard transformer and linear transformer baselines. Adding recurrence to the fast weight mechanism seems to be especially important on the code execution and list ops domains, which largely close the gap with the LSTM baseline.

While the paper presents a formalism from which a few novel architectures are designed, there is a downside that the results do not always provide a clear advantage to one particular architecture over another. For example, Delta LSTM does best out of the presented architectures at language modeling, but is twice as slow as a regular transformer and still underperforms it at test perplexity. As another example, at code execution it is either the RDN or the Delta RNN which are best performing. This somewhat limits the significance of the results, since a practitioner can't immediately choose one architecture and might need to ablate over many of them. It could make the narrative of the experiments section cleaner if the authors chose to focus on a particular architecture, and demonstrated a clearer performance improvement under some chosen metrics (wall-clock training time, inference time, etc.).

It would also be interesting to see how these models perform at larger scale. I would be very interested to see the scaling properties of the recurrent attention architectures, although I understand that running these experiments could be expensive. 

Additionally, is there an intuition on why any of the memory architectures on Atari would be particularly helpful? The results seem to support that, but a more thorough understanding of why exactly would be informative. Is this caused by increased parameter count, ease-of-optimization, or is the memory actually being used effectively in the policy?

In conclusion, I believe the paper has presented an interesting perspective on recurrent attention and, despite some issues with the focus of the results, I vote for acceptance. 
limitations_and_societal_impact: The authors have discussed the limitations of their model. The model also does not seem to have any significant potential for negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies linear attention variants within transformers from the perspective of fast weight programmers from the 90s.  In particular, it explores new more sophisticated variants that have recurrence on both the slow (fixed after training) and fast networks (generated) as opposed to previous variants such as Delta Net that uses one-layer feedforward networks for both.  The evaluation shows that recurrence leads to improved results compared to both standard and linear transformers.
main_review: The exploration of more sophisticated fast weight programmers for linear attention is an interesting and valuable direction since it enhances the understanding of the linear attention variants. Even though the connection to fast weight programmers itself is not new the proposed variants have sufficient novelty and increase the expressiveness in maintaining the internal state when used for attending to long sequences. 

The technical exposition was quite clear and well-positioned with respect to prior work. The motivation for using recurrence becomes apparent given the simplicity of the networks used in previous variants. The study reveals that using a Delta Net with simple recurrence leads to best results and more sophisticated recurrence based on LSTMs removes the efficiency benefits in linear transformers. One interesting insight from this finding is that it is not possible to increase the expressiveness of linear attention beyond a certain point without being slower compared to the original transformer; albeit the model would still be scalable while the original transformer is not. 
 
The evaluation is thorough, contains controlled comparisons, and uses a configuration similar configurations to prior work. The results support the claims in the paper and shed light on the strengths of different variants. I also found the details provided in the supplementary quite helpful. Perhaps one limitation is that the evaluation does not include any non-autoregressive tasks. 

Questions:
- Is the difference in LSTM performance on the code execution task from prior work due to the data creation process or some other factor?  
 
- Can the proposed attention variants be applied to non-autoregressive tasks? Recent efficient variants are applicable to such settings, including linear transformers, Performer/RFA.   

- The statement that fast weight variants maintain the standard one-dimensional state and two-dimensional fast weight states in Sec 3.2 was a bit confusing. Where do these dimension sizes come from? 

limitations_and_societal_impact: The paper discusses the limitations of the proposed models and explores their speed-quality tradeoffs in the evaluation section. I would suggest that the authors discuss the applicability of the proposed variants on non-autoregressive tasks and whether the proposed methods can handle them (and why not if they can't) so that the scope of this paper is clear to the readers. There is also a section dedicated to the potential negative societal impact.
review_title: null

Meta-review: This paper uses the connection between linear attention transforms and fast weight programmers to introduce a novel transformer architecture, recurrent fast weight programmers, which includes recurrence in the fast weights. 

The authors demonstrate competitive performance on three distinct benchmarks and provide interesting conceptual insights into the functioning of their new architecture.



The discussion focussed on the role of the recurrence (“Why is it needed?”) and novel baselines / ablations. In particular, the authors provided new experimental evidence showing that the recurrence indeed helps, even when compared to a feedforward NN with the same number of parameters. 

The clarifications and new evidence provided made one reviewer increase their score, while all other reviews remained unchanged. 



Given the positive reviews, the discussion and the overall contributions of the paper, I recommend this paper to be accepted.

Decision: Accept (Poster)