Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: 
The main topic of this paper is modification and enhancement of Transformers originally proposed in Vaswani’17.
As we all know, Transformers are now used as a core technology in a wide range of research communities such as natural language, vision, and speech. 
Many researchers aim to improve such core technology since it might provide a high impact to the communities. Thus, tons of papers propose a wide variety of modifications for Transformers in recent years.
In this perspective, this paper can be categorized as one of such papers.
Therefore, the audience and influence of this paper could be significantly broader.


This paper focuses on the limitations of the Transformer architecture as an autoregressive model.
This paper points out two drawbacks.
One is the original Transformers do not handle the higher layer representations of the past states that have already been computed as a viewpoint of the autoregressive model.
The other is that the model depth bounds the number of transformations possible on the input.
This paper then proposes a method called “Feedback transformers“ that can effectively overcome such drawbacks by explicitly incorporating all the past state representations, including higher-layer representations, when using transformers as an auto-regressive sequential generator.

The top-level concept is rather straightforward and can be easily noticeable by many researchers in some sense, nothing innovative or unique.
From this perspective, it seems that this paper is incremental study rather than an innovative one.
However, the idea of injecting auto-regressive computation is the somewhat totally counter concept for the original Transformers since Transformers try to significantly reduce the computational cost on the specialized computational environment like GPUs by ignoring the auto-regressive nature.
Although the proposed method does not obey the original concept of Transformers, the findings from this paper's experiments are very impressive.
I think the findings in this paper can help many researchers as a new insight into the community.
In my feeling, this is basically an insightful paper.


The following are the questions/concerns of this paper. 

1, The implicit explanation for the target situation
The discussion in this paper only focuses on the auto-regressive generation or sequentially predicting tokens one-by-one.
However, the Transformer architectures are also popular to be used in many other situations, such as masked language models like BERT.
Unfortunately, the current version does not explicitly distinguish how the Transformer architectures are used for.
Some readers might misunderstand that the discussion of this paper could include such a situation.
Even if not so much, the authors should clearly state the target of their claims and discussions at the very beginning of this paper.


2, Calculation cost
The calculation cost is one of the main discussion points in the proposed method.
However, there is no clear experimental results shown about this part.
This is a clear disadvantage of this paper.


3, Intuition of additional parameters w^l appeared in Eq. 1. (Ablation study)
The proposed method suggests using the weighted sum of all the hidden vectors in all the layers.
However, there is no reasonable explanation about intuition for this introduction.
We have many other possible choices.
This paper does not discuss such a possible variant at all.
To better understand the proposed method, the authors should provide a certain amount of ablation studies.

For example, what would happen if we used all the hidden vectors independently, not just weighted summing ups them.
Moreover, what would happen if just average them (without weighting factor), etc.





4,
I am not totally convinced that MT and LM's results are really significant improvements from the original Transformers or the comparative previous methods.
Regarding the WMT experiments, the one-point BLEU difference can be often observed by just changing random seeds in the identical method.
The authors should somehow provide additional evidence that the proposed method significantly differs from the baseline methods.


I am willing to change my score if I got reasonable answers for all the questions and concerns written in the above reviews.


review_title: Interesting findings

final_score: rating: 6: Marginally above acceptance threshold
review_content: > Summary: This paper proposes some changes to the classical Transformer architecture to address its major limitations, such as limited access to higher-level representations. It specifically introduces recurrence to the Transformer architecture by feeding the activations of all previous time steps to a later time step (in the form of self-attention). Empirical results on language modeling and small-scale RL tasks seem to suggest the usefulness of doing do.

--------------------

Post-rebuttal thoughts:

See the comment block below.

--------------------

Overall:

I found this paper interesting and relatively easy to follow. The idea is simple, and seems useful, although I do find some arguments handwavy and not quite convincing (e.g., the "maintaining a belief state" one). It is unclear to me how exactly the efficiency compares, though the authors did report the # of days on WikiText-103 (see my detailed question below). I overall think that this could be a good architectural improvement on the condition that the authors provide more details.

Pros:

1. Simple idea and the flow of the paper is easy to follow. 
2. An extensive set of experiments to verify both the usefulness of the Feedback Transformer and the limitations that the authors hypothesize to be true for transformers.

Cons:

1. The introduction of sequential-ness to Transformer is good but obviously would slow things down especially as the sequence gets longer. The authors reported on this very briefly, but I think it is an important enough aspect to warrant more analysis.
2. Lack of certain ablative settings in the experiments (which is unavoidable in a certain sense, given that the paper proposes various changes to the architecture).

----------------------------------------------

Additional comments and questions:

1. The core of the hypothesis on the value of high-level representation feedback is the autoregressiveness, is this correct? As the paper claims, typical Transformers are restricted from "taking full advantage of the input's sequential property" because they can't access the higher-level representations of previous time steps. I have two questions in this respect, and wonder if the authors have verified this (if not, I think you probably should (?)):
    1) Would you expect a "feedback LSTM" to work better than an LSTM as well? In other words, an LSTM that when computing $h_t^{(l)}$ of time $t$ at layer $l$, uses $h_{<t}^{(L)}$ just like in the Feedback Transformer?
    2) In pixel sequences like CIFAR-10 density modeling (e.g., see Sparse Transformer by Child et al. 2019), where the autoregressiveness is not rather obvious (e.g., you can do column-based or row-based, or even Hilbert curves), does Feedback Transformer still help? If so, then it means higher-level representation is not exactly a "temporal" phenomenon, because there's nothing in pixels that's temporal...

2. Regarding the sharing of keys and values in a Feedback Transformer--- is the motivation for this just to speed up the architecture? How well does Feedback Transformer perform without this sharing, and how slow would it be?

3. I'm confused about the "maintaining a belief state" paragraph. The authors claim that Transformers are limited by "only a fixed number of transformations can be applied to its internal states". But aren't those internal states already aggregated by lower levels? Why might more transformations be better? Can't one simply increase the number of layers of a Transformer? I also don't see the logical connection between this claim and the end of this paragraph: "This means Transformers cannot maintain an internal state for a long time if it has to be frequently updated". Can the authors clarify on this part?

4. In cases especially like NMT, where decoders are trained in parallel (because at training time, the decoder is trained just like an LM) and used for inference in sequence (at test time, it generates tokens one by one), wouldn't it make more sense to pre-train a classical Transformer (with no feedback memory) and then directly use, or probably with slight fine-tuning, the feedback version of it at inference? Is this possible?

5. One of the most important thing that I believe the current version is missing is a more comprehensive analysis of the efficiency, which seems to be an important drawback (if any). I noticed that the authors claim that key-value sharing compensated for the loss on parallelism--- but by how much exactly? Specifically, I'd appreciate if the authors can provide an analysis of at least some of the following:
    1) How many GPUs (and what sort of GPU) did you use to train your models, e.g., for WikiText-103 and for char-PTB? Did you use the same setting for the classical Transformer? (The 1.2 vs. 3.5 days on WikiText-103 is still a large gap, almost 3x slower...)
    2) How does the efficiency of the training (not in terms of days of training, but ms per batch) scale as you use longer and longer sequences? I'm asking because I noticed in Table 7 that these sequence lengths are still pretty short; e.g., I believe SOTA char-PTB uses length > 256 and WikiText-103 uses length > 1024 at inference. Does Feedback Transformer further improve when you use longer sequences?
    3) If the "high-level representation" of Transformers is indeed a major limitation, does a deeper (but still the same # of parameters, so probably smaller hidden dimensionality) Transformer perform better, because it can have "more updates" to its internal state? Or maybe a weight-sharing Transformer? How does the efficiency vs. performance compare in these cases?
 
6. Did you train all of the Feedback Transformers from scratch (i.e., `train_step`=0), or did you warm-up/pretrain the models?

7. Have you ever tried non-toy-scale RL tasks? I think this proposed architecture would be very useful in these very sequential settings (e.g., in robotics, where the data stream actually has temporal dimensions), and these large-scale RL tasks (e.g., Doom FPS game, etc.) could make the paper even stronger.

8. I'd suggest expanding Tables 3 and 4--- there are plenty of prior works that evaluated on these two datasets and it'd be worth it to cite them to compare. In addition, for Table 3, does increasing parameters further improve the performance? It is impressive that you can achieve the same level of result as Transformer-XL with only half of the parameters, which seems to suggest there's still room for improvement?

9. Section 4.3: "we first" ---> "first"

---------------------------------

Overall, I think this paper presents relatively solid results, but there are some key ablative settings, efficiency details & analysis, and large-scale RL results that are missing. I'm putting a 5 on this paper for now, but I look forward to the authors' response and am happy to adjust my score positively once my questions are further clarified.


review_title: Good empirical results on LM and RL; lack of more detailed efficiency & large-scale RL validations

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
This paper modifies transformers with feedback memory. Specifically, for each timestep, it merges hidden representations of all layers into a high-level single vector and stores it in memory. For the current timestep, it attends past memory vectors. The authors claim that in this way, low layers of the current timestep can utilize high-level representations of past timesteps. The authors show that the proposed models with shallow layers can achieve stronger performance than comparable transformers. However, it seems that the models need a much longer time to train.


### Strengths
* With feedback memory, the modified can speedup decoding (autoregressive generation) as shown in Figure 4.
* Since the proposed model can directly utilize previous high-level representations, it just needs a small size and shallow layers to achieve comparable performance as shown in Table 3 and Table 4.

### Weaknesses and Questions
* Training time and inference speed are important for such practical models. It is better to complement these to Table 1/2/3/4. It seems that the proposed needs to take a much longer time to train as the authors mentioned it in a sentence on page 8. The authors can give more results and discussions so that future users can know whether to choose transformers with feedback memory according to their situations.
* (optional) In Table 3/4, how about feedback transformer that keeps the same number of layers and similar parameters as Trans-XL. Transformers usually can achieve better performance when the number of layers increases.  It is just an optional discussion as feedback transformers seem to need much time to train and the rebuttal time is limited.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors try to identify several problems in the Transformer model and modify the model architecture.

Major:

1. The authors argue that for any position k and layer l, the standard Transformer can only access previous positions (<k) and lower layers (<l). Instead, the authors propose to leverage >l layers for <k positions. First, apparently, compared to standard Transformer, the training of this model (teacher forcing setting) is much slower as the computation of any positions requires the whole forward outputs of all previous positions (Standard Transformer run positions together (in parallel)). The author should demonstrate the training efficiency of the proposed model.

2. As far as I know, for other architectures, such as deep RNN(LSTM) or convseq, as the same as Transformer, the computations of any position k and layer l only access to previous positions (<k) and lower layers (<l). Therefore, I think the authors should discuss the problems in those model architectures and test their proposal in more settings. 

3. The strong requirement of a belief state in a model architecture is not convincing evidence to me. Or I can also view the ffn outputs at any (position, layer) as a virtual belief state. I also have difficulty understanding the arguments in the 'Maintaining a Belief State' paragraph for concrete reasons. I hope the author could pay more attention to describing the motivation behind.

4. The authors argue that the proposed model is memory efficient than the standard Transformer, but this seems to be not a fair comparison. They share the key and values across different layers as in Albert, Universal Transformer, and DEQ, but fail to connect to these previous works. This trick cannot be viewed as a contribution to the paper.

5. Regarding experiments and comparisons.
a. If you need to highlight long memory tasks (table 1),  please include the Transformer-XL, sparse Transformer into the comparison, which are very typical baselines in this scenario.
b. For the experiment in 4.2.1, the single decoding layer setting, what is the difference between your model and Transformer? In such a setting, both model access to all previous states. Where does the benefit come from? 
c. In section 4.2.1, in the main body, you write the performance of your model (12-layer encoder + 12-layer decoder) is 29.0. But in Table 2, you write the performance of your model is 29.5, but the baseline models are only (6-layer encoder + 6-layer decoder)
d. If you need to highlight the fast decoding, please include the non-autoregressive models and linear transformers as baselines.


Minor:

1. In the 2nd paragraph, the 'feedforward nature' is not clear.

Overall:

The general problem that the authors want to solve is not very clear or very well-motivated. The experimental comparisons and baselines are not adequate. There is much room for better paper writing and presentation.
review_title: Review summary

Meta-review: This paper focuses on the limitations of the transformer architecture as an autoregressive model. The paper is relatively easy to follow. Though most reviewers find the paper interesting, the idea is not very novel. The introduction of sequential-ness to Transformer is good, though it also slow things down especially as the sequence gets longer.

An extensive set of experiments are performed, though the results are not entirely convincing. The authors are encouraged to add more ablative experiments, efficiency analysis, and large-scale results.
Decision: Reject