Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a new architecture called the Block-Recurrent Transformer. Input sequences are chunked as a block, and each block is operated by a transformer layer. Each block is connected with a recurrent layer. Block id encoding similar to position encoding is introduced. The authors include ablation studies on different variants of the gate mechanism in the recurrent layer.
strengths_and_weaknesses: The paper is well-written, and the motivation is clear. The Block-Recurrent Transformer improves the efficiency-accuracy trade-off compared to the Transformer-XL, which is a strong language model baseline. Here, the efficiency is the number of parameters and runtime, and the accuracy is language modeling perplexity. It would be much better to show that the Block-Recurrent Transformer could also be effective in many applications where Transformers are successful. However, the new architecture is only tested on language modeling.
questions: Can we extend the method to a bidirectional recurrent layer?

Why is adding only one recurrent layer enough?

Can you formalize the complexity by a function of N, W, and S?

limitations: The paper does not explain how we can use the Block-Recurrent Transformer for a wide range of applications.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper is interesting. The authors propose a new RNN structure for transformer by introducing new transition states between transfomers and acheived new SOTAs. 
strengths_and_weaknesses: Strengths
1. New SOTA results by cascading many transfomers with trival modifications.
2. Tackled the bottleneck the transformer by using RNN structure for language modelling.
Weaknesses
1. This network may be more effective for very long sequences.

questions: 1. Can this network be applied to other NLP tasks ?
limitations: 1. Some failure cases can be given in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a new model which adds block-wise outer recurrency to a Transformer.
The block-wise recurrency operates on a block of tokens, i.e. the hidden state is updated not after every single token but after a block of tokens.
The hidden state is not just a feature vector but a matrix, or a vector of features.
The Transformer accesses the hidden state by cross attention on the hidden state, i.e. those vectors of features.
The Transformer additionally uses sliding-window attention like the Longformer. At the beginning of the sequence, it has access to the previous sequence and also the previous hidden state via a cache, like the Transformer-XL and Longformer. The gradient flow obviously stops at the sequence boundary.
In training, one sequence of tokens consisting of multiple blocks is given to the model.
Going over the blocks and updating the recurrent state is orthogonal to going over the tokens. For any token, the hidden state it would access is the last hidden state it has access to.
The recurrency update is similar to the layerwise structure of a Transformer. Considering one Transformer layer including the cross attention to the hidden state, mostly the same structure is used for updating the hidden state. Specifically, it uses self-attention on the hidden state itself, and cross attention to the tokenwise features of the current block. However, instead of residual connections, they use a gating mechanism here. Thus, because there is the self-attention and cross-attention (already together) and then a following MLP, there are two gates per recurrent state update. They also have some variations on removing some of the Transformer components, like the MLP, such that the recurrent state update only has one gate. In their experiments, this simpler variant actually performs better.
This described actually only one single block-recurrent Transformer layer. Their overall model is actually mostly like Longformer, i.e. Transformer-XL with sliding-window attention, and they simply replace one single layer by such a block-recurrent Transformer layer.
They also test another variant (referred to as "Feedback" in the table) where they also modify the other standard Longformer layers by additional cross attention to the last recurrent state, where the recurrent state still only comes from a single layer.

The sliding-window attention is implemented also in a blockwise fashion for better efficiency in training. I think it actually uses the same blocks both for the recurrent state and the sliding-window attention, although this is a model aspect for the recurrency, and an implementation detail for the sliding-window attention.

They perform experiments on the PG19 dataset, arXiv dataset and GitHub dataset. They reach or surpass state-of-the-art performance in all.
strengths_and_weaknesses: Strengths:

The idea to combine recurrency with self-attention in some way is not novel and has often been tried before in various ways but often only with little success. From the results, it looks like the proposed model really improves considerably due to the recurrency. The results look good.

Overall, the authors do a good job in summarizing related work.

Plan to release the code.

Weaknesses:

In the next part under questions (suggestions), I will list more things individually. Here this is just a summary of the weaknesses.

Unfortunately the model definition is unclear in many parts. My summary is what I assumed after re-reading many parts again and again and inferring what would have made sense. But this is not good. The model definition should be completely unambiguous and very clear.

The analysis on aspects of the model is a bit short and leaves many open questions.

The experimental comparison to the Memorizing Transformer looks a bit unfair. It's not clear whether the Memorizing Transformer can yield better overall perplexity.

Some more standard language modeling benchmarks like enwik8 or Wiki-103 are not used. This would have been nice as there are more results from the literature to compare to.

Code not released yet. It's also not clear whether it really will be done. I have read statements like "we plan to" so often where it was never released in the end...
Also, in this case, as there were so many things unclear, the code could have helped a lot in clarifying everything exactly up to the latest detail.
questions: Memory Networks, Neural Turing Machines are related models considering the type of hidden state.

Perceiver also somewhat similar?


> Instead of processing the sequence one token at a time, our recurrent cell operates on blocks
> of tokens; see Figure 1. Within a block, all tokens are processed in parallel. 

Processing refers to training or inference/generation? This should be made very clear. I assume this is about training because otherwise you cannot process tokens in parallel, right? At inference, even within a block, tokens are handled sequentially, I assume. This should be made clear. When it is not specifically said that it is about training, I assume inference and then this statement is wrong.

How is training implemented? A loop over the blocks?
How is inference implemented (or would be)?

I think, to make it really clear, the full definition of the model as it is used in inference (not training) should be given. Just a figure is not enough.
Recurrent cell is (mostly) clear, but the overall model is not clear. How is this recurrent cell integrated? How does the overall model look like?

> standard transformer layers ...
> We also normalize queries and keys

It would be good to clarify exactly how the overall model looks like. It seems like "standard Transformer layers" is not totally correct and there are differences, as mentioned later. Also, using sliding-window attention is also some big difference to the standard Transformer.

Is sliding-window attention and also the cache of the previous sequence also used for all the other Transformer layers?

It is not exactly clear: The self-attention in vertical direction, does it always get the last W tokens, i.e. potentially including the last block? Or is it always only within a block, so the first token would not get any context (except the recurrent state)? I assume always the last W tokens, but it's not exactly clear.

What is the effective context length in each case for each model? For the proposed recurrent case, it should be infinite, right? For the other variants, it is probably still quite high, sth like num layers * segment length but I'm not exactly sure. But this is an important aspect and should be discussed. For the Slide model variants, it should be less than the XL models, right?

> Appendix: Given a segment of N tokens (N = 4096 in our experiments), the sliding window applies a causal mask

How is this related to the blocks mentioned before?
Is this about self-attention? The self-attention is anyway only inside a block and not over the whole sequence?

In general (paper + appendix):

The terms "sequence" and "segment" also seem to be mixed up. This is confusing. The terminology should be consistent and it should be made clear what is meant.

Also the terms "window" and "block" seem to be mixed up. "blocks of length W", "window size W". I'm not exactly sure whether they refer to the same thing or not. Is there any reason it uses the same length variable W?

> Appendix: The sequence of N tokens is subdivided into blocks of size W

This is not the block concept from the block-recurrency, right? This is just an implementation detail here to make it more efficient, I assume. It's unfortunate that this causes some confusion here. This distinction should be made more clear. I think before implementation details for better efficiency are discussed, the model itself should be clearly defined.

> Appendix: The sliding window architecture has a context length of W for every token.

This statement is wrong when there are multiple layers. The context length is multiplied by the number of layers.Or if this is not the case, then something is very unclear here, and should be explained.

> current cell state at time t

What is time t? "Time" is highly confusing. This either means the state index (s in {1...S}) or the block index. I assume the block index but this is not clear here. So it means, the same gate is applied for the whole block? This should be clarified. And then also discussed. I'm not sure that this is the natural solution. Why not apply the gating per state index s?


Analysis:

Presentation in table 1 is a bit confusing. It's not explained well what the model notation means. From a first glance after not having carefully read the paper, it's not clear at all, which of those models actually correspond to the proposed block-recurrent transformer, etc. It becomes more clear after having carefully read everything, but still I think this is bad. Also, step time is unclear, is this for training or inference? PG19 perplexity on token-level is not so helpful - better would be on word-level, like it is also in table 2.

The Slide model (table 1), is this actually exactly a Longformer model? It would be better if this becomes clear.Or if the Slide model is not a Longformer, how exactly is it different? This is not clear then, and should be explained.


> Scaling up / table 2

It should be clarified how exactly the scaling is done. The dimension is increased?

Recurrent block size W, what is the effect? (Window size seems to be studied in the appendix, but not (recurrent) block size.)

I wonder about more extreme cases, like:

- S=1024, block size W=1, window size = 1, like NTM (Even though the trend in Appendix D might suggest this is bad, I think this is still interesting, whether it works at all, and how good this still is. Also, there might be a point where it improves again because it is forced to make more use of the recurrency also for short context, so it might learn that better.)
- S=1024, block size W=1, window size = 1024

(More recurrent layers? Discussed but not really systematically.)

Analysis on usefulness of recurrent state, depending on where you are in a block? Later on inside a block, the recurrent state might be less important?

Analysis on usefulness of self-attention in recurrent state update (horizontal direction). Is it actually used?

Analysis is quite short.

> We find that recurrence performs roughly on a par with the memorizing transformer, but trains considerably faster. We do not report the exact speedup ...

It would still be useful to get some impressions.

The Memorizing Transformer, I assume they have some own implementation because the numbers they report are not from the original paper. In the original paper, the best configuration used a segment/window length of 2048, and a memory size of 65k. Here they choose to use seg/win len 512 and mem size 32k. Why? This looks like they deliberately choose a suboptimal model such that their numbers look better in relation. Or if there is a reason, this should be explained, and then it would still be interesting to see the best Memorizing Transformer setting for sake of completeness.

Considering scaling the model up (Table 2), there the Memorizing Transformer would also be interesting in comparison.

After these results, I cannot tell whether the Memorizing Transformer is better or the Block-Recurrent Transformer.
 
> As with Transformer-XL, the keys and values from the last block are stored in a non-differentiable cache

This is the last block of the segment, which is for the next segment?
So this is actually different to Transformer-XL, where the whole segment is stored?
Or maybe I misunderstand sth here. This should be clarified.

> The baseline model is a 12-layer transformer

Where is this model in the table? Or is the baseline actually the XL model?

I wonder, why do the Slide models perform better than the XL models on PG19 but worse on Arxiv and Github? Because they have less context and context is more important on Arxiv or Github? This should be discussed.

limitations: 
It is mentioned that the full potential is likely not achieved yet and the current recurrent structure is maybe suboptimal.

Negative social impacts are also shortly but adequately addressed.
review_title: null

Meta-review: metareview: This paper describes a modification to the transformer architecture to use block-recurrence to more accurately model very long sequences, borrowing some ideas from the LSTM. The idea is fairly simple to implement, as it doesn't require much code over a traditional transformer, and results seem good, if not completely overwhelmingly so.

All reviewers voted to accept this paper and I agree. It's a fairly simple idea with fairly good results and adds to the body of knowledge regarding how to model very long sequences.
confidence: Certain
Decision: Accept