Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes Temporal Latent Bottleneck (TLB) to help transformer-based models benefit from the compression of sequential inputs while not having to (significantly) forego their expressivity. This is done by dividing the computation stream into a slow stream that is recurrent in nature, and a fast stream that allows for attention mechanism between tokens. One of the salient abilities of the proposed architecture is the mutual interaction between both the streams, through a top-down connection between the recurrent and transformer modules (allowing for cross attention), and a bottom-up connection (allowing the recurrent module to take as inputs the final layer of transformer module). Experiments are performed across Vision, Language, and RL domains to show that the improvements brought by the proposed architecture are consistent across domains.
strengths_and_weaknesses: ## Strengths
1. The paper does a good job of explaining details of the newly proposed architecture. 
2. Comparisons with baseline methods over multiple domains are convincing and the performance improvements are consistent.

## Weakness
1. The underlying hypothesis to motivate the introduction of a recurrent stream was that it may benefit generalization since it may contain lesser irrelevant details. I find (a) this hypothesis unconvincing, and (b) no evidence in the paper to support the same.
2. The paper motivates the use of a transformer architecture by arguing how past work has found recurrent architectures to be incapable of remembering long-range dependencies. In light of the same, it would make a lot more sense if the recurrent stream of the input happened in chunks of K-sized tokens, and the transformer part acted upon the outputs of each layer. This would allow the recurrent stream to do what it is good at (compressing not-so-long range dependencies), and the attention part to attend to all pairs over compressed tokens from anywhere in the input. The current architecture choice appears misplaced, especially based on the motivation that the authors use to position the proposed architecture.
3. One of the claims of the work is about the ability of the architecture to perform well in resource-constrained scenarios. I would have liked to see an analysis of how the model fairs as more pertaining data becomes available (like in typical language models). Do the advantages of such an architecture still hold up? Will such an architecture be of relevance to the deep learning community at large? Even in case of resource-constrained settings, I find an important analysis missing about the performance change with sequence length. Since the attention module is limited by its attention window to K chunks, this may have an unwanted impact as the sequence length changes. In this regard, text classification on datasets of long sentence length (like IMDb) and short sentence lengths (like SST) would be an interesting point of comparison.

## Other Comments
1. The use of vspace is very evident. Please take it into consideration.
2. In Figure 3, it appears that the baseline method has not yet converged. Did you train for longer?
3. In Figure 3, should the accuracy for Multitask be comparable to a single task?
4. The computation cost seems incorrect. There would be an additional term for (T/K) for each of the K-sized modules (line 90). This would of course still be less than O(t^2)
5. Can you use TLB with Swin Transformer?
6. Why are you not using a bidirectional recurrent layer. Bidirectional RNNs and LSTMs are known to outperform traditional ones. It appears that except for the sequential tasks (4.2), using a bidirectional recurrent layer should improve performance.
7. What is the time taken for training and inference as opposed to baseline methods?
questions: Mentioned above
limitations: Memory and Time complexity should be talked about. While the computation complexity does decrease, the time complexity is expected to rise as each individual attention block waits for the previous block, and the corresponding RNN module to complete and provide input for the cross attention layer. I would also like to see the difference in training and inference time. Since GPU parallelization is used pervasively, reporting computation complexity is an unfair metric.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a modification to the transformer architecture called Temporal Latent Bottleneck (TLB). The motivation is to provide a more compressed representation by forcing the model to pass chunks of the input stream through a bottleneck. Whereas a standard transformer may attend to any part of the sequence when encoding the current element, the TLB transformer can only attend to elements in the current chunk; the information beyond this chunk is received only through the bottleneck vector.

The authors maintain that their modified transformer architecture can be applied in any setting where standard transformers were previously used. They provide a series of demonstrations spanning image perception, sequential decision making, and language modeling. In each of these domains, the proposed model outperforms alternative transformer baselines.
strengths_and_weaknesses: The writing throughout the paper is clear and technically sound. Moreover the quality of the experiments is high. The originality I will rate as low to medium: although the authors describe a grand motivation for the architectural ideas & principles of their proposed model, the proposal is ultimately only a modification to the existing transformer, and there is little support provided to justify the claims made in their introduction (see specific comments below). Nevertheless, the work has a high potential impact and significance given that transformers have become engrained in almost every area of ML/AI, and the authors have demonstrated the effectiveness of their approach in several different transformer use cases.

My biggest suggestion to improve this paper is the following: I would suggest scaling back the tone and broader framing of the introduction to focus on providing a new transformer architecture that outperforms the default transformer and that is applicable to a number of different settings. At the moment, there’s a major emphasis placed on architectural ideas/principles which are not adequately studied. These include, but are not limited to:
1.  Bottlenecked/compressed representation for improved sample complexity. Throughout the intro the authors talk about learning a more compressed representation, with the motivation being that a more compressed model should have better sample complexity. However, there is very little discussion or exploration of sample complexity in the provided experiments. In the CIFAR experiments, for example, it’s unclear how many training examples are being used, or whether there is a certain sample complexity at which alternative models begin to outperform the proposed. The idea of sample complexity is touched on very briefly in the Baby AI experiment, although “number of timesteps” is not a perfect analogue to sample size (there may be repeated examples over any period of time, and a model with strong sample complexity could have slow convergence for other reasons). 
2.  Top-down conditioning. The authors suggest that, by conditioning the chunk encoder F on the latent bottleneck state, the model is required to reason in a “top-down” manner, where information from a higher level of abstraction informs the encoding of low-level raw inputs. They justify this claim with an ablation study (Table 2) where they show that, when they remove the top-down connection in TLB, performance deteriorates. Although I think this ablation is interesting, I don’t think it provides support that the model is reasoning “top-down” as the authors say. Given the constraint of the chunking architecture, the ablated model has no form of information about the past whatsoever, at any level of abstraction. It seems obvious that the model would underperform in this configuration, and I don’t feel that it evidences the use of higher level abstraction.
questions: My major suggestion is outlined above. In addition I have a smaller suggestion about the enwiki8 experiment (section 4.3). This experiment is described in a total of 2 sentences with very little detail about the task, the choice of baseline, or the evaluation criteria. There is also little to no reflection provided about the results. I suggest either removing this experiment or expanding it to a complete description.
limitations: The limitations are adequately addressed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work addresses the aspect of memory (state) and its expressiveness in deep learning models for sequences. Recurrent neural networks (such as LSTMs and GRUs) on one end compress the entire history into a single state representation vector while on the other end the memory state in Transformers is a continuously appended list of (key, value) vectors from all previous timesteps and therefore has a higher fidelity representation of the entire past. They seek to find an optimal balance between compression and expressiveness of these 2 contrasting forms of memories. Their proposed solution strategy is a kind of best-of-both- worlds which has 2 processing streams where the “slow” stream is instantiated as a recurrent module which updates a set of state vectors (once every chunk size steps of the input) using a cross attention mechanism and the “fast” stream which updates its internal representation at every input timestep. This “fast” stream is instantiated as a Transformer Encoder except with the addition of cross-attention blocks as well in addition to the standard self-attention. They empirically evaluate their proposed architecture against several standard Transformer models across various domains such as visual representation learning (ViT), offline reinforcement learning (Decision Transformer) and language modeling (Transformer-XL).   

strengths_and_weaknesses: Strengths:

- The high-level idea behind the proposed TLB architecture is interesting and the authors have taken a lot of effort to show empirical results of their proposed architectural modification to the standard Transformer on a wide range of domains such as offline reinforcement learning, visual representation learning, language modeling etc. and their benefits. 

- The proposed architecture shows very good empirical gains on Atari in the offline setting over the Decision Transformer, and to a lesser extent on classification tasks on CIFAR-10 and CIFAR-100 length (i.e. image resolution) generalization, language modeling on enwiki8 dataset and learning long-range dependencies (ListOps and Text Classification) tasks. The proposed TLB architecture provides benefits both in terms of computational efficiency and performance. 


Weaknesses:

- Please cite related work on fast and slow learning mechanisms such as the DCEM model [3] which uses a dual stream processing to allow for fast associative memories and slow episodic memories in object-word association/RL tasks, Hierarchical Transformer [5] which biases the Transformer to process long sequences in a more efficient manner by using inductive biases to model the sequence hierarchically, Ordered Memory [6] which learns to represent a hierarchical structure in RNNs.

- The novelty of the overall architecture seems marginal and that it is more an amalgamation of several components introduced by other work such as Perceiver [1], Slot Attention [2], DCEM [3], HCAM [4], Hierarchical Transformer [5] which have similar motivations behind their architectural improvements to the standard Transformer model. Regardless, I think there could be significant value in this specific architecture. But the current experimental design and analysis of results does not attempt to provide an answer to this. It is difficult for me to judge what the main takeaway messages are in terms of main design principles and how much each of the modules (design choices for sub-components) contribute towards the varying amounts of empirical gains observed on the different tasks.

- The presentation of technical contributions and their motivations, design of experiments and analysis of results can be improved. (see below)       


Comments on writing, presentation and typos:

- In general, I found the flow of argumentation in the Introduction section very winding and confusing. At certain parts it almost reads like a historical account of research in deep learning for sequence processing. Currently the motivation for the central research problem is expressed in a scattered manner throughout the section but not explicitly stated in a succinct manner. It would greatly help if the authors introduced and motivated their research question in a succinct manner and how their proposed solution directly addresses this challenge. The writing in the introduction can be significantly modified to favor brevity and precision about the research question being addressed and its significance.

- In Section 2, again the flow of argumentation is very winding and confusing. It would help to directly get to the main points (section 2.1 can be cut down significantly to deliver the main points). Further, in section 2.2 the paragraph on preliminaries can be significantly cut down. At this point, the multi-head attention module and Transformer architecture are fairly standard in the deep learning toolkit that they can be simply referenced. Only specific modifications applied to these standard modules used or the way they’re combined in the overall TLB model architecture can be explained. 

- Further, the mathematical notation used in section 2.2 is unclear. I’m confused as to what the intended notational format is with regards to usage of capital letters, small letters and boldface characters? Is it that capital letters indicate a set, small letters a and boldface a vector? If so, then there are several inconsistent usage of this notation throughout section 2.2 which reduces the clarity and readability of equations etc. I request the authors to specify the notation format used upfront and be consistent with it. 

- The authors seem to only present the details of the proposed approach . I believe it is absolutely essential that this be present (at least large parts of it) in the main text as it is the central technical contribution of the paper. Space can be created by re-structuring/condensing other sections of the paper as mentioned earlier. 

- Lines 34-35: “... which force the entire hidden state to be compressed into a single hidden state” -> “... entire history into a single hidden state”?    

- Lines 40-41: “…, especially in the presence of long-range dependencies” -> missing citation. Also sentence construction can be improved -> use of long-sequences & long-range dependencies is redundant. 

- Lines 53-54: “... and may bias them towards modeling unnecessary, irrelevant or noisy interactions” -> unclear what this means and how it is connected to motivating the TLB?
 
- Line 105: “... is not completely new.” -> “is not a completely new idea”?

- Line 104: “.... and slow processing mechanisms in machine learning” -> “... and slow processing mechanisms in neural networks/deep learning” a more precise phrasing perhaps? 

- Line 115/215: “… [53] explored” and Line 118: “... [22] showed”, please correct the citation format appropriately to XYZ et. al [53].   

- Line 126: “We first describe our notation” -> uninformative sentence - can be removed.

- Repeated usage of the phrase “temporal latent bottleneck” in sections 1 & 2 can be replaced with the abbreviation TLB which is later adopted in section 4. 

- Line 158: “As mentioned previously,... ” -> uninformative/redundant phrase, can be removed.  

- Line 324: “Baby AI. BabyAI [15]” -> minor typo. 

- Easily noticeable that the spacing between sections and subsections has been significantly altered (reduced) with respect to the standard template (for instance spacing before sections 2, 3, 4). Please adhere to the conference template. 

questions: Following are a list of questions (please note that the order does not represent the importance/significance):

1) Several component modules which form the overall TLB architecture bear resemblance to several other recent works like the Perceiver [1], Slot Attention [2], HCAM [4] (some of which the authors have mentioned). Therefore it would be insightful to understand why the specific model architecture of the TLB architecture is advantageous to these recent architectures that also are motivated by similar considerations such as latent bottlenecks for compression of information using cross-attention mechanisms (Perceiver [1], Slot Attention [2]), representing information at different temporal granularities (HCAM [4] and Hierarchical Transformer [5] in Transformers), a dual stream of information processing for fast and slow learning (DCEM [3]). 

2) As one of the ways the authors motivate the use of the TLB architecture seems to be the gains in training data and memory. It would help to show some quantitative metrics (number of training steps to converge, hardware requirements in terms of memory and wall clock times for training the TLB vs the vanilla Transformer) on a few representative benchmarks/tasks to support it.
 
3) While I greatly appreciate the efforts of the authors to test their proposed TLB architecture on a variety of tasks across different domains. I find the analysis and insight from each of the experimental results lacking depth. The authors do not present an analysis of the results beyond how the model improves evaluation metrics on the various empirical benchmarks. It would be beneficial to move some of the experiments/results to the Appendix and try to show deeper analysis of the empirical performance and reasons behind it on fewer tasks. What is the broader takeaway message/insight for a practitioner beyond the empirical gains it yields to varying degrees over standard Transformer architectures on various tasks?

4) In Figure 3, on the BossLevel the average success rate of the Transformer (in orange) seems to still be rising and not converged yet (looking at the slope of the line) whereas the Transformer + TLB (in blue) has largely flattened out. Could it be that while the Transformer + TLB converges faster it could yield a worse final performance than the standard Transformer (in orange). Could the authors comment if tracking the performance after further timesteps the standard Transformer would yield a higher success rate?  

5) On some benchmarks the authors show the average and standard returns over 10 seeds whereas on others only on 3 seeds (Tables 1 & 2). Could the authors please use a consistent experimental protocol throughout (atleast 5 seeds consistently)? Also, the authors report evaluation metrics on visual datasets inconsistently i.e. in Tables with classification accuracy (Tables 1 & 2) and some others as bar plots (Figure 2). Could the authors maintain consistency here as well? Could the authors clarify how many seeds are used in the self-supervised learning experiment (Figure 2)?  

6) Lines 224-227: On the similarities of the proposed model with concurrent work on the Block Recurrent Transformer. The authors claim that they have found performing cross attention and self attention in sequential performs better than doing them in parallel as used by the Block Recurrent Transformer (lines 224-225). I could not find any experiment in the paper where they provide empirical evidence to support this claim. Further, they also state that “they use special tricks to deal with some instabilities in their case”. It is unclear what the authors are exactly referring to here, could the authors please elaborate on what these “special tricks” are and what the potential downsides of them are as compared to the proposed TLB architecture?    

Overall, I like the high-level idea and greatly appreciate the efforts of the authors to empirically test their proposed architecture’s performance on a wide variety of domains. But the presentation (writing) of ideas and motivation behind the specific design choices seem rather unclear to me. Also, the design of experiments do not help to get a deeper insight about its working and also why this specific architecture is better than several previous related ideas to improve the standard Transformer architecture. I think it would help gain a better understanding of the proposed architecture and its several parts by designing careful experiments that show why these specific choices are better than some competing alternatives that attempt to do the same as highlighted above.  

References:

[1] Perceiver: General Perception with Iterative Attention, Jaegle et. al, 2021.

[2] Object-centric Learning with Slot Attention, Locatello et. al, 2020.  

[3] Grounded Language Learning Fast and Slow, Hill et. al, 2020.

[4] Towards mental time travel: a hierarchical memory for reinforcement learning agents, Lampinen et. al, 2021. 

[5] Hierarchical Transformers are more efficient language models, Nawrot et. al, 2021.

[6] Ordered Memory, Shen et. al, 2019. 


************************** Post-Rebuttal Update  ****************************************************** 

I've updated my rating to a 5 post-rebuttal in light of the changes/ improvements 
made to the paper. I believe this submission is very much a borderline one and its 
overall quality could greatly be improved with more revisions that demand more time 
beyond the rebuttal period. I hope the authors can take the time to address any 
remaining concerns raised all the reviewers well to continue to improve the submission.  

***************************************************************************************************************
limitations: The authors only very briefly describe the limitations of their method by simply commenting about the fixed chunk size used, a hyperparameter which requires some domain knowledge to tune. Ideally, it would be great if the authors can show at least one task/domain on which the proposed TLB architecture struggles with and comment on the plausible reasons behind it. This would  benefit the research community to develop a deeper understanding of the proposed architecture including its limitations and also allow for future work to build on these insights.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors introduce a transformer-based architecture that processes sequences in sequential chunks or blocks. Each chunk includes K elements from the sequence and is processed by a stack of transformer layers that include self-attention and cross-attention heads. Then the output of each chunk is processed sequentially using another cross-attention-based transformer. The hidden or latent state generated at output of this layer, is included as input to the cross-attention layers in the subsequent chunk. The method is then compared against transformers in image classification, self-supervised learning, reinforcement learning, and language tasks. The paper exhibits improved results across all tasks.


strengths_and_weaknesses: The idea introduced in the paper of mixing recurrent networks using attention and transformers is not novel. It has been presented in several works in different flavors. Referenced papers [22] and [53] are two examples. Authors would like to check [a,b,c,d] below and explain how novel their solution is compared to these. The novelty of the proposed method seems limited. 

The paper is of empirical nature. The experiments compare results of the proposed architecture with other transformers that are not necessarily the state of the art. It is unclear whether there is a benefit to this method when comparing to more advanced transformers that aim to resolve similar problems. The generalization results from resolution of 64 to 128 pixels is nice but not sufficient (e.g., the results still drop by 20% in accuracy for CIFAR100). Some works show other possibilities for generalization. Other analysis beyond the ablation of the "top-down" connection would have been relevant to show the value of this method. 

The notation in the paper is confusing for moments, it could be more consistent. The experiments section is missing many details in the main text.

The work is relevant to this community. However, given the limited novelty, its significance is low. 

[a] Recurrent Independent Mechanisms, Goyal et al., ICLR 2021
[b] Block-Recurrent Transformers, Hutchins et al., 2022
[c] MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning, Lei et al., ACL 2020
[d] Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention, Katharopoulos et al., ICML 2020


*************** UPDATE *******************
I updated the score to reflect the additional results presented by the authors during the rebuttal.  
questions: Questions to the authors:
* How is the network trained given the recurrent connections?
* How is the network initialized? In particular, how is the initial $I$ selected?
* What ways the input can be partitioned for chunking? Why the one you choose is the best? How does the chunking influence the quality of the network? Can it always be applied?
* What are the model sizes? Is there any difference when using the recurrent model presented here versus the other transformer models?
* How does your method compare to [22], [53] (and other similar papers) in the experiments you presented?
* Do you use "bidirectional" (or non-causal mask) for language modeling within each chunk?
limitations: Societal impacts are not described in the paper.
Limitations: What would happen if the critical input data is "chunked" between two related pieces of information? Would the model be able to solve it at the same quality of a fully parallel transformer?
What are the pros and cons in the runtime to this method compared to the vanilla transformer?
review_title: null

Meta-review: metareview: This paper proposes a new architecture consisting of both a recurrent and transformer-based component. The reviewers found the performance of the model to be impressive across multiple domains, but were less convinced by some of the quasitechnical claims in the storytelling (even Reviewer p6Zz, who champions the paper expressed this concern). I encourage the authors to be a but more disciplined about these claims in the camera ready version if accepted. Overall the authors engaged admirably in the discussion period providing a strong effort to improve the paper and digestible summaries of the original complaints and corresponding changes in the improved draft. 
confidence: Less certain
Decision: Accept