Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning
Authors: Aniket Rajiv Didolkar
Kshitij Gupta
Anirudh Goyal
Nitesh Bharadwaj Gundavarapu
Alex Lamb
Nan Rosemary Ke
Yoshua Bengio
Venue: NeurIPS 2022 Accept (2022)
Keywords: Transformers
Fast and Slow Mechanisms
Temporal Bottleneck

Abstract: Recurrent neural networks have a strong inductive bias towards learning temporally compressed representations, as the entire history of a sequence is represented by a single vector.  By contrast, Transformers have little inductive bias towards learning temporally compressed representations, as they allow for attention over all previously computed elements in a sequence.  Having a more compressed representation of a sequence may be beneficial for generalization, as a high-level representation may be more easily re-used and re-purposed and will contain fewer irrelevant details. At the same time, excessive compression of representations comes at the cost of expressiveness.  We propose a solution which divides computation into two streams.  A slow stream that is recurrent in nature aims to learn a specialized and compressed representation, by forcing chunks of $K$ time steps into a single representation which is divided into multiple vectors.  At the same time, a fast stream is parameterized as a Transformer to process chunks consisting of $K$ time-steps conditioned on the information in the slow-stream.  In the proposed approach we hope to gain the expressiveness of the Transformer, while encouraging better compression and structuring of representations in the slow stream. We show the benefits of the proposed method in terms of improved sample efficiency and generalization performance as compared to various competitive baselines for visual perception and sequential decision making tasks. 


Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning
Aniket Didolkar' 1 Kshitij Gupta ' Anirudh Goyal 1 Nitesh B. Gundavarapu
Alex Lamb Nan Rosemary Ke Yoshua Bengio 1,4
Abstract
Recurrent neural networks have strong inductive bias towards learning tempo- rally compressed representations, as the entire history of a sequence is represented by single vector. By contrast, Transformers have little inductive bias towards learning temporally compressed representations, as they allow for attention over all previously computed elements sequence. Having more compressed rep- resentation of a sequence may be beneficial for generalization, as a high-level representation may be more easily re-used and re-purposed and will contain fewer irrelevant details. At the same time, excessive compression of representations comes at the cost of expressiveness. We propose solution which divides compu- tation into two streams. A slow stream that is recurrent in nature aims to learn a specialized and compressed representation, by forcing chunks of K time steps into single representation which is divided into multiple vectors. At the same time, fast stream is parameterized
a Transformer to process chunks consisting of K time-steps conditioned on the information in the slow-stream. In the proposed approach we hope to gain the expressiveness of the Transformer, while encourag- ing better compression and structuring of representations in the slow stream. We show the benefits of the proposed method terms of improved sample efficiency and generalization performance as compared to various competitive baselines for visual perception and sequential decision making tasks.
Introduction
The interplay between fast and slow mechanisms for information processing and perception has been studied in both cognitive science and machine learning (5] In the brain, short-term and long-term memory have developed in specialized way. Short-term memory is allowed to change very quickly to react to immediate sensory inputs and perception. also tends towards high capacity storage of all pieces of information which may be relevant for future reasoning
42 4 By contrast, long-
term memory changes slowly [45] 411, is highly selective and involves repeated consolidation. contains of memories that summarize the entire past, only storing details about observations
which are most relevant 28 6)
Deep Learning has seen variety of architectures for processing sequential data 36 example. recurrent neural networks compress information about sequence into a single hidden state. Transformers get rid of the recurrent state by dynamically capturing information between positions using multi-head dot product attention [61]. Transformers have become the dominant architecture across wide range of domains including vision (25), natural language 224 54 II
57
18 For
69
They have eclipsed recurrent neural networks 18 in almost all sequence processing domains due their high representational capacity and scalability. Despite their wide applicability, is well known that Transformers are very data
20
55
and reinforcement learning 13
40
36
57
01 Mila, University of Montreal, Microsoft Research, New York, NY, Google Deepmind, CIFAR Fellow, Google Research, Corresponding authors: Maliollkar13e@mall com
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

JF
FF
X
XG Chunk!
Jacs Chunki++
Sequeence
Figure I: Perceptual module + Temporal Latent Bottleneck Model. F denotes the perceptual module or the fast stream which is a Transformer. I represents the temporal latent bottleneck state (consisting of set of vectors) that are updated using recurrent function denoted by G. The given sequence is first divided into chunks of size K and each chunk X; is processed by JF which consists of interleaved SELF ATTENTION + FFN (denoted in blue) and CROSS ATTENTION FFN (denoted in green) layers. The CROSS ATTENTION FFN layers allow the representation of F to be condi- tioned on top-down information from Z. The representations of the temporal latent bottleneck state is updated using the outputs of F by recurrent function G, which consists of a CROSS ATTENTION FFN layer as shown the circle.
hungry and work well mainly at scale. This can be attributed to their inductive bias towards modeling all possible pairwise interactions in the sequence which results in no consolidation of information. This lack of selectivity in the attention mechanism also leads to a high computational complexity which scales quadratically with input size. Additionally, modeling all possible pairwise interactions maybe extremely wasteful and may result in capturing unnecessary information not useful for the downstream task 22 39 The goal of this work is design an architecture for autoregressive modeling that has an inductive bias towards learning temporally compressed representation that retains the benefits of Transformers while preserving long-rane rangrranee interactions.
For learning temporally compressed representations, we start by dividing the computation of the Transformer into two streams of processing fast stream and slow stream. Inspired by the idea of long-term and short-term memory, we want the fast stream to have a short-term memory with high capacity that reacts quickly sensory input. We refer to this fast stream the perceptual module and implement it using a Transformer since they are known to have high representational capacity. On the other hand, want the slow stream to have long-term memory which updates at slower rate and summarizes the most important information in the input sequence. We refer to this slow stream as the Temporal Latent Bottleneck.
Implementation- wise, we divide the input into fixed size chunks (Figure []. The fast stream operates within each chunk while the slow stream consolidates and aggregates information across chunks up- dating itself once per chunk. This leads to information asymmetry between fast and slow stream as the fast stream contains fine- fine-grained local information while the slow stream contains coarse -gaained distant information. Such kind of information asymmetry has shown to improve generalization and adaptation performance of learned policies in the context of RL 30 277 The fast and slow streams interact with each other though bottleneck of attention. The division of computation into a fast and slow stream eliminates the need for capturing all possible pairwise interactions and thus introducing selectivity in the attention mechanism resulting in a much lower computational complexity which is not quadratic in the input size. We show that the limited capacity of the slow stream and consoli- dation of information by recurrent neural network prevents the model from capturing unnecessary information not useful for the downstream task. We evaluate the proposed model in a number of domains showing that it consistently outperforms competent baselines showing improved general- ization to scenarios not seen during training.
2
Methodology
We now present the proposed approach in detail. Our model jointly leverages the strengths of Trans- formers 61 and recurrent neural networks [8 36
2.1 Desiderata for Fast and Slow Streams of Processing
We give the detailed description of the proposed model in the next section. Here, we give an overview of our architecture and discuss some of its key properties. Given an input sequence, it

---

Algorithm 1: PyTorch-style pseudocode for proposed model
C(query, key, value): CROSS ATTENTION + FFN LAYER
# (query, key, value):
SELF ATTENTION FFN LAYER
L: Num. Layers
# R: Num. C <pr S
X: Input sequence of length shape: [ DJ
# I: The Temporal Bottleneck
K:
Chunk Size
torch. chunk (X, K, dim 1) List of length [T/K] with each element of size [B K x D]
for Ne in X: for 1 in range(L),
Xe = S'Xe
Xex X
if 1 % R 0:
X C
(L I. I)
I= C(I,
X X.)
is first divided into chunks of size K. Each chunk is processed by perceptual module represented by Transformer (denoted as F). While processing each chunk, F is also conditioned information from the Temporal Latent Bottleneck module G. The slow stream a recurrent stream which has its own state consisting of set of N vectors (or slots) also called temporal latent bottleneck state denoted as I in Figure | In the following sections, we use the term temporal latent bottleneck to re- fer to the temporal latent bottleneck state I. This state is updated once per chunk using information from the perceptual module through a cross attention mechanism.
The perceptual module operates within each chunk while the temporal latent bottleneck operates across chunks slowly updating itself after each chunk has been processed by the perceptual module. Thus, the only way the perceptual module gets information about inputs beyond its chunk through the temporal latent bottleneck. An added advantage of this that the computational complexity of the attention mechanism in the proposed model is O T (K2 + KN)) while that of Transformer is OTT²). where T is the length of the sequence, K is the chunk size, and N is the number of temporal latent bottleneck state vectors. Since K << T and N << T, we can that ( KN < T Therefore the proposed model has much lower computational complexity compared to a Transformer. Furthermore, the capacity of the temporal latent bottleneck is limited and much smaller than that of the perceptual module. This encourages the temporal latent bottleneck to represent the most salient information about the past while the perceptual module represents only local information. This creates an information asymmetry between the two streams. This information asymmetry leads to the perceptual module having fine grained view of the nearby inputs but very coarse grained view of the distant past. This is very different from the usual self- attention which attends to all tokens in the sequence at the same level of granularity.
K
An advantage of having compressed representation of the past is that it allows the model to forget irrelevant information. For example, if an agent is navigating in a large maze, it does not need to have fine grained knowledge of its actions from the distant past. In the case of Transformer, it would attend to every step from the past (including steps from the distant past) which may be irrelevant in the present context thus wasting its capacity in modeling irrelevant details. Another important component of the proposed model is top-down attention which conveys contextual information from the high-level Temporal Latent Bottleneck module the processing of low-level perceptual mod- ule. Past works [52 26
34 23 have shown that top-down attention improves generalization and adaptation performance of the learned model. One difference between these works and the proposed model is that in their case the multiple streams operate at the same temporal granularity while in our different time scales (because of information asymmetry). Through
case the streams operate our experiments, we show the advantage of the proposed architecture over these works. Next, we describe the detailed implementation of the proposed model.
2.2 Computational Steps
We denote the input X sequence of T tokens X [To, 21, 11, Z2...., Øz). We chunk this input into chunks of size K resulting in [T/ KJ chunks. We refer to tt chunk as X,. We represent the
3

---

state of the temporal latent bottleneck I (i.e. the slow stream) as set of M d-dimensional vectors. As mentioned previously, we denote the temporal latent bottleneck module & and the perceptual module as F. G updates the temporal latent bottleneck state while F processes chunks X₃ to form the latent representation X1-
Perceptual Module = FXI,T) = 9[,, Xi)
(1)
Temporal Latent Bottleneck Module
(2)
Preliminaries. The central components of our model are the key value attention mechanism and the FFN module [6]] We use two forms of the attention mechanism -(1) Self Attention 66
61
In
this the query and key vectors refer to the same set of vectors; (2) Cross Attention (29 39 this the query and key vectors refer to seperate sets of vectors.
In
Perceptual Module F. As mentioned previously, the perceptual module refers to the fast stream that acts directly on the input. The perceptual module operates on each chunk separately. Therefore, at any time the input to the pereeptual module are the tokens corresponding a particular chunk X; = [ Jix K+11-++---- Itx K +K]- The perceptual module is a Transformer with self attention layers, cross attention layers, and FFNs. has kinds of layers (1) SELF ATTENTION + FFN; (2) CROSS ATTENTION + FFN. The SELF ATTENTION + FFN layers process the input tokens and the CROSS ATTENTION + FFN layers integrate top-down information from the temporal latent bottleneck state I as follows-
X; = INY + X X, = FFN(LN(X,)) + X₁
(3)
We include one CROSS ATTENTION FFN layer per R SELF ATTENTION + FFN layers. The diagramatic representation of the perceptual module is presented in Figure (in the processing of chunk X1). In the figure, we set R 1.
Temporal Latent Bottleneck Module G. The temporal latent bottleneck (TLB) module represents the slow stream that operates on the temporal latent bottleneck state is updated using infor- mation from a particular chunk processed by the perceptual module. This update happens once for each chunk of the perceptual module resulting in [T/K] updates for Z. Since the temporal latent bottleneck state I updates at lower frequency than the perceptual module, it is expected to cap- ture more stable and slowly changing features while the perceptual module captures faster changing features resulting in multiple scales of information representation. An update to the temporal la- tent bottleneck state I consists of cross attention operation where the queries come from I and the keys and values come from the output of the perceptual module. This cross attention operation is followed by an FFN update to I. Consider the perceptual module outputs for a chunk / to be X1 [TIk ................................ kk] The update operation is implemented as follows:
I = ALTEETION(L((I), LN(()) + +I
4+ = FFN(LN(Z)) +I
(4)
The temporal latent bottleneck module introduces the notion of recurrence in model. We show the details of this module in Figure [
(inside the circle).
Perceptual Module Temporal Latent Bottleneck Model. We now present our complete archi- tecture integrating both the perceptual module and the temporal latent bottleneck together. Given a sequence of tokens X = [No, 21, ......... xt]. We chunk this input into chunks of size K resulting in [T/K] chunks. The chunks are processed sequentially one after the other. For a chunk k it is first processed using the perceptual module conditioned on information from the temporal latent bottleneck state. The outputs of the chunk are used to update the temporal latent bottleneck state I. The resultant temporal latent bottleneck state is then used to process the next chunk. The full model is presented in Figure]] [ We use a Transformer as the perceptual module in our experiments. Thus our main contribution is introducing temporal latent bottleneck into Transformers and show- ing its advantages through a variety of experiments. We also present the detailed algorithm for the proposed approach in Algorithm I
The proposed model is similar to a parallel work called Block Recurrent Transformers (37. There are few differences between our work and theirs. First, they use sliding window attention, while

---

we divide the input into chunks. In their paper, they perform cross attention and self attention in parallel while we find that doing them sequentially and performing cross attention once per R self attention steps yields better results. We defer the rest of the discussion on related works to Appendix Section 6
3 Experiments
Our goal is to show the wide applicability and benefits offered by the temporal latent bottle- neck, which we refer to as TLB. We demon- strate that the proposed model outperforms competitive baselines across many domains in- cluding vision, reinforcement learning, and nat- ural language. Our main goal is to show that the proposed approach has high expres- sive power like Transformers while also being sample efficient unlike Transformers. Thus our main baselines are based on the original Trans- former architecture. For example, we compare against ViT (25] in image classification, Deci- sion Transformer 13 in Reinforcement Learning, and Vanilla Transformer in rest of the tasks. We also compare against some representative baseline that offer some of the key properties that our model offers. For example, we compare against state-of-the art Swin Transformer [50] which is a strong baseline for image classification and is also hierarchical similar to the proposed model. We also compare against Transformer LS (70) which also processes long-term and short-term informa- tion using different attention streams. Furthermore, we also compare against Feedback Transformer 226, which also introduces top-down communication into Transformers. Another key point of the proposed model that any position cannot attend to any information from the future beyond its chunk since the temporal latent bottleneck only summarizes the past, not the future. Meanwhile, all the baselines we consider have bidirectional context i.e. they can attend all of the past and the future. We observe that despite this limitation, the proposed model outperforms all the considered baselines.
Table 1: Image Classification. Here we com-
pare the performance of the proposed VIT + TLB model against VIT and SWIN V2 on CIFARIO and CIFARIOM datasets for 64 x 64 images and 128 x 128 images. Note that the model is trained only the 64 x 64 sized images and then trans- ferred to 128 x 128 sized images. Results averaged across 3 seeds.
CIFARTO 128
CIFARI00 128
MODRL VII
93.75
73.18
69.53
47.4
SWIN V2 VIT TLB
97.66
84.9
79.95
58.59
94.79
84.38
79.17
59.19
3.1 Temporal Latent Bottleneck For Perception
Image Classification. Recently, Transformers have been widely applied for visual perception and have shown strong performance improve- ments over CNNs in tasks such as image clas- sification, semantic segmentation, instance seg- mentation, etc. In this work we focus on image classification using Transformers. For model to do well on image classification, it should learn to only focus on the relevant information and ignore other details (eg. background infor- mation). Self attention does not inherently have this inductive bias of ignoring irrelevant infor- mation since models all pairwise interactions between the inputs. We posit that adding lim- ited bandwidth temporal latent bottleneck into the Transformer will allow the model to focus only the most important information in the image which should enable the model to perform well.
Table 2: Here we show the performance of the proposed ViT TLB model against two baselines One with no access to the past and One with no top-down information (i.e. high level to low level communication). We can see that the model suf- fers a drop in performance for both the baseline thus showing the importance of past information and top-down communication. Results averaged across 3 seeds.
CIFARIO
MODEL
PAST
TOP DOWN
64
128
INFO
VIT TLB No PAST INFO.
94.79
84.38
91.30
72.92
No Tow-DowN CONDN
93.75
83.59
Results. We test our hypothesis on the CIFAR10 and CIFAR100 47 image classification datasets. We also test the generalization abilities of the models by comparing their performance on images of higher resolution (128 x 1 128) than seen during training (64 x 64). We use ViT (25) and Swin Trans- former V2 (denoted as Swin V2) (50) as our baselines. Swin Transformer V2 has key strength of generalizing to higher resolution images than those seen during training, making it strong baseline. The input image split into patches of size 4 and fed in rastor order to all the models. For the
proposed model we use VIT as the perceptual module and add temporal latent bottleneck module
5

---

to it. We call this model VIT + TLB. To predict the classification scores, we take the mean across the final temporal latent bottleneck state vectors and pass the resulting representation through an MLP. We present the results for this experiment in table[]] We can see that VIT TLB oulperforms VIT for all cases and performs comperitively to Swin Transformer V2. For further hyperparameter details, we refer the reader to Appendix section 7.1
Quantitative Analysis. One essential component of our model is top-down conditioning. Top down information helps in integrating information from the past as well as high-level information into the perceptual module. We hypothesize that both these kinds of information are im- portant for the model to perform well. To test this, we design two baselines- (1) VIT TLBB (No PAST INFO): In this baseline, we do not allow the TLB to communicate to the perceptual module, therefore the perceptual mod- ule has no information about the past; (2) VIT TLB (No ToP-DowN CONDN): In this baseline, we have a separate temporal latent bottleneck module at every layer, therefore the perceptual module has access to past infor- mation but does not have access to any high-level infor- mation through top-down feedback. We show the results for this ablation in Table 2 We can see that the perfor- mance of both the baselines worse than the proposed VIT TLB model. This shows that both high-level infor- mation through top-down feedback and information from the past is important for the model to perform well.
We want to analyze the in- formation which is being used to influ- ence the processing of Temporal Latent Bottleneck. To calculate this, culate the attention scores for different patches and we mask out all the patches that are not in the top 30% of the atten- tion scores. We can see that for both the images it recovers the foreground al- most perfectly which shows learns to focus on the most important information required to solve the downstream task.
Figure 2:
cal-
Qualitative Analysis. To get better understanding of
what the temporal latent bottleneck is doing, we visualize the parts of the image where the temporal latent bottleneck pays most attention while it is being updated by the perceptual module. We present this visualization in Figure 2] We can most attention to the foreground in each case. This further confirms our hypothesis that the limited capacity bottleneck focuses on the most important information required to solve the downstream task.
that the temporal latent bottleneck learns to pay the
Many recent works have used Vision Transformers for self- 49 48). Here
Self Supervised Learning.
supervised learning 8 2
32
12
80.0
TLB
we show a proof-of-concept that introducing temporal latent bottleneck in Vision Transform- ers results in better self-supervised representa- tions. We consider the SiT model from (2] for this experiment. They use 3 objectives to pre-
77.5
Linear Probing Accuracy
75.0
72.5
70.0
67.5
train their model (1) The Reconstruction Ob- jective- Reconstructs the input image, (2) The
Rotation Prediction Objective Predicts the ro- tation angle from [0", 90", 180", 270%], and (3) The Constrastive Objective (similar to SimCLR 14). For the proposed approach, we introduce a temporal latent bottleneck into SiT resulting in the SiT + TLB model. SiT also uses addi- tional trainable contrastive and rotation tokens as input for calculating the contrastive and rota- tion objectives respectively. For SiT + TLB, we take the mean across the temporal latent bottleneck state vectors and use the resulting representation for computing the rotation and contrastive objectives. We use chunk length of for the SiT TLB model. We pretrain the model for 400 epochs and evaluate the pretrained model at different epochs using linear probing.
200
Pretraining Epochs
Figure 3: Self Supervised Learning Results of linear probing on the CIFAR 10 dataset for models pretrained on the STL 10 dataset. We can see that the proposed SIT + TLB approach outperforms SiT.
Results. To evaluate the model, we pretrain the model on the STL10 dataset 22 and evaluate
the learned representation using linear probing on the CIFARIO dataset [7]. We present the re-

---

Transtommer
Transtormer
TLB
BossLavel
MultiTask
10000
Figure 4: Single Task BIbyAL. (Left) Here we compare the performance of Transformer and Trans- former TLB on the BossLevel task from BabyAL. We can see that while both the models con- verge to similar success rate, Transformer + TLB converges faster than Transformer. Multi Task AI (Right) Here compare the performance of Transformer and Transformer TLB on tasks from the BabyAl suite of environments, A single model is trained for all the 8 tasks. We can see that Transformer TLB converges faster and achieves a better performance than Transformer.
sults for this experiment in Figure 3 We can see that the proposed approach outperforms SiT thus showing the effectiveness of the proposed architecture for self-supervised learning. For additional experimental results and details, we refer the reader to Appendix section
7.2
3.2 Temporal Latent Bottleneck for Sequential Decision Making
Transformers have recently been used for sequential decision making in reinforcement learning tasks such as Atari and AabyAl (8). These works deploy Transformers in the offline RL setting where large number of trajectories The Transformer is trained as an autoregressive generative model that predicts actions conditioned on the past context. We incorporate the temporal latent bottleneck module into the Transformer and explore its benefits in the RL setting. We test the proposed model in the Baby Al [5] and Atari [9] benchmarks. We describe our setups detail below.
available either through another trained agent an expert agent.
Instruction Based Decision Making: BabyAL BabyAI [5]] provides a suite of environments
given instruction in partially-observable maze. These instruc- tions include competencies such as going to an object in the maze, placing an object beside another object in the maze, opening door with key, etc. Some environments in the benchmark contain instructions that combine multiple competencies sequentially. For example, pick up a red ball and open the door in front of you after you pick up the grey ball on your left and pick up red box. Each environment in Baby Al benchmark has a different type of instruction that tests a different competency. The BossLevel is the most complicated environment that contains instructions from all competencies. For more details regarding the various environments from the Baby AI benchmark, we refer the reader to Appendix section 7.4
where the agent has carry
We train our models with behavior cloning using expert trajectories from an oracle. For evaluation, test the model by directly deploying it in the environment. We report the success rate which mea- sures whether the agent successfully carried out the given instruction or not. We use a Transformer [6]] as the baseline in these experiments. For the proposed model, introduce temporal latent bottleneck into the Transformer-based perceptual module. For the baseline Transformer model, we append the language instruction the sequence of states allowing the model to attend to the lan- guage instruction at each layer. For the proposed model, the language instruction is appended to each chunk, allowing each chunk attend to
Results. We consider two settings Single task and Multi task. In the single task setting, we evaluate the proposed approach on individual environments from the BabyAI benchmark while in the multi-task setting we train a single model on 8 different environments.
Single Task. We present the results for BossLevel in Figure 4 (left) and present the results for the other tasks Appendix Figure D We can see that while Transformer and Transformer + TLB achieve almost similar performance al convergence. However, Trunsformer + TLB is much more sample efficient, converging much faster. We posit that the temporal latent bottleneck module pro- hibits the model from paying attention to unnecessary information which allows it to converge faster.

---

Multi Task. We present the results for the multi task setting in Figure 4 (right). We train the model on environments PutNext, Unlock, Synth, GoToSeq, SynthLoe, GoTolmpUnlock, BossLevel. We evaluate the model on the same 8 environments. We report the average success rate across 8 games. We can see that the Transformer + TLB model converges faster and also outperforms the Transformer We refer the reader to the appendix for more details regarding the model and training.
Atari 13 recently introduced the Decision Transformer (DT) which learns to play various games in the Atari benchmark from suboptimal trajectories of a learned agent. Decision Trans- former models the offline RL problem as a con- ditional sequence modelling task. The model uses a causal mask and supervised training to match the actions in the offline dataset condi- tioned on the future expected returns and the past history. This is done by feeding into the model the states, actions, and the return-to-go
Table 3: Atari. Here show that adding tem- poral latent bottleneck into decision Transformer improves performance across various atari games. Results are averaged across 10 seeds.
GAME
DT
DT TLB
BREAKOUT
71.51
87.63_ 16.24
PONG
13.6842.00
14.71
QBERT
3268+11880
5019.75 41677:5
SEAQUEST
1039.11 g.77.0
1248.22
R. = -e'= Ve: where c denotes the timesteps. This results in the following trajectory repre- sentation:
(Ri, $1,188,a1, a1, R2, $2, (22, R2, 83, a3,...), where de denotes the actions and se de- notes the states. At test time, the start state 81 and desired return RG is fed into the model and it autoregressively generates the rest of the trajectory. Experimental results show that DT can leverage the strong generalization capabilities of Transformers and achieve the desired returns in wide variety of tasks in Atari and OpenAI Gym, outperforming previous approaches in offline RL.
We use the same setup as used in [3] for our experiments. We set the context length to fixed number C. During training, C timesteps from an episode are sampled and fed into the model resulting in a trajectory of length 3C (considering 3 modalities returns-to-go states, and actions). Each modality is processed into an embedding of size d. The state is pro- cessed using a convolutional encoder into an embedding of size d. The resulting trajectory is fed into the decision Transformer. The out- puts corresponding the states 8, are fed into a linear layer to predict the action a₂ to be taken at timestep c. For the proposed model, we in- corporate a temporal latent bottleneck module into the Decision Transformer.
Table 4: Long Range Dependencies. Here we compare the performance of the proposed model against the recently proposed long-short Trans- former model (70) and the vanilla Transformer model [61] We can see that the proposed model outperforms both the baselines thus showing the superiority of the proposed model in modelling long-ran Sangerane and hicrarchical dependencies. Re-
sults averaged across 5 seeds.
MODEL
LISTÖPS
TEXT CLASSIFICATION 64.0_g.000 05.5
TRANSFORMER
37.64 200000
TRANSFORMER LSS
37.5 +0,0002
TRANSFORMER TL.
40,000
82.08 +0.44
Results. We present our results in Table The model is trained on 1% of the Atari DQN-replae dataset 11 (500K transitions for each game). We use the same 4 games used in [13: Pong, Seaquest, Qbert, and Breakout. We can see that the proposed model outperforms Decision Transformer in all the considered games thus showing the effectiveness of the proposed model. More details regarding the model and training can be found in the appendix section 7.5
3.3 Temporal Latent Bottleneck for Long Range Dependencies
Here, we test the effectiveness of the proposed model in modelling long range dependencies. We apply the proposed model on the ListOps and text classification tasks from the Long Range Arena (LRA) benchmark [60]. Both these tasks have very long sequences ranging from 1K to 4K tokens. Thus, for a model to do well, it has to learn to capture dependencies across very long time scales. Additionally, all these tasks have an inherent hierarchical structure. For example, Listops consists of nested list operations which makes it hierarchical. For text classification, the inputs consist of text in the form of bytes. Therefore, the model has to learn to compose bytes into characters and characters into words. We hypothesize that the multi-scale hierarchical nature of the proposed model will be extremely useful in modelling such hierarchical information.
8

---

Results. For this experiment, we use the same setup as a0]. For the proposed model, we use Transformer as the perceptual model and implement the temporal latent bottleneck as described in Section 2.2 We take the mean across the temporal latent bottleneck state vectors and use the resulting representation for classification. We compare the model against the long-short Transformer (LS) model [70], which is a recently proposed model for the long range arena benchmark, and the vanilla Transformer model [6]]. We present the results in Table 4 We can see that the proposed model outperforms both the baselines in both the tasks thus showing its usefulness in modeling long range dependencies. For further details, we refer the reader to Appendix section 7.3
In Fig. 5 we plot the convergence curves for ListOps and Text Classifi- cation. For ListOps (Figure a)). we plot the convergence curves against the number of samples i.e. we do only one pass over the dataset hence the model does not see any exam- ple more than once. We can see that the proposed Transformer + TLB model is much more sample effi- cient than the baseline Transformer LS model. For Text Classification (Figure []b)), gence curves against the number of training steps. We find that doing only one pass over the dataset does not work well for both the baseline and the proposed model hence we use number of training steps on the x-axis. We can see that while ini- tially both models converge at simi- lar pace, the proposed model achieves a much higher performance.
mwy
(a) ListOps
(b) Text Classification
Figure 5: (a) Here we show the performance on ListOps as function of the number of samples in the dataset. We do only one pass over the entire data and find that Transformer +TLB takes much fewer samples to converge as compared to the baseline Transformer LS. (b) Here we show the con- vergence curves of both the Transformer + TLB model and the Transformer LS model on the text classification task. In this case, we do not perform only one pass over the dataset since we observe that both models do not reach convergence in a single pass. Therefore, we report the number of train- ing steps the x-axis. We can see that the proposed model achieves much higher score than the baseline.
plot the conver-
We measure the wall-clock time and memory required for text classification task the chunk size in Table All TLB models have an increased memory efficiency and supports faster inference speeds with respect to the base- line transformer model. The training speeds also get better with increased chunking. The only exception is very small chunk sizes, where the training is slower than the baseline because of increased temporal unrolling. However, as shown in Figure such models are very sample efficient resulting in lesser training steps over- all.
vary
Table 5: Text Classification Performance Ab- lation Here, we compare the wall-clock time and memory during the training and inference phase of the text classification task w.r.t baseline trans- former model.
CHUNK
1000
SIZE
INFERENCE SPEED
3.5x
3.6x
3.3x
2.2X
1.2x
INFERENCE MEMORY
0.09x
0.08x
0.12X
0.08x
0.1X
Analysis. Here we perform an ablation show that the Temporal Latent Bottleneck does not only contain short-term information but also summarizes information from long term past. To test this hypothesis we design a baseline in which the current chunk attends to the previous few chunks instead of attending to the temporal latent bottleneck. We find that this baseline achieves performance of 32.10 0.019 % compared to the proposed models 38.2.10.000 % on the ListOps task. This shows that the Temporal Latent Bottleneck contains information about the long-term past. Additionally, here also we perform an experiment to probe the importance of top-down communi- cation (i.e. high level to low level feedback). To do this we use the same Transformer TLB (No Top-Down Condn) baseline used Table 2 We find that this baseline achieves a performance of 37.57 +0.003 % which is lower than the performance of the proposed Transformer TLB model which
TRAINING SPEED
4.4x
4.4x
2.2x
1.4x
0.7x
TRAINING MEMORY
0.14%
0.08x
0.49x
0.40x
0.42x

---

Sequence Length
Feedback Transformer
Transformer +
TLBB
100
11800
6200
sccuracy
200
16600
9100
300
35100
12700
400
NA
14600
500
NA
13600
Nocel
Teechark Tranformrr
600
NA
19300
Parafomme:
100
200
300
400
300
100
Table 6: Copying Sample Efficiency Abla- tion. Here we present the number of unique samples required for the models to reach to perfect accuracy on the copying task. NA indicates that the model does not reach per- feet accuracy. We can see that in all cases the Transformer TLB model more sam- ple efficient than the Feedback Transformer model.
Sequence Length
Figure Copying Task. Here we com- pare the performance of the proposed Trans- former + TLB model to the Feedback Trans- former model on the copying task. We can see that the Transformer + TLB achieves per- fect accuracy for all the studied sequence lengths while the the performance of Feed- back Transformer starts dropping after se- quence length 400.
achieves 38.210 % which further shows that top-down information from high-level to low-level is important for the model to perform well.
We perform additional experiments to give more insight into the behavior of the proposed model. We present these experiments Appendix Section 7.3 We also compare the model to additional efficient transformer baselines for all LRA tasks in Appendix Table
Temporal Latent Bottleneck for Copying Task. Here, we study the copying task used in [36] In
the copying task, the model receives a sequence of 10 digits followed by blank inputs for a large number of steps, and then the model asked to output the sequence of digits it received initially. Therefore, the model has to remember the original sequence of digits across long time scales. We can control the sequence length of this task by controlling the length of the blank input.
The main motive behind studying this task is comparing the model to the Feedback Transformer model introduced in (26) which also has top-down attention similar to the proposed model but does not represent information at multiple scales. We compare both the models on the copying task for sequence lengths 100, 200, 300, 400, 500, and 600. We present the results for this task in Figure 6 We can see that while both Transformer + TL.B and Feedback Transform perform well for low sequence lengths, the performance of Feedback Transformer drops for longer sequence lengths above 400 while the proposed Transformer + TLB model still achieves perfect accuracy at long sequence lengths. We also compare the sample efficiencies models. We present this result in Table 6 We can see that the proposed Transformer + TLB is more sample effecient than the baseline Feedback Transformer achieving perfect accuracy in much lesser number of samples in each case. For further details we refer the reader to Appendix Section 7.6
achieve perfect accuracy for both the
4 Conclusion
We have developed an approach aimed at introducing selectivity in the interactions across time-steps in transformer by splitting processing into two streams: (a) a slow stream that is updated in recur- rent manner and (b) a fast stream that processes the visual input. The two streams are parameterized independently and interact with each other via attentional bottleneck. The information processed by the fast stream is used to change the state of the slow stream, and the information in the slow stream used by the fast stream contextual information process the input. Through our experiments we show that the proposed approach works well across wide range of domains and problems. One limitation of the proposed model is that the chunk size is fixed and treated hyperparameter which requires some domain knowledge. Future work should explore methods for dynamic chunking.
5 Acknowledgement
The authors would like to thank Compute Canada for providing the computational resources used in this project. The authors also gratefully acknowledge the funding from Samsung, IBM and CIFAR.
10

---

References
[1] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. Striving for simplicity off- policy deep reinforcement learning. CoRR, abs/1907.04553 2019.
[2] Sara Atito Ali Ahmed, Muhammad Awais, and Josef Kittler. Sit: Self-supervised vision trans- former. CoRR, abs/2104.03602, 2021.
[3] Richard C Atkinson and Richard M Shiffrin. The control of short-term memory. Scientific american, 225(2):82-91, 1971.
[4] Emanuel Averbach and Abner S Coriell. Short-term memory in vision. The Bell System Technical Journal, 40(1):309--3)))) 328, 1961.
[5] Jimmy Ba, Geoffrey E. Hinton, Volodymyr Mnih, Joel Z. Leibo, and Catalin Ionescu. Us- ing fast weights to attend to the recent past. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29; Anmual Conference on Neural Information Processing Systems 2016, 2016.
December 5-10, 2016, Barcelona, Spain, pages
[6] Alan Baddeley, Vivien Lewis, Margery Eldridge, and Neil Thomson. Attention and retrieval from long-term memory. Journal of Experimental Psychology: General, 113(4))518, 1984.
[7] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. CoRR, bbss/149.0043,. 2015.
[8] Hangbo Bao, Li Dong, and Furu Wei. Beit: BERT pre-training of image transformers. CoRR, 10s22406,,, 2021.
[9] Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. CoRR, abs/1207.4708 2012.
[10] Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document trans- former. CoRR, abs/2004.05150, 2020.
[11] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learn- ers. CoRR, abs/2005. 14165, 2020.
[12] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. CoRR, Abs/2104.14294 2021.
[13] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. CoRR, 0bs/2106.01344. 2021.
[14] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. simple frame- work for contrastive learning of visual representations. CoRR, abs/2002.05709, 2020.
[15] Maxime Chevalier-Boisver Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Sa- haria, Thien Huu Nguyen, and Yoshua Bengio. Babyai: First steps towards grounded language learning with human in the loop. CoRR, abs//880.00827.. 2018
[16] Maxime Chevalier-Boisvere Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Sa- haria, Thien Huu Nguyen, and Yoshua Bengio. Babyai: First steps towards grounded language learning with a human in the loop. CoRR, abs/ abs/1810.08272, 2018.
[17] Rewon Child, Scott Gray, Alee Radford, and llya Sutskever. Generating long sequences with sparse transformers. CoRR, abs/1904. 105599 2019.

---

[18] Kyunghyun Cho. Bart van Merrienboer, Çaglar Gülgehre. Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statisti- cal machine translation. CoRR, abs/1406. 1078, 2014.
[19] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Be- langer, Lucy J. Colwell, and Adrian Weller. Rethinking attention with performers. CoRR, abs/2009.14794 14794, 2020.
[20] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shiv- ani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz. Orhan Firat, Michele Catasta, Jason Wei, Kathy MeiereHellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
[21] Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. CoRR, abs//160900174, 2016.
[22] Adam Coates, Andrew Ng. and Honglak Lee, An analysis of single-layer networks in unsuper- vised feature learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors, Pro- ceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 215-223, Fort Lauderdale, FL., USA, 11 11-13 Apr 2011. PMLR.
[23] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoe V. Le, and Ruslan Salakhut- dinov. Transformer--it Attentive language models beyond fixed-length context. CoRR, abs/1901.02860, 2019.
[24] Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018.
[25] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. CoRR, abs/2010.11929, 2020.
[26] Angela Fan. Thibaut Lavril, Edouard Grave, Armand Joulin, and Sainbayar Sukhbaatar. Ad- dressing some limitations transformers with feedback memory, 2021.
[27] Alexandre Galashov, Siddhant M Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojciech M Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry in kl-regularized rl. arXiv preprint 1:05.0/905001222 2019.
[28] Philip Goelet, Vincent F Castellucci, Samuel Schacher, and Eric R Kandel. The long and the short of long-term memory- molecular framework. Nature, 322(6078):419442) 1986.
[29] Anirudh Goyal, Aniket Didolkar, Alex Lamb, Kartikeya Badola, Nan Rosemary Ke, Nasim Rahaman, Jonathan Binas, Charles Blundell, Michael Mozer, and Yoshua Bengio. Coordina- tion among neural modules through a shared global workspace. CORR, abs/2103.01199 2021.
[30] Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Yoshua Bengio, and Sergey Levine. Infobot: Transfer and exploration via the information bottleneck. arXiv preprint arXiv: 1et.//9/.19990 2019.
12

---

[31] Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Ben- gio, and Bernhard Schölkopf. Recurrent independent mechanisms. CoRR, abs/1909.10893, 2019.
[32] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. CoRR, abs/211 .06377, 2021.
[33] Salah Hihi and Yoshua Bengio. Hierarchical recurrent neural networks for long-term depen- dencies. In D. Touretzky, M. C. Mozer, and M. Hasselmo, editors, Advances in Neural Infor- mation Processing Systems, volume MIT Press, 1995.
[34] Felix Hill, Olivier Tieleman, Tamara Von Glehn, Nathaniel Wong, Hamza Merzie, and Stephen Clark. Grounded language learning fast and slow. arXiv preprint arXiv:2009.0/719 2020.
[35] Geoffrey E. Hinton and David C. Plaut. Using fast weights to deblur old memories. In IN PROCEEDINGS OF THE 9TH ANNUAL CONFERENCE OF THE COGNITIVE SCIENCE SOCIETY, pages 177-186. Erlbaum, 1987.
[36] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9:1735 80, 1997.
[37] DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. Block- recurrent transformers, 2022.
[38] Donald Joseph Hejna III, Pieter Abbeel, and Lerrel Pinto. Improving long-horizon imitation through language prediction, 2022.
[39] Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General perception with iterative attention, 2021.
[40] Michael Janner, Qiyang Li, and Sergey Levine. Reinforcement learning big sequence modeling problem. CoRR, 0ts/21106.02039 2021.
[41] Annette Jeneson and Larry R Squire. Working memory, long-term memory, and medial tem- poral lobe function. Learning & memory, 19(1): 2012.
[42] John Jonides, Richard L Lewis, Derek Evan Nee, Cindy A Lustig, Mare G Berman, and Kather- ine Sledge Moore. The mind and brain of short-term memory. Annu. Rev. Psychol., 59:193- 224, 2008
[43] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. CoRR, abs/2006.16236, 2020.
[44] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. CoRR, 0e5/200100445.. 2020.
[45] Janet L Kolodner. Maintaining organization in a dynamic long-term term memory. Cognitive sci- ence, 7(4):243-280) 1983.
[46] Jan Koutník, Klaus Greff, Faustino J. Gomez, and Jürgen Schmidhuber. A clockwork RNN. CoRR, 402/140 2014.
[47] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[48] Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self- superviper vision transformers for representation learning. CoRR, abs/2106.0975$ 09785. 2021.
[49] Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, and Jinqiao Wang. MST: masked self-supervised transformer for visual representation. CoRR, (aa/2106.05656 2021.
13

---

[50] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin transformer V2: scaling up capacity and resolution. CoRR, abs/211 11 .09883, 2021.
0/bs/111.00ff
[51] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Bain- ing Guo. Swin transformer: Hierarchical vision transformer using shifted windows. CoRR, 11122003.14000 2021.
[52] Sarthak Mittal, Alex Lamb, Anirudh Goyal, Vikram Voleti, Murray Shanahan, Guillaume La- joie, Michael Mozer, and Yoshua Bengio. Learning to combine top-down and bottom-up sig- nals recurrent neural networks with attention over modules. CoRR, abs/2006. 16961, 2020.
[53] Michael C Mozer.
Induction multiscale temporal structure. J. Moody, S. Hanson, and R. P. Lippmann, editors, Advances in Neural Information Processing Systems, volume 4. Morgan- Kaufmann, 1991.
[54] Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre- training. 2018.
[55] Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driess- che, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean- Coan-Baptiste Lespiau, Maria Tsim- poukelli, Nikolai Grigorev, Doug Fritz,, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scal- ing language models: Methods, analysis; insights from training gopher, 2021.
[56] Jürgen Schmidhuber. Neural sequence chunkers. Technical report, 1991.
[57] M. Schuster and K.K. Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on 1997.
Signal Processing,
[58] Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention in transformer models. CoRR, abs/2005.000433 2020.
[59] Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. CoRR, abs/2002.1 1296, 2020.
[60] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang. Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. CoRR, abs/201 1. .04006, 2020.
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017.
[62] Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self- attention with linear complexity. CoRR, Abs/2000.04468, 2020.
[63] Wenhai Wang. Enze Xie, Xiang Li. Deng-Pine Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. CoRR, abs/2102.12122. 2021.
[64] Haiping WU, Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang Cvt: Introducing convolutions to vision transformers. CoRR, abs/2103. 15808, 2021.
14

---

[65] Jianwei Yang. Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jian- feng Gao. Focal self-attention for local-global interactions in vision transformers. CoRR, abs/2107.0064) 2021.
[66] Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. CoRR, abs/2103. 1816, 2021.
[67] Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang. Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. CoRR, abs/2007.14066 2020.
[68] Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image en- coding. CoRR, abs/2 2021.
[69] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre- trained transformer language models, 2022.
[70] Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandku- mar, and Bryan Catanzaro. Long-short transformer: Efficient transformers for language and vision. CoRR, abs/2107.02192. 2021.
Checklist
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you deseribe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Appendix Section
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] All the datasets we use are openly
available
(c) Did you include any new assets either in the supplemental material or as a URL?? [No]
15

---

(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] The datasets we use do not contain any sensitive formation.
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "rNblfUQcAbN", "reviewer": "Reviewer_scsa", "summary": "The authors propose a modification to the transformer architecture called Temporal Latent Bottleneck (TLB). The motivation is to provide a more compressed representation by forcing the model to pass chunks of the input stream through a bottleneck. Whereas a standard transformer may attend to any part of the sequence when encoding the current element, the TLB transformer can only attend to elements in the current chunk; the information beyond this chunk is received only through the bottleneck vector.\n\nThe authors maintain that their modified transformer architecture can be applied in any setting where standard transformers were previously used. They provide a series of demonstrations spanning image perception, sequential decision making, and language modeling. In each of these domains, the proposed model outperforms alternative transformer baselines.", "questions": "My major suggestion is outlined above. In addition I have a smaller suggestion about the enwiki8 experiment (section 4.3). This experiment is described in a total of 2 sentences with very little detail about the task, the choice of baseline, or the evaluation criteria. There is also little to no reflection provided about the results. I suggest either removing this experiment or expanding it to a complete description.", "limitations": "The limitations are adequately addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The writing throughout the paper is clear and technically sound. Moreover the quality of the experiments is high. The originality I will rate as low to medium: although the authors describe a grand motivation for the architectural ideas & principles of their proposed model, the proposal is ultimately only a modification to the existing transformer, and there is little support provided to justify the claims made in their introduction (see specific comments below). Nevertheless, the work has a high potential impact and significance given that transformers have become engrained in almost every area of ML/AI, and the authors have demonstrated the effectiveness of their approach in several different transformer use cases.\n\nMy biggest suggestion to improve this paper is the following: I would suggest scaling back the tone and broader framing of the introduction to focus on providing a new transformer architecture that outperforms the default transformer and that is applicable to a number of different settings. At the moment, there’s a major emphasis placed on architectural ideas/principles which are not adequately studied. These include, but are not limited to:\n1.  Bottlenecked/compressed representation for improved sample complexity. Throughout the intro the authors talk about learning a more compressed representation, with the motivation being that a more compressed model should have better sample complexity. However, there is very little discussion or exploration of sample complexity in the provided experiments. In the CIFAR experiments, for example, it’s unclear how many training examples are being used, or whether there is a certain sample complexity at which alternative models begin to outperform the proposed. The idea of sample complexity is touched on very briefly in the Baby AI experiment, although “number of timesteps” is not a perfect analogue to sample size (there may be repeated examples over any period of time, and a model with strong sample complexity could have slow convergence for other reasons). \n2.  Top-down conditioning. The authors suggest that, by conditioning the chunk encoder F on the latent bottleneck state, the model is required to reason in a “top-down” manner, where information from a higher level of abstraction informs the encoding of low-level raw inputs. They justify this claim with an ablation study (Table 2) where they show that, when they remove the top-down connection in TLB, performance deteriorates. Although I think this ablation is interesting, I don’t think it provides support that the model is reasoning “top-down” as the authors say. Given the constraint of the chunking architecture, the ablated model has no form of information about the past whatsoever, at any level of abstraction. It seems obvious that the model would underperform in this configuration, and I don’t feel that it evidences the use of higher level abstraction.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "q9ut5YUbyVR", "reviewer": "Reviewer_EwzR", "summary": "This work addresses the aspect of memory (state) and its expressiveness in deep learning models for sequences. Recurrent neural networks (such as LSTMs and GRUs) on one end compress the entire history into a single state representation vector while on the other end the memory state in Transformers is a continuously appended list of (key, value) vectors from all previous timesteps and therefore has a higher fidelity representation of the entire past. They seek to find an optimal balance between compression and expressiveness of these 2 contrasting forms of memories. Their proposed solution strategy is a kind of best-of-both- worlds which has 2 processing streams where the “slow” stream is instantiated as a recurrent module which updates a set of state vectors (once every chunk size steps of the input) using a cross attention mechanism and the “fast” stream which updates its internal representation at every input timestep. This “fast” stream is instantiated as a Transformer Encoder except with the addition of cross-attention blocks as well in addition to the standard self-attention. They empirically evaluate their proposed architecture against several standard Transformer models across various domains such as visual representation learning (ViT), offline reinforcement learning (Decision Transformer) and language modeling (Transformer-XL).   \n", "questions": "Following are a list of questions (please note that the order does not represent the importance/significance):\n\n1) Several component modules which form the overall TLB architecture bear resemblance to several other recent works like the Perceiver [1], Slot Attention [2], HCAM [4] (some of which the authors have mentioned). Therefore it would be insightful to understand why the specific model architecture of the TLB architecture is advantageous to these recent architectures that also are motivated by similar considerations such as latent bottlenecks for compression of information using cross-attention mechanisms (Perceiver [1], Slot Attention [2]), representing information at different temporal granularities (HCAM [4] and Hierarchical Transformer [5] in Transformers), a dual stream of information processing for fast and slow learning (DCEM [3]). \n\n2) As one of the ways the authors motivate the use of the TLB architecture seems to be the gains in training data and memory. It would help to show some quantitative metrics (number of training steps to converge, hardware requirements in terms of memory and wall clock times for training the TLB vs the vanilla Transformer) on a few representative benchmarks/tasks to support it.\n \n3) While I greatly appreciate the efforts of the authors to test their proposed TLB architecture on a variety of tasks across different domains. I find the analysis and insight from each of the experimental results lacking depth. The authors do not present an analysis of the results beyond how the model improves evaluation metrics on the various empirical benchmarks. It would be beneficial to move some of the experiments/results to the Appendix and try to show deeper analysis of the empirical performance and reasons behind it on fewer tasks. What is the broader takeaway message/insight for a practitioner beyond the empirical gains it yields to varying degrees over standard Transformer architectures on various tasks?\n\n4) In Figure 3, on the BossLevel the average success rate of the Transformer (in orange) seems to still be rising and not converged yet (looking at the slope of the line) whereas the Transformer + TLB (in blue) has largely flattened out. Could it be that while the Transformer + TLB converges faster it could yield a worse final performance than the standard Transformer (in orange). Could the authors comment if tracking the performance after further timesteps the standard Transformer would yield a higher success rate?  \n\n5) On some benchmarks the authors show the average and standard returns over 10 seeds whereas on others only on 3 seeds (Tables 1 & 2). Could the authors please use a consistent experimental protocol throughout (atleast 5 seeds consistently)? Also, the authors report evaluation metrics on visual datasets inconsistently i.e. in Tables with classification accuracy (Tables 1 & 2) and some others as bar plots (Figure 2). Could the authors maintain consistency here as well? Could the authors clarify how many seeds are used in the self-supervised learning experiment (Figure 2)?  \n\n6) Lines 224-227: On the similarities of the proposed model with concurrent work on the Block Recurrent Transformer. The authors claim that they have found performing cross attention and self attention in sequential performs better than doing them in parallel as used by the Block Recurrent Transformer (lines 224-225). I could not find any experiment in the paper where they provide empirical evidence to support this claim. Further, they also state that “they use special tricks to deal with some instabilities in their case”. It is unclear what the authors are exactly referring to here, could the authors please elaborate on what these “special tricks” are and what the potential downsides of them are as compared to the proposed TLB architecture?    \n\nOverall, I like the high-level idea and greatly appreciate the efforts of the authors to empirically test their proposed architecture’s performance on a wide variety of domains. But the presentation (writing) of ideas and motivation behind the specific design choices seem rather unclear to me. Also, the design of experiments do not help to get a deeper insight about its working and also why this specific architecture is better than several previous related ideas to improve the standard Transformer architecture. I think it would help gain a better understanding of the proposed architecture and its several parts by designing careful experiments that show why these specific choices are better than some competing alternatives that attempt to do the same as highlighted above.  \n\nReferences:\n\n[1] Perceiver: General Perception with Iterative Attention, Jaegle et. al, 2021.\n\n[2] Object-centric Learning with Slot Attention, Locatello et. al, 2020.  \n\n[3] Grounded Language Learning Fast and Slow, Hill et. al, 2020.\n\n[4] Towards mental time travel: a hierarchical memory for reinforcement learning agents, Lampinen et. al, 2021. \n\n[5] Hierarchical Transformers are more efficient language models, Nawrot et. al, 2021.\n\n[6] Ordered Memory, Shen et. al, 2019. \n\n\n************************** Post-Rebuttal Update  ****************************************************** \n\nI've updated my rating to a 5 post-rebuttal in light of the changes/ improvements \nmade to the paper. I believe this submission is very much a borderline one and its \noverall quality could greatly be improved with more revisions that demand more time \nbeyond the rebuttal period. I hope the authors can take the time to address any \nremaining concerns raised all the reviewers well to continue to improve the submission.  \n\n***************************************************************************************************************", "limitations": "The authors only very briefly describe the limitations of their method by simply commenting about the fixed chunk size used, a hyperparameter which requires some domain knowledge to tune. Ideally, it would be great if the authors can show at least one task/domain on which the proposed TLB architecture struggles with and comment on the plausible reasons behind it. This would  benefit the research community to develop a deeper understanding of the proposed architecture including its limitations and also allow for future work to build on these insights. \n", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The high-level idea behind the proposed TLB architecture is interesting and the authors have taken a lot of effort to show empirical results of their proposed architectural modification to the standard Transformer on a wide range of domains such as offline reinforcement learning, visual representation learning, language modeling etc. and their benefits. \n\n- The proposed architecture shows very good empirical gains on Atari in the offline setting over the Decision Transformer, and to a lesser extent on classification tasks on CIFAR-10 and CIFAR-100 length (i.e. image resolution) generalization, language modeling on enwiki8 dataset and learning long-range dependencies (ListOps and Text Classification) tasks. The proposed TLB architecture provides benefits both in terms of computational efficiency and performance. \n\n\nWeaknesses:\n\n- Please cite related work on fast and slow learning mechanisms such as the DCEM model [3] which uses a dual stream processing to allow for fast associative memories and slow episodic memories in object-word association/RL tasks, Hierarchical Transformer [5] which biases the Transformer to process long sequences in a more efficient manner by using inductive biases to model the sequence hierarchically, Ordered Memory [6] which learns to represent a hierarchical structure in RNNs.\n\n- The novelty of the overall architecture seems marginal and that it is more an amalgamation of several components introduced by other work such as Perceiver [1], Slot Attention [2], DCEM [3], HCAM [4], Hierarchical Transformer [5] which have similar motivations behind their architectural improvements to the standard Transformer model. Regardless, I think there could be significant value in this specific architecture. But the current experimental design and analysis of results does not attempt to provide an answer to this. It is difficult for me to judge what the main takeaway messages are in terms of main design principles and how much each of the modules (design choices for sub-components) contribute towards the varying amounts of empirical gains observed on the different tasks.\n\n- The presentation of technical contributions and their motivations, design of experiments and analysis of results can be improved. (see below)       \n\n\nComments on writing, presentation and typos:\n\n- In general, I found the flow of argumentation in the Introduction section very winding and confusing. At certain parts it almost reads like a historical account of research in deep learning for sequence processing. Currently the motivation for the central research problem is expressed in a scattered manner throughout the section but not explicitly stated in a succinct manner. It would greatly help if the authors introduced and motivated their research question in a succinct manner and how their proposed solution directly addresses this challenge. The writing in the introduction can be significantly modified to favor brevity and precision about the research question being addressed and its significance.\n\n- In Section 2, again the flow of argumentation is very winding and confusing. It would help to directly get to the main points (section 2.1 can be cut down significantly to deliver the main points). Further, in section 2.2 the paragraph on preliminaries can be significantly cut down. At this point, the multi-head attention module and Transformer architecture are fairly standard in the deep learning toolkit that they can be simply referenced. Only specific modifications applied to these standard modules used or the way they’re combined in the overall TLB model architecture can be explained. \n\n- Further, the mathematical notation used in section 2.2 is unclear. I’m confused as to what the intended notational format is with regards to usage of capital letters, small letters and boldface characters? Is it that capital letters indicate a set, small letters a and boldface a vector? If so, then there are several inconsistent usage of this notation throughout section 2.2 which reduces the clarity and readability of equations etc. I request the authors to specify the notation format used upfront and be consistent with it. \n\n- The authors seem to only present the details of the proposed approach . I believe it is absolutely essential that this be present (at least large parts of it) in the main text as it is the central technical contribution of the paper. Space can be created by re-structuring/condensing other sections of the paper as mentioned earlier. \n\n- Lines 34-35: “... which force the entire hidden state to be compressed into a single hidden state” -> “... entire history into a single hidden state”?    \n\n- Lines 40-41: “…, especially in the presence of long-range dependencies” -> missing citation. Also sentence construction can be improved -> use of long-sequences & long-range dependencies is redundant. \n\n- Lines 53-54: “... and may bias them towards modeling unnecessary, irrelevant or noisy interactions” -> unclear what this means and how it is connected to motivating the TLB?\n \n- Line 105: “... is not completely new.” -> “is not a completely new idea”?\n\n- Line 104: “.... and slow processing mechanisms in machine learning” -> “... and slow processing mechanisms in neural networks/deep learning” a more precise phrasing perhaps? \n\n- Line 115/215: “… [53] explored” and Line 118: “... [22] showed”, please correct the citation format appropriately to XYZ et. al [53].   \n\n- Line 126: “We first describe our notation” -> uninformative sentence - can be removed.\n\n- Repeated usage of the phrase “temporal latent bottleneck” in sections 1 & 2 can be replaced with the abbreviation TLB which is later adopted in section 4. \n\n- Line 158: “As mentioned previously,... ” -> uninformative/redundant phrase, can be removed.  \n\n- Line 324: “Baby AI. BabyAI [15]” -> minor typo. \n\n- Easily noticeable that the spacing between sections and subsections has been significantly altered (reduced) with respect to the standard template (for instance spacing before sections 2, 3, 4). Please adhere to the conference template. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Mry_ZMFpshp", "reviewer": "Reviewer_CN5N", "summary": "The authors introduce a transformer-based architecture that processes sequences in sequential chunks or blocks. Each chunk includes K elements from the sequence and is processed by a stack of transformer layers that include self-attention and cross-attention heads. Then the output of each chunk is processed sequentially using another cross-attention-based transformer. The hidden or latent state generated at output of this layer, is included as input to the cross-attention layers in the subsequent chunk. The method is then compared against transformers in image classification, self-supervised learning, reinforcement learning, and language tasks. The paper exhibits improved results across all tasks.\n\n", "questions": "Questions to the authors:\n* How is the network trained given the recurrent connections?\n* How is the network initialized? In particular, how is the initial $I$ selected?\n* What ways the input can be partitioned for chunking? Why the one you choose is the best? How does the chunking influence the quality of the network? Can it always be applied?\n* What are the model sizes? Is there any difference when using the recurrent model presented here versus the other transformer models?\n* How does your method compare to [22], [53] (and other similar papers) in the experiments you presented?\n* Do you use \"bidirectional\" (or non-causal mask) for language modeling within each chunk?", "limitations": "Societal impacts are not described in the paper.\nLimitations: What would happen if the critical input data is \"chunked\" between two related pieces of information? Would the model be able to solve it at the same quality of a fully parallel transformer?\nWhat are the pros and cons in the runtime to this method compared to the vanilla transformer?\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea introduced in the paper of mixing recurrent networks using attention and transformers is not novel. It has been presented in several works in different flavors. Referenced papers [22] and [53] are two examples. Authors would like to check [a,b,c,d] below and explain how novel their solution is compared to these. The novelty of the proposed method seems limited. \n\nThe paper is of empirical nature. The experiments compare results of the proposed architecture with other transformers that are not necessarily the state of the art. It is unclear whether there is a benefit to this method when comparing to more advanced transformers that aim to resolve similar problems. The generalization results from resolution of 64 to 128 pixels is nice but not sufficient (e.g., the results still drop by 20% in accuracy for CIFAR100). Some works show other possibilities for generalization. Other analysis beyond the ablation of the \"top-down\" connection would have been relevant to show the value of this method. \n\nThe notation in the paper is confusing for moments, it could be more consistent. The experiments section is missing many details in the main text.\n\nThe work is relevant to this community. However, given the limited novelty, its significance is low. \n\n[a] Recurrent Independent Mechanisms, Goyal et al., ICLR 2021\n[b] Block-Recurrent Transformers, Hutchins et al., 2022\n[c] MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning, Lei et al., ACL 2020\n[d] Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention, Katharopoulos et al., ICML 2020\n\n\n*************** UPDATE *******************\nI updated the score to reflect the additional results presented by the authors during the rebuttal.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LHtDHQGOGj", "reviewer": "Reviewer_p6Zz", "summary": "This paper proposes Temporal Latent Bottleneck (TLB) to help transformer-based models benefit from the compression of sequential inputs while not having to (significantly) forego their expressivity. This is done by dividing the computation stream into a slow stream that is recurrent in nature, and a fast stream that allows for attention mechanism between tokens. One of the salient abilities of the proposed architecture is the mutual interaction between both the streams, through a top-down connection between the recurrent and transformer modules (allowing for cross attention), and a bottom-up connection (allowing the recurrent module to take as inputs the final layer of transformer module). Experiments are performed across Vision, Language, and RL domains to show that the improvements brought by the proposed architecture are consistent across domains.", "questions": "Mentioned above", "limitations": "Memory and Time complexity should be talked about. While the computation complexity does decrease, the time complexity is expected to rise as each individual attention block waits for the previous block, and the corresponding RNN module to complete and provide input for the cross attention layer. I would also like to see the difference in training and inference time. Since GPU parallelization is used pervasively, reporting computation complexity is an unfair metric.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The paper does a good job of explaining details of the newly proposed architecture. \n2. Comparisons with baseline methods over multiple domains are convincing and the performance improvements are consistent.\n\n## Weakness\n1. The underlying hypothesis to motivate the introduction of a recurrent stream was that it may benefit generalization since it may contain lesser irrelevant details. I find (a) this hypothesis unconvincing, and (b) no evidence in the paper to support the same.\n2. The paper motivates the use of a transformer architecture by arguing how past work has found recurrent architectures to be incapable of remembering long-range dependencies. In light of the same, it would make a lot more sense if the recurrent stream of the input happened in chunks of K-sized tokens, and the transformer part acted upon the outputs of each layer. This would allow the recurrent stream to do what it is good at (compressing not-so-long range dependencies), and the attention part to attend to all pairs over compressed tokens from anywhere in the input. The current architecture choice appears misplaced, especially based on the motivation that the authors use to position the proposed architecture.\n3. One of the claims of the work is about the ability of the architecture to perform well in resource-constrained scenarios. I would have liked to see an analysis of how the model fairs as more pertaining data becomes available (like in typical language models). Do the advantages of such an architecture still hold up? Will such an architecture be of relevance to the deep learning community at large? Even in case of resource-constrained settings, I find an important analysis missing about the performance change with sequence length. Since the attention module is limited by its attention window to K chunks, this may have an unwanted impact as the sequence length changes. In this regard, text classification on datasets of long sentence length (like IMDb) and short sentence lengths (like SST) would be an interesting point of comparison.\n\n## Other Comments\n1. The use of vspace is very evident. Please take it into consideration.\n2. In Figure 3, it appears that the baseline method has not yet converged. Did you train for longer?\n3. In Figure 3, should the accuracy for Multitask be comparable to a single task?\n4. The computation cost seems incorrect. There would be an additional term for (T/K) for each of the K-sized modules (line 90). This would of course still be less than O(t^2)\n5. Can you use TLB with Swin Transformer?\n6. Why are you not using a bidirectional recurrent layer. Bidirectional RNNs and LSTMs are known to outperform traditional ones. It appears that except for the sequential tasks (4.2), using a bidirectional recurrent layer should improve performance.\n7. What is the time taken for training and inference as opposed to baseline methods?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept