Adaptive Computation with Elastic Input Sequence
Authors: Fuzhao Xue
Valerii Likhosherstov
Anurag Arnab
Neil Houlsby
Yi Tay
Mostafa Dehghani
Yang You
Venue: Submitted to ICLR 2023 (2023)
Keywords: Adaptive computation
dynamic allocation of computation budget.

Abstract: When solving a problem, human beings have the adaptive ability in terms of the type of information they use, the procedure they take, and the amount of time they spend approaching and solving the problem. However, most standard neural networks have the same function type and fixed computation budget on different samples regardless of their nature and difficulty. Adaptivity is a powerful paradigm as it not only imbues practitioners with flexibility pertaining to the downstream usage of these models but can also serve as a powerful inductive bias for solving certain challenging classes of problems. In this work, we propose a new strategy, AdaTape, that enables dynamic computation in neural networks via adaptive tape tokens. AdaTape employs an elastic input sequence by equipping an existing architecture with a dynamic read and write tape. Specifically, we adaptively generate input sequences using tape tokens obtained from a tape bank that can either be trainable or generated from input data. We analyze the challenges and requirements to obtain dynamic sequence content and length, and propose the Adaptive Tape Reader (ATR) algorithm to achieve both objectives. Via extensive experiments on image recognition tasks, we show that AdaTape can achieve better performance while maintaining the computational cost.

Under review as a conference paper at ICLR 2023
ADAPTIVE COMPUTATION WITH ELASTIC INPUT SEQUENCE
Anonymous authors Paper under double-blind review
ABSTRACT
When solving a problem, human beings have the adaptive ability in terms of the type of information they use, the procedure they take, and the amount of time they spend approaching and solving the problem. However, most standard neural net- works have the same function type and fixed computation budget on different sam- ples regardless of their nature and difficulty, Adaptivity a powerful paradigm it not only imbues practitioners with flexibility pertaining to the downstream usage of these models but can also serve as a powerful inductive bias for soly- ing certain challenging classes of problems (Dehghani al. 2018 Banino et al., 2021). In this work, we propose a new strategy, AdaTTape, that enables dynamic computation in neural networks via adaptive tape tokens. AdaTTape employs elastic input sequence by equipping an architecture with dynamic read and write tape. Specifically, we adaptively generate input sequences using tape tokens ob- tained from tape bank that can either be trainable or generated from input data. We analyze the challenges and requirements obtain dynamic sequence content and length, and propose the Adaptive Tape Reading (ATR) algorithm to achieve both objectives. Via extensive experiments on image recognition tasks, we show that Ada Aape can achieve better performance while maintaining the computational cost.
INTRODUCTION
Adaptive computation is central to human intelligence. This is clear, given that humans spend variable amount of time and energy on different problems depending on their complexity (Meunier 2009). 2009). Adaptivity in neural networks is attractive for two key reasons. Firstly, adaptive compu- lation could potentially be a powerful and essential inductive bias for solving challenging problems that would have been significantly harder otherwise (Dehghani et al. 2018 Banino et al. (02I). Secondly, adaptive computation could imbue practitioners with downstream flexibility pertaining to the usage of these models. For the most part, altering the computation budget of model after it has been trained becomes almost impossible. Hence, the ability to flexibly scale computational costs and budgets dynamically is highly desirable.
This paper proposes AdaTTape, a new general-purposs adaptive computation method. The key idea is introduce elastic input sequences via the means of a dynamic read and write memory tape. Unlike all prior works that investigate adaptivity via sparse conditional computation (Fedus et al. 2022
2021 Lepikhin et al. 2020) or adaptivity through recursion over architecture (Dehghani al. 2018
Banino et al. [222]] Graves 2016), this work presents a new perspective that explores adaptivity with respect input sequence lnggh (or read/write memory tapes from the perspective a Neural Turing Machine (Graves et al. )2014). ) We postulate that this exploration is crucial for the development of
this class of methods and is very complementary to the existing suite of methods developed to encourage adaptive computation in neural networks.
Aaae promotes adaptivity in both type and amount of computation. Specifically, AdaTape con- trols (I) the contents of the tape tokens (2) the number of tape tokens, that are used for each input. To this end, AdaTape is characterized by a tape bank that can be dynamically read from, using newly proposed dynamic halting algorithm which
call Adaptive Tape Reading (ATR). Con- cretely, ATR method adaptively and dynamically selects the content and length of this memory tape which is appended to the inputs of standard Transformer /Vaswani al. 2017). Given that the

---

Under review as a conference paper at ICLR 2023
Tranifforee Layer IN
Transfo mer Layer
Transformer Layer 12
Tope 19M
Transformer Layer
Tope Bark
1ample
Sample?
tolkens
Tape talicnc
Tokenzzer
Adaptive Reading
Tapao Ithketa
Complet
Figure 1: An overview of AdaTTap. For different samples, we pick a variable number of different tokens from the tape bank. The tape bank can be driven from input, e.g., by extracting some extra fine-grained information or it can be a set of trainable vectors. The Adaptive Tape Reading is used recursively select different sequences of tape tokens, with variable lengths, for different inputs. These tokens are then simply appended to inputs and fed to the transformer encoder.
increasing computation budget generally leads to improved quality (Kaplan al. 2020 Dehghani
et al. 2021 HHffmann et al. [022]] Zhai et al. 2022; Abnar et al. (022)). I, this enables a new way for adaptively scaling the computation budget without adding new parameters applying part of the model recursively.
To ascertain the effectiveness of the proposed AdaTape method, we first evaluate on the chal- lenging Parity task (Graves] 2016; Banino et al. 22021), standard verification check for Adaptive Computation Time (ACT) algorithms (Graves 2016p. Our results demonstrate that AdaTape per- forms well on this problem. Meanwhile, this problem remains completely unsolvable by vanilla Transformers. This not only verifies that the AdaTape inductive bias crucial in solving certain classes of problems but also asserts its correctness. Given that the standard Transformer is touted as the true universal algorithm with ubiquitous impact across all fields (Jumper et al.) 2021 Dosovit- skiy et al., 2020 Vaswani et al. 2017), it would be unthinkable if Transformers lack the inductive bias for a standard vector parity problem.
Finally, we conduct large-scale experiments vision tasks (e.g., image recognition and evaluate their sho- accuracy), showing that AdaTape performs well and outperforms vanilla Transformers when compute-matched -machhed (Dehghani et al. (021) across both FLOPs and throughput. While AdaTape does not improve efficiency during training, the granted flexibility and adaptivity allow dynamic scaling of the computation budget during inference. Given that the standard practice is to train and serve n models to potentially cater to variable computation budgets (e.g., base models for less important workloads and large models for prioritized examples), we consider the property of having a single model flex between multiple requirements to be highly desirable.
2 ADATAPE: ADAPTIVE COMPUTATION WITH ELASTIC INPUT SEQUENCE
Neural networks can obtain adaptive ability by using different functions or variable computation budgets for different inputs. Assume a deep neural network is a function fa; 0). The output of this function relies on both input æ and parameter 0. For adaptive function types, we usually sparsely activate a subset of parameters 0 conditioned on æ. This type of adaptive ability can also named as conditional computation. Research on Mixture-of-
(Fedus et al. 2021] Lepikhin et al.) 2020 Xue et al. 2021 Lou et al., 2021 Riquelme et al.) 2021 fact introduce adaptivity on the function type via routing and specifying the computation for each input sample.
Another line adaptive computation research is the dynamic computation budget. For standard neural networks (e.g., transformer), the computation budget is fixed for different samples. However, recent studies show that adaptive computation budgets can help to solve many tasks where vanilla transformer totally fails (Dehghani et al. 2018 Banino et al. 202 Abnar et al.) 2020). Most of these works use dynamic depth to achieve adaptivity in the allocation of the computation budget. For instance, the Adaptive Computation Time (ACT) algorithm proposed by Graves (2016)
2

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Adaptive Computation Time
1: Input: Initial states S [81. .... SN ). where s.. Output states S {(",, where + 0: Initial halting score hp = O; Initial update times
Max ponder times T; Trainable linear layer g(-): Set as small number like 0.01; Initial ponder loss lae> = 0.
2: while hp < 1.0 and n < ' do
3: Compute balting score for each token at this step p (g(())
4: Compute the average halting score for each sample p mean (p)
5:
if hip + - > 1.0 e then
6:
Update remainder r ===-ht, htp
7: Update output states gout += $ break
8:
Increase update times +=
10:
Update states S = g(S)
11:
Update output states S S += S "p
12:
Increase the accumulated halting score hp +=p
lact =
13:
return lact,
for an adaptive computational budget on recurrent neural networks (Hochreiter & Schmidhuber, 1997). Universal Transformer (UT) (Dehghani al. 2018) extends the ACT algorithm trans- formers (Vaswani et al. 2017) and makes the computation budget rely on the number of transformer layers used for processing each input example or tokens. Many recent studies like PonderNet (Ban- ino et al. 2021) follow the same framework while improving the dynamic halting mechanisms.
AdaTape not only uses different function types per input via conditioning the adaptive tape read- ing mechanism on the input representation but also adjusts the computation budget by employing variable length memory tape for different inputs. Figure presents a high-level schema of AdaTape in the context of image recognition. For given batch of input images, after tokenization (e.g., linear projection of non-overlapping patches from the image the vision transformer), AdaTape uses vector representing each input to dynamically select variable-sized sequence of tape tokens from a tape bank. A tape bank could be a set of trainable vectors or generated on-the-fly -ly from inputs, c.g., by using more fine- fine-grained tokenizer. The selected tape tokens are appended to the original input and fed to the Transformer layers. More precisely, AdaTTape defines [(x: O). where ax! aTT and x being input tokens and xx" being sequence of tape tokens that is generated by fATR Zbank). When using a learnable bank, fATR in fact selects different parameters of the model (learnable tapes) per input, which means AdaTape has adaptivity in terms of the function type. Also given that the length of æ; is different for each input, AdaTape allocates different com- putation budgets for different inputs. Finally, sequence containing both input and tape tokens goes to transformer encoder. For each transformer layer, we use the same multi-head attention across all input and tape tokens. However, we use two different feed-forward networks, one applied to all tokens from the original input and the other is applied to all tape tokens. We observed slightly better quality by using separate foed-forward networks for input and tape tokens.
In the following section, we briefly introduce the original ACT algorithm and explain its contra- diction with the adaptive sequence setting. Then, to satisfy the requirements of the elastic input sequence, design the Adaptive Tape Reading algorithm, explain the different components in detail, and share the reasoning behind all the design choices.
2.1 ADAPTIVE COMPUTATION TIME
As shown in Algorithm D. ACT uses trainable linear component with sigmoid activation sigmoid(g(-)) that computes the halting score at each step. When the summation of the halting scores hp is greater than the halting threshold, the computation stops and we conduct early exit- ing to drop the future layers. At each time step, the states S are weighted by halting score p and accumulated. When the model decides end the algorithm, the summation of the halting score
Pram -E=1P: 1.0.
ACT employs loss function lact to encourage less computation. The loss lact includes two com- ponents, the number of updates n and the remainder r. The main goal of ACT is to control the computation by minimizing the number of updates n. However, such objective is not differen-
3

---

Under review as a conference paper at ICLR 2023
tiable and cannot be minimized directly via back-propagation. To address that in round-about way, ACT optimizes n together with the remainder r. When decreasing r, the summation of all used Prum Σx= Pt increases. When Σ Pt > 1.0 e, n would decrease by 1.0 thus ACT mini- mizes n indirectly. The final loss function can usually be written represents the main training objective, e.g., cross-entropy loss in the classification task. Note that the stability and performance of the model when using ACT is sensitive the ACT loss coefficient, so, using ACT requires careful tuning of X.
= mmain Mact where (main
2.2 TAPE BANK
AdaTape uses a bank of tokens where all the candidate tape tokens are stored. The adaptive tape reading mechanism independent of where are the tape tokens from and simply requires interact- ing with a R RB* tensor, where B is the size of the bank and H is the dimension of each token. We explore two different ways of creating the tape bank this paper, an input-driven bank and a learnable bank.
Input-driven Bank For an input-driven bank, we use input data as the source to generate bank tokens on-the-fly. The general idea is extract bank of tokens from the input, while employing a different approach than what the model tokenizer uses for mapping the raw input to a sequence of input tokens. This enables dynamic, on-demand access to information from the input that is obtained using different point of view, e.g., a different resolution or a different level of abstraction. For instance, for the image recognition task, when using Vision Transformers (Dosovitskiy et al., 2020). simple, yet effective idea is to use different patch sizes for generating input tokens and bank tokens. For instance, using a smaller patch size for generating the bank can be seen as dynamic multi-scale processing of the input where we can select what fine-grained information from the input is useful which is much more efficient than using all the small patches and consuming a large amount of computation. Appendix providds more details input-driven bank setup for Vision Transformer. In this paper, we use the input-driven bank in one of our image recognition experiments, however, the idea be generalized to other tasks and setups, e.g., language tasks where more fine-grained tokenization (Kudo & Richardson 2018) has shown to be effective.
Learnable bank Using a finer-grain tokenizer for generating tape bank is not always applicable. For instance, it is hard to further split each node in graph transformer (Yun et al. 2019). A different and more general approach for generating the tape bank, is to simply use a set of trainable vectors tape tokens. The learnable bank can be seen size Zbank E IR BxH One difference between an input-driven bank is, instead of building the bank on-the-fly, the content is fixed for different samples. On the other hand, selecting tape tokens from a learnable bank means partially using the parameters of the model, which is form of sparsity in AdaTapes
embedding layer with trainable tensor of
2.3 ADAPTIVE COMPUTATION TIME FOR ELASTIC INPUT SEQUENCE
In order to have elasticity the input length, need mechanism that selects variable sequence of tape tokens from the bank conditioning on the input representation. Such a mechanism needs to be able to make decision how many tape tokens should be appended the input, e.g., using halting mechanism where tape tokens are recursively added to the input. As the input representation, we can use [CLS] ] token or the average pooling of all input tokens. This input query vector used for making the decision on the number of tape tokens for each input.
One straightforward method to obtain adaptive sequence is adapting ACT algorithm explained in Section 2.1 To this end, we first summarize the requirements of ACT algorithm as follows: (1) the summation of the halting score Paum ΣT, Pt- 1.0; (2) each halting score p, is proportional to the importance of the current step output; (3) the computing process of halting score pt is required to be differentiable and can be well-trained.
However, unfortunately, all these requirements in ACT are not desirable in the adaptive sequence scenario. First, in adaptive token reading. the output state of hh step is tth tape token are going to use as the model input. For the steps that we do not want to stop, we expect the weights of the tokens to stay close 1.0 1.0. On the other hand, for the step want stop at, which means this incoming
4

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Adaptive Tape Reading
1: Input: Initial halting score h list of tokens selected states = []; Initial query q R 1x;; Tape
token bank Zbank E _ ; Token bank mask m O, where m RR" 1x where m 0; Max ponder times T: Halting threshold T; Initial ponder loss latr 0.
2: while hp < do
3: Compute the inner product of query and tape tokens d = Zhank : where d E R
4: Mask inner product d d (1.0 m))
5: Compute index of top K elements in d, set the index vector as E Ri*K where K
6: Take top K elements from d by the index vector and denote this new vector w. where R Normalize the elements by softmax: w softmax( w V
8: Take top K tape tokens from Zrank by the index vector i, and merge the tape tokens (s1, (.. SK
selected from bank one single tape token = >E Wk8k
9:
Append this token into input sequence states
10:
if hp + max > r then
11:
break
12:
Increase halting score: hp += max(w)
13:
Increase ponder loss: latr += 1.0 Et= w or loollect +
14:
Update token mask by the one-hot vector of i: m += sum((one-ho(((), axis=0)
15:
Update query q == 1 (s q) or g=s
16:
return lutr, states
tape token is not as important as others, we need a smaller weight than previous tokens. Such requirement totally contradicts requirements (1) and (2) in ACT. Additionally, the existence of layer normalization LayerNorm(-) will make trainable halting score computation layer g(-) in ACT algorithm invalid. The main reason is the normalization layer will ignore the halting score pet LayerNorm(PI: (p > LayerNorm(2)) (2.). The halting score computation layer g(-) will then cannot be well-trained. We introduce the detailed reasoning in Appendix [A.4 In summary, after analyzing the requirements of ACT algorithm and adaptive sequence length, we found there are some clear contradictions. Therefore, we devote ourselves to designing a novel dynamic halting algorithm in the following section.
2.4 ADAPTIVE TAPE READING MECHANISM
Algorithm presents a pseudo code for the Adaptive Tape Reading. ATR is an iterative process for selecting tape tokens from the bank. At each iteration, we select new set of tokens (here we select and add tape tokens one-by-one) without replacement, conditioned on the previously selected tokens. ATR uses a query vector q !R representing the input at the current iteration (i.e., the sequence of all input tokens plus already selected tape tokens) to select the next set of tokens from a tape bank Zbank R Rx H! We compute d as the inner product of q and Zbank- Note that in practice, to reduce the computation, we can use part of the q and tape tokens for computing the inner product (e.g., q(: It], and Zbank [;; h). meaning to use the first h elements). This can be seen using first h elements in and Zbank [b. h] as key and the remaining elements as value, where Zoank [b, denotes 6th tape token. To avoid the repeated selection of tape tokens, each iteration, we adjust the inner product d by masking out weights of tokens that are selected before (using the bank mask m in Algorithm 2 that gets updated each iteration). Then we select top-K tape tokens according to their scores from d and create w E IR! x containing weight logits of the selected tape tokens form. We normalize w and apply softmax. We also create the corresponding tape tokens vectors (8). ..., 8x) collected from the Zhank, and compute a single vector as the weighted average of the selected K tokens, which is the tape token that we add to the input at the current iteration.
To make the halting decision, equal to a threshold r. Larger max max(w) indicates the need for less iteration (i.c,, adding fewer tape tokens). Note that the query vector q used for selecting the tape token is updated by averaging the input query and current tape token or just replacing the input query with the current tape token.
accumulate the largest value in into hp until it is greater or
In order incentivize shorter sequences for efficiency and penalize the model for adding tape tokens when there is no need, we use a similar loss term to what the original ACT uses, i.c., Imain + Xlatr- We observe, unlike ACT, the adaptive tape reading is less sensitive the value of X. We
5

---

Under review as a conference paper at ICLR 2023
also empirically found that, when using learnable bank, we omit the latr loss term without observing a significant change in the performance.
ATR offers template for clastic input sequences. To emphasize some of the details in the above algorithms: (1) we observe better performance and stability when using a lower dimension for the
query; we found normalizing the inner product by query dimension vh before applying softmax critical following the original design of attention (Vaswani et al. 2017p: (3) we found that normal- izing the bank and query using a shared LayerNorm()) layer improves performance and stabilizes training.
3 EXPERIMENTS
We first present our experimental setup and implementation details in Section results on the parity task and image classification benchmarks in Section (.atiionn.22am and
3.1 We report the
3.3 To further val-
idate the effectiveness of our model, we conduct an ablation study in Section 3.4 Finally, Section 3.5
provides some insights into the behavior of adaptive tape reading in AdaTape.
3.1 EXPERIMENTAL SETTING
We evaluate AdaTape on both parity task and image recognition benchmarks. First, the parity task is, given sequence of numbers æ [In, ,2, NN.. where I, is 1, -1, or O, we are to predict whether the number of I's in even or odd. For the parity task, we employ an input-driven bank and use the bank as the way the model accesses the input information, and as the input token, s, we simply use a single trainable token vector (c.g., [CLS] ] token) to create the initial query for tape token reading. We train all models for 10000 steps with batch size 128. The learning rate is set as 3e-5, and we use a linear warm-up for 1000 steps. We use transformer- Tiny in this task since we cannot see improvements when scale it up. similar to the observation of UT (Dehghani et al., 2018) for algorithmic tasks. The configuration of transformer-Ting can be found in Appendix A.5
For image classification benchmarks, we first conduct large-scale pre-training on JFT-300M (Sun ee al. 2017) followed by few-shot learning on a wide range of datasets, including ImageNet (Deng al. 2009). Ci (Krizhevsky et al. 2009) and Pets (Parkhi et al. 20122) following the protocol of vanilla ViT (Dosovitskiy et al. 2020 and Big Transfer (Kolesnikov et al. 2020). That is, we use the hidden representation before logits computation as feature adapt the upstream model, and evaluate the resulting model on the validation/test set. We also train models on ImageNet from scratch for easier comparison in future work. Following existing work on ViT with adaptive com- putation (Yin et al. 2022). on ImageNet, we train models mainly at Tiny and Small scales. Please see Appendix [A.6lane and Appendix A..7 for details on hyper-parameters and training strategies used for AdaTape with a learnable bank.
We compare AdaTape with standard transformers and adaptive transformers. The standard trans- formers include ViT (Dosovitskiy et al. 2020). DeiT (Touvron et al. 2021) and Plain ViT (Beyer 2022 DeiT and Plain VIT are
et al.
models for training on ImageNet from scratch. We also compare with adaptive transformers like UT (Dehghani et al. 2018) and A A-ViT Yin
eet al. 2022). To further enhance our baselines and achieve a more comprehensive comparison, we develop two improved versions of UT our strong baselines equipped with adaptive computation. We first consider UT without parameter sharing and refer to it as Unshared Universal Transformer (U2T). We increase the maximum number of pondering steps in U2T, which means U2T has more layers, more computation and more parameters. To further enhance the baselines, we also consider stacking UT and VIT together. We found putting shallow UT on top of VIT totally failed due to an unstable training process, but putting ViT on top of UT can achieve decent performance. We refer to this model as UViT which our strongest baseline on image classification with adaptive depth. In UVIT, we employ 3 UT layers on Tiny, Small, and Base models and 6 UT layers on Large models. The number of standard transformer layers is the same as VIT.
3.2 EVALUATION ON THE PARITY TASK
We evaluate AdeTape on the parity task to study the effect of the inductive biases that it introduces to vanilla Transformer on synthetic task. The Parity is the simplest non-counter-free, periodic

---

Under review as a conference paper at ICLR 2023
Parity Task Comparison
100
90
Percision (%)
80
Random
Universal Transformer
AdaTape
2
Input Length (Difficulty)
Figure 2: Evaluation the parity task. The standard transformer and universal transformer totally
failed on this task, both showing performance at the level of a random guessing baseline.
Qual Comparison with Adaptive Baselines on ImageNet
UVIT-5/16
AVT--5/14
78
Accuracy
A-TT-//24(OOU)
76-
U2T
.UVT-Ti/16
U2T-5/16
A-VIT
74
Adaa Addaape--T/11
A-ViT(Durs)
A-VIF-S/16(Ours)
AdaTape
72
AVVI--//16
U2TT TV16
350
300
250
200
Img/Sec/Core
Figure 3: We evaluate AdeT by training ImageNet from scratch. For A- ViT, we not only report their results from the paper but also re- re-implement A V-V by training from scratch, i.e., A-ViT(Ours).
regular language (Bhattamishra al., 2020). Simple recurrent neural networks solve this task
well because the memory in the recurrent neural network can record the states for finite-state au-
tomation (Abnar et al., 2021 Schwarzschild et al.. 2021| Veličković al. 2022 Ibarz al. 2022
Bansal et al.) (022). Standard transformer totally failed in modeling such sequences (Hahn 2020
as they are incapable of directly maintaining a counter. Specifically, in AdaTape, we can use the tape tokens to record the behavior in each recurrence step. We first use the input sequence to generate the bank. The Adaptive Tape Reading algorithm plays the role to model the parity task recurrently. Considering the target is predict the number of in the input sequence is even or odd, there are actually two states in the finite-state automation. We then fix the K as 2 in Algorithm to check whether there is state switching after each step. When K is fixed as 2, the maximum ponder time T=2r= 2. where N is the length of input parity sequence. Such syper-parameter setting is to match the period in parity and to ensure AdaTape can collect all required tokens to make the prediction.
Tigure[showw the results the parity task. We can both the standard transformer and universal transformer completely fail on the parity task, even if they are trained and evaluated on a very short (simple) sequence. However, the AdaTape performs much better than all baselines. One reason is that AdaTape incorporates the recurrence within the model input selection and such an inductive bias could be of crucial for solving the parity task, as it is a way to implicitly enables AdaTape to maintain a counter, which is not possible in the standard transformer.
3.3 EVALUATION ON IMAGE CLASSIFICATION
We pre-train AdaTape on JFT-300M and report few-shot results on popular image recognition bench- marks in Table I We can see U2T performs worse than other models, we suggest the reason is the unstable training loss when making new halting decisions. The advanced bascline, UViT performs better than VIT because of adaptivity, more computation, and parameters. Our AdaTape is using less computation than UVIT and performs better at scales. AdaTTape with an input-driven bank is superior at larger scale. We suggest larger scale model is better at finding informative tokens from input and validate this reasoning in Section 3.5 We can also observe AdaTape is not very good at throughput. Both types of AdaTTap are not hardware friendly as baselines. This is the limitation of AdaTape. However, please note this only means the training speed is slightly slower on TPU, which is highly optimized by large-scale matrix multiplication. Due to less real computation (GFLOPs) and dynamic sequence length, AdaTape has the potential to speed up inference on other hardware smaller batch size.
7

---

Under review as a conference paper at ICLR 2023
Table 1: Pre-training and transfer learning results on image classification benchmarks. We report two AdaTape models with different bank types at each scale. BeeTTapeeB32-Leeee means using a learnable bank for AdaTape-B/33 B/32. Ta--e-//222-Ipuu denotes we are using an input-driven bank. We report both GFLOPs and throughput (measured in images second core). For all models with adaptive computation budget, we report the upper bound computation cost. The pre-training is conducted on JFT-300M dataset and we report precision @ (%) on the validation dataset. The few- shot experiments are on ImageNet, Cifar100 and Pets datasets with Top-1 accuracy. IN25 denotes
the result on ImageNet 2h-shot.
Model
GFLOPs
Throughput
JFT-300M
IN10
IN25
Cifar100
PetS1o
ViT-B/32
4.437
793.8
43.3
59.4
62.5
72.2
90.3
U2T-B/32
5,513
343.4
38.8
53.6
64.4
84.1
UVIT-B/32
5,511
478.7
44.4
61.9
64.6
74.4
91.6
BeTTTapeBBB32-Lee
5.585
431.8
44.4
63.0
65.8
75.0
93.2
6.057
342.3
44.7
63.8
65.9
75.5
93.0
VIT-B/16
17.634
253.7
48.6
67.4
70.0
77.5
93.0
U2T-B/16
22.016
102.9
46.6
63.0
66.0
73.1
92.5
UViT-B/16
22.011
151.7
49.0
68.5
71.0
75.8
94.0
AdsTape-BBl16LLam
B/16-Leam
18.837
167.1
49.1
70.3
72.6
78.7
94.0
AdaTape-B/16-Ing
19,304
148.8
48.9
69.0
71.5
77.4
94.4
ViT-L/32
15.628
192.3
49.9
69.5
72.1
79.4
94.0
UViT-L/32
19.242
127.1
50.5
70.5
72.6
80.1
94.1
AdaTape-L/32-Lear
19.497
108.6
50.2
70.8
73.3
79.6
95.0
AdaTape-LL/32IIpuu
20.141
95.5
50.2
71.8
73.8
81.3
95.0
VIT-L/16 61.724
63.1
54.3
75.2
77.0
83.1
95.9
UVIT-L/16
77.114
44.7
54.8
75.5
77.4
81.6
95.7
AdsTape-L.116-Laam
65.941
44.2
54.7
76.5
78.0
82.7
96.7
Add/Tape-1/16-Input
66.570
40.1
54.8
76.7
78.5
84.7
96.4
We also evaluate AdaTape by training ImageNet-IK from scratch. For A-VIT (Yin et al. 2022 authors fine-tuned their model to obtain adaptivity from a pre-trained ViT checkpoint, which has not only report their results
different setting with ours. For straightforward comparison, from the paper but also re-implement A-VIT to reproduce the results when training from scratch. The quality-cost curve is shown in 3 We can see AdaTTape performs much better than the baselines in terms of quality-cost balance. For efficiency, AdaTape-S/1 S/16 is even faster than the Tiny- level baselines. Such results match well with the finding from Dehghani al. 2021). which shows that the adaptive model depth architectures are not well suited for many accelerators like TPU. AdaTape is much more efficient compared to other adaptive baselines because we are injecting adaptivity into the input sequence instead of model depth. For effectiveness, AdaTTpe can even outperform the non-adaptive models, VIT and DeiT, with comparable computation costs. More detailed results can be found in Appendix A.8
3.4 ABLATION STUDY
We first ablate the adaptive abilities of AdaTape including adaptive sequence content and adaptive sequence length. For adaptive sequence content, we expect it can improve effectiveness. For the adaptive sequence length, it has the potential save computation during inference, we expect AdaTape can keep comparable accuracy when we use the fixed sequence length.
Adaptive sequence length from ATR algorithm with a recurrent token selection process. The adaptive sequence content is mainly from a selective use of the tape bank. To ablate the model by removing only the "adaptivity in length", we the halting threshold infinite high and always select T tokens for different samples, which means all samples use the maximum sequence length and the same amount computation costs. Note that models without adaptive sequence length use the upper bound of computation used by the adaptive ones. Different from adaptive sequence length, removing only the "adaptivity of content" is not possible, because our ATR algorithm at the end of the day uses a tape bank (which is inherently adaptive in content). Therefore, remove the "adaptivity length and content" together by appending a fixed set trainable tokens to every input sample.
8

---

Under review as a conference paper at ICLR 2023
Table Ablation study on the adaptive abilities of Adae We use AdaTape with an input-driven bank as a platform. For the model without adaptive length, pick ' tape tokens in parallel. For the model without adaptive content and adaptive length, we remove the tape bank and use a fixed set of trainable tokens enhance the input. We also report average/max sequence length and sequence length variance.
Model
JFT-300M
IN 10-shot
IN 25-shot
Avg/Max SeqLen
Var SeqL.en
Ada adaTape-B/33
63.8
65.9
57.4/59.0
4.8
w/o Ada Length
44.2
63.3
66.4
59.0/59.0
0.0
w/o Ada Length&Conten
43.9
61.0
64.1
59.0/59.0
0.0
AdaTape-B/16
48.9
69.0
71.5
203.4/206.0
7.3
w/o Ada Length
49.2
69.4
71.8
206.0/206.0
0.0
AdaTapc-L/32
50.2
71.8
73.8
56.8/59.0
8.5
w/o Ada Length
50.5
71.7
74.2
59.0/59.0
0.0
AdaTape-L/16
54.8
76.7
78.5
203.0/206.0
10.4
w/o Ada Length
54.5
76.7
78.4
206.0/206.0
0.0
Yindex
10
20
20
10 20
100
20
X-index
X-nndex
Figure 4: We visualize the tape token selection heatmap of AdaTape-B/31 (left) and AdaTape-B/16 (right). The hotter color means the patch this position more frequently selected.
Results are shown in Table 2 We can see, without the adaptive content, there is a significant perfor- mance drop. For instance, compared with Ada Tape without adaptive sequence only, Ada Tape-B/332 without adaptive content degrades 2.3 points on ImageNet 10-shot accuracy. This shows the effec- tiveness of adaptive sequence content. When remove the adaptive sequence length, models perform comparably instead of much better at all scales, which shows the tape tokens se- lected are condensed and make full of limited input tokens. Another interesting finding is, when we scale the model up, we can observe the increasing sequence length variance. This indicates larger AdaTape is better controlling sequence length and computation cost. We also conduct the ablation study on the hyper-parametert computation cost, and the number of trainable parameters in Appendix A.6 a.1 A.11 respectively.
3.5
VISUALIZATION
In this section, we study the behavior of AdaTape by visualization. First, collect the token selec- tion results of AdaTTape with an input-driven bank on JFT-300M validation set, and visualize them heatmaps in Figure 4 We colors). This matches well with our prior knowledge, central patches are usually more informative. This shows AdaTape prefers more informative patches improve performance. We also visualize the token selection distribution. Please see Appendix A.12 for details.
the central patches more frequently picked (with lighter
CONCLUSION
We propose Adae novel strategy and a general-purpose adaptive computation method. AdaT- ape is characterized by elastic sequence lengths generated from a newly proposed Adaptive Tape Reading mechanism. This new inductive bias enables Ada'Tape to have the potential to solve chal- lenging tasks that standard transformers and existing adaptive architecture transformer struggle Via extensive experiments on image recognition benchmarks, we show that AdaTape outperforms standard transformers and adaptive architecture transformers compute-matched setup.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Samira Abnar, Mostafa Dehghani, and Willem Zuidema. Transferring inductive biases through knowledge distillation. arXiv preprint arXiv:2006. 00555, 2020.
Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. arXiv preprint arXiv:2 (0t:://00.020955 2021.
Nikita Balagansky and Daniil Gavrilov.
Palbert: Teaching albert to ponder. arXiv preprint
arXiv:2204.088 03276, 2022.
Andrea Banino, Jan Balaguer, and Charles Blundell. Pondernet: Learning to ponder. arXiv preprint arXiv:2107.05407, 2021.
Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-end algorithm synthesis with recurrent networks: Logical extrapola- tion without overthinking. arXiv preprint arXiv:2202.05826, 2022.
Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain vit baselines for imagenet- k. arXiv preprint arXiv:2205.0/580 2022.
Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. On the ability and limitations of transformers to recognize formal languages. arXiv preprint arXiv:2009. / 1264, 2020.
Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V Sapunov. Memory transformer. arXiv preprint arXiv:2006. 115000 2020.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and łukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03873 2018.
Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, and Yi Tay. The efficiency mis- nomer. arXiv preprint arXiv:21 10..:2100../2... /2894, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scall hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 245.255. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint 2xXiv:0010.11922 2020.
Mohsen Fayyaz, Soroush Abbasi Koohpayegani, Farnoush Rezaei, and Sommerladel Hamed Pirsi- avash2 Juergen Gall. Adaptive token sampling for efficient vision transformers. ECCV, 2022.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.
William Fedus, Jeff Dean, and Barret Zoph. A review of sparse expert models in deep learning. arXiv preprint axX1v:2009.01667, 2022.
Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint 100.016603.0883.. 2016.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv:1410.544 2014.
arXiv preprint
Michael Hahn. Theoretical limitations of self-attention neural sequence models. Transactions of the Association for Computational Linguistics, 8:156-171, 2020.
10

---

Under review as a conference paper at ICLR 2023
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv: 155522 2022.
Borja Ibarz, Vitaly Kurin, George Papamakarios, Kyriacos Nikiforou, Mehdi Bennani, Róbert Csordás, Andrew Dudzik, Matko Bošnjak, Alex Vitvitskyi, Yulia Rubanova, et al. general- ist neural algorithmic learner. arXiv preprint arXiv:2209. 1/142 2022.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-5555 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXhv:2001.0836/ 2020.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European confer- ence on computer vision, pp. 491-507. Springer, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Taku Kudo and John Richardson. SentencePiece: A simple and language independent sub- word tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Con- ference on Empirical Methods Natural Language Processing: System Demonstrations, pp. 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 1..18653/v1/D18 2012.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning. arXiv preprint arXiv:2104.0862) 2021.
Yuxuan Lou, Fuzhao Xue, Zangwei Zheng, and Yang You. Sparse-mlp: A fully-mip architecture with conditional computation. arXiv e-prints, pp. arXiv 2109,, 2021.
Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser- Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12309-12318, 2022.
David Meunier, Renaud Lambiotte, Alex Fornito, Karen Ersche, and Edward Bullmore. Hier- archical modularity in human brain functional networks. Frontiers in neuroinformatics, 3:37, 2009.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 859555838595 2021.
Tal Schuster, Adam Fisch, Tommi Jaakkola, and Regina Barzilay. Consistent accelcrated inference via confident adaptive transformers. arXiv preprint arXiv:2 088000 2021.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q Tran, Tay, and Donald Metzler. Confident adaptive language modeling. arXiv preprint arXiv:2207. 070600, 2022.
11

---

Under review as a conference paper at ICLR 2023
Roy Schwartz, Gabi Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A. Smith. The right tool for the job: Matching model and instance complexities. In Proc. of ACL. 2020.
Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn algorithm? generalizing from easy to hard problems with recurrent networks, 2021.
Hwanjun Song, Deqing Sun, Sanghyuk Chun, Varun Jampani, Dongyoon Han, Byeongho Heo, Wonjae Kim, and Ming-Hsuan Hssan Yang. Vidt: An efficient and effective fully transformer-based object detector. arXiv preprint arXiv:2110.03921 2021.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843 852, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826 2016.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Petar Veličković. Adrià Puigdomènech Badia, David Budden, Razvan Pascanu, Andrea Banino, Misha Dashevskiy, Raia Hadsell, and Charles Blundell. The clrs algorithmic reasoning bench- mark. arXiv preprint arXiv:2205.. 15659, 2022.
Yulin Wang. Rui Huang, Shiji Song, Zeyi Huang, and Gao Huang. Not all images worth 16x16 words: Dynamic transformers for efficient image recognition. Advances in Neural Information Processing Systems, 34:1 1960-11975 2021.
Fuzhao Xue, Ziji Shi, Futao Wei, Yuxuan Lou, Yong Liu, and Yang You. Go wider instead of deeper. arXiv preprint arXiv:2107.11817, 2021.
Hongxu Yin, Arash Vahdat, Jose Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-ViT: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph trans- former networks. Advances in neural information processing systems, 32, 2019.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 12104-1211 12113, 2022.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 17000:1710 2017.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 RELATED WORK
A.1.1 DYNAMIC HALTING ALGORITIM
2016 was first proposed for dynamic halt- ing in recurrent neural networks. Dehghani et al. (2018) adapted this algorithm to transformer-bast models and proposed universal transformer. Universal transformer with shared trainable parame- ters across transformer blocks has adaptive depth for different samples, and this work shows that the adaptivity enables the transformer to solve some tasks that the vanilla transformer totally failed. Yin et al. (2022) adapted ACT algorithm to the vision transformer along the depth. However, the ACseb transformer training is not as stable as the vanilla transformer and it needs careful hyper- parameters tuning. To overcome this weakness, PonderNet (Banino et al. 2021) improves the uni- versal transformer by reformulating the halting policy as a probabilistic model. Balagansky & Gavrilov 2022) further stabilizes this training process by removing the sampling process in Ponder- Net. Another way to halt adaptively is confidence-based dynamic halting algorithms. [Schuster et al.)
Adaptive Computation Time (ACT) algorithm Graves
2021 2022 Schwartz et al. (2020) investigated the confidence-basee dynamic halting algorithms on adaptive model depth.
Our work also proposed a dynamic halting algorithm, but are interested in pondering input sequence length instead of model depth. Another difference is that we have no trainable layers to compute the halting score, which is requirement of adaptive transformer input. Alternatively, use the entropy of logits from the token selection layer as an indicator to make a halting decision.
A.1.2 ADAPTIVE SEQUENCE LENGTH
Adaptive sequence length can be achieved by token pruming. For instance, Meng et al. the patch tokens in the vision transformer via a light-weight decision network, and Fayyaz et al. 2022) drop the patch tokens by sampling adaptively. Our AdaTTape is different from their work as AdaTape appends a small number of tokens to the original input adaptively instead of pruning the intermediate token representations within the model. Also, the content of the tokens in the existing work is not adaptive. The appended tokens in AdaTape are sparsely selected from the tape bank, which is helpful to improve the effectiveness of the transformer.
(0222) prune
Wang et al. ( proposed another way to achieve adaptive sequence length for ViT. The proposed model uses large patch size first and decreases the patch size adaptively for different samples. This is similar to our input-driven bank. However, first, in our input-driven bank, instead of using all smaller patches, we use only a subset of the fine-grained patches adaptively. In addition, instead of limiting the setup to image-only inputs, AdaTape supports learnable bank and formulates this problem in a more general way.
A.1.3 EXTRA TOKENS
Extra tokens are another important component of AdaTTape. Song et al.
(202)) use extra special introduces memory tokens
token for vision munsformer-based object detection. Burtsev et al. (2020 to augment transformer capacity. Using learnable prompt tokens NLP (Lester et al. 202T can be also perceived as adding extra tokens to the input. In all these works, the number of added tokens is fixed across different samples and often uses deterministic set of tokens (imagine having more than one [CLS] token). However, AdaTTape selects and appends a variable number of tokens from the bank adaptively per sample. Such adaptive input sequence provides not only a larger capacity but also a flexible computation budget.
A.2 INPUT-DRIVEN BANK DETAILS
We introduce more details about the input-driven bank of AdaTape. We generate the input-driven
bank by:
2 Bank (x(()((X)) + Epos)
13

---

Under review as a conference paper at ICLR 2023
where Xp is the input sequence with fine-grained tokenization (e.g., smaller patch size for ViT), h and h2 are both trainable linear projection, pos is position embedding. Since Zbank is conditioned on the input sample, need generate the content of the bank on-the-fly. This is also the reason why an input-driven bank is slightly more computationally expensive than a learnable bank.
A.3 SCALING WITH PATCH SIZE
Scaling VIT with Patch Size
62.5
V/T-5/B
60.0
Vit Mit
VIT5
57.5
55.0
52.5
VVTT-5228
50.0
VVT-5/332
47.5
10a
10
GFLOPs
Figure 5:
We scale VIT with patch size. The model is pre-trained on JFT-300M (Sun et al. 2017 We report the few-shot performance on ImageNet (Deng et al. 2009)
We fix all other parameters and scale VIT with smaller patch size to check the effectiveness of smaller patches. We experiment with VIT-S which will not become expensive as the sequence length grows. For example, training ViT-B/8 with data parallelism on 512 TPUv3 cores has the OOM issue. In 5i when adding more computation, although there is significant accuracy improvement, but more patches introduce very expensive computation costs. Such a trade-off inspires us to use smaller patches the source to generate bank for a efficient fine-grained data modeling.
A.4 LAYERNORM IS MAKING ACT INVAILD IN ADAPTIVE SEQUENCE
We argue that naively applying layer normalization the transformer invalidates the ACT algorithm in AdaTape. To justify this, we first assume are using transformer with the following standard transformer architecture:
X' MSA(LayerNorm((X +
X" IFN(LaayeNorr((X + X'
where MSA(-) (-) is multi-head attention layer and FFN(.) is feed-forward network. We are going replace X by Pi where Pt is the halting score of 1 tape token z,. Then, the output of first LayerNorm(-) should be:
P.z: H -h= PIZLA
LayerNorm(p::z))
+ /
(4)
EH (Dizi, Σ/ p2
since is very small constant, it reasonable to write the equation above as:
LayerNorm(pizz)
Σm= (21, 2イ *7 9 LayerNorm(z))
(5)
We can see Pt is ignored during the normalization, which means the value of px cannot change the output of the first LayerNorm.() in practice. Since p. is the output of the trainable layer g(.) in ACT, the 9(-) cannot be really trained well by back-propagation. To alleviate this issue, one possible solution is applying the weight p. to tape tokens after normalization layers. This is a simple and straightforward solution that may make sense. However, empirically, observe the weights of all tokens will increase to 1.0 very fast in practice and the model will then always only append one token even if we do not any loss function penalize longer sequence. We suggest the reason is
14

---

Under review as a conference paper at ICLR 2023
that uniformed token mean and variance are highly desired by attention layers. Anyhow, this result not expected because there is no adaptive ability observed. In addition, we observed the training is also extremely unstable under this design.
In summary, we cannot apply scalar weight to the tape token, so we cannot have a trainable linear layer to compute the halting score p. That makes the ACT-based dynamic halting mechanism, in- cluding PonderNet, not applicable to AdaTTape. We, therefore, are required to design a new adaptive computation algorithm for elastic input sequence, i.e., Adaptive Tape Reading. Such a reasoning process can also be extended to other future conditional and adaptive computation work. For in- stance, if you want feed the layer norm with one token after applying weight on the token, you need to be careful about where is the weight from. If this weight is from a trainable layer like ACT, we must check whether this layer can be well-trained via reasoning.
A.5 CONFIGURATION FOR TRANSFORMER- TINY
Table 3: Transformer configuration for all tiny-level models.
Transformer- Tiny
Depth Hidden Dimension
12
196
MLP Dimension
768
#Attention Heads
3
We use the tiny configuration for all models on the parity task. As shown in Table the Tiny level transformer still uses 12 layers, which is the same the transformer base. The difference that tiny configuration has smaller hidden dimensions and fewer attention heads.
A.6
Table 4: Hyper-parameters for AdaTape image classification
AdaTape-Leam
AdaTape-Input
Max ponder times T
10
10
Halting threshold T
2.0
2.0 (B) /1.0
Loss weight A
0
0.01
Bank Size C
10000
784
On JFT-300M pre-training, follow Dosovitskiy al. (2020) and train all models for 7 epochs. We use the same learning rate, batch size, and learning schedule. Customized hyper-parameters for AdaTape are summarized in Table We employed the fixed max ponder times for all models. Smaller 7 on AdaTape-L with an input-driven bank. The bank size is 10000 for AdaTappe-lean.. We use bank size 784 for AdaTape Input as we set patch size x24x 224 resolution. AdaTape with a learnable bank can be trained without halting loss. Also, note that we append tape tokens after first transformer encoder layer for better query quality and tape selection.
generate tokens from images with
For ImageNet training from scratch, we summarized the data augmentation and corresponding hyper-parameters in Table Similar with existing work (Beyer et al. 2022), we used Mixup (Zhang al. 2017), RandAug (Cubuk et al. 2020) and label smoothing (Szegedy et al. 2016 to improve the robustness.
A.7 TRICKS OF LEARNABLE BANK TRAINING
We found the training of Taae with learnable bank is relatively unstable. To alleviate this, propose two tricks to improve the training process. The core idea of these two tricks is to improve the diversity of tape tokens and encourage the model to explore the tape bank. We first add noise
15

---

Under review as a conference paper at ICLR 2023
Table 5: Hyper-parameter when training on ImageNet-IK only. Ti, S and B denote Tiny, Small, and Base scales.
Name
Value
Learning Rate
0.001
Linear Warmup Steps
10000
Learning Rate Decay
Cosine Decay
Optimizer
Adam
(31,33 .32)
(0.9, 0.999)
Weight Decay
le-4(Ti), 8c-5(S&B)
Epoch
300
Batch Size
1024
Mixup
0.2(Ti), 0,5(S&B)
Label Smoothing
O(Ti), (SI(BB))
RandAug
(2,10)(TT), (2,1))(S&B)
to query q + À where is sampled from standard normal distribution and À is the weight of noise. We À 0.01 during training. We also mask subset of the tape tokens in the bank randomly. To implement this, we initialize m = 0 + b for ATR algorithm, where E RIxC and be Bernoulli(p). We set p as 0.1 by default. We also observed that larger p can further improve the training stability.
A.8 MORE RESULTS ON IMAGENET
Table 6: Results of training on ImageNet-IK only. We use the input-dirven bank for AdaTape. denotes that we approximate the throughput by the models with similar architecture and input pipeline. For A- ViT, we not only report their results from the paper but also re-implement A-VIT by training from scratch, i.e., A (Ours).
Model
Adaptive
#Param
Throughput
ImageNet Top-1 (%)
ViT-Ti/16
5.7
387.5
58.7
DeiT-TV/16
5.7
381.8*
71.3
PITinVTT-TT/166
5.7
381.8
73.0
U2T-Ti/16
6.1
364.3
70,2
A-ViT-Ti/16
226.6*
71.0
A-ViT-Ti/16(Ours)
V
5.7
226.6
73.2
Ad/T
V
6.3
380.9
73.6
VIT-S/16
22.0
385.7
75.2
DeiT-S/16
22.0
365.8*
78.9
Plain
22.0
365.8
79.2
U2T-S/16
23.8
163.2
74.5
A-ViT-S/16
22.0
166.6*
78.6
A-ViT-S/16(Ours)
22.0
166.6
77.0
Ad/Tape-S/16
24.3
366.5
79.5
Plain V
87.1
159.9
79.5
AdaTape-B/16
94,9
130.5
80.8
We train with ImageNet-1K only and summarized the results in Table We can see AdaTTape outperforms all adaptive baselines by a large margin. Even compared to hothly-optimized baselines without adaptivity like Plain (Beyer et al. 2022) and DeiT (Touvron al. 2021). AdaTape can still surpass them with a comparable computation budget. For instance, Ada Tape-S/16 S/16 uses only 0.4x training cost and 0.3x parameters but achieves almost comparable results with Plain We may also note that Tiny scale models cannot achieve much higher throughput than Small scale models on ImageNet. We suggest the reason is that the efficiency bottlenecks are mainly from data loading and preprocessing instead of the computation budget within neural networks.
16

---

Under review as a conference paper at ICLR 2023
63.4
10.Shot
63.3
63.1
63.0
62.9
62.8
2 Number of Tape Takens
Figure 6: We sweep the number of tape tokens over {5, 10, 20, 40). Considering more tape tokens mean much more computation cost, 10 our default choice.
A.9 ABLATION ON HYPER-PARAMETIE
Number of tape tokens We investigate the effect of the number of tape tokens in this section. In the model with adaptive sequence length, the number of tape tokens is controlled by the model adaptively. To control the real length directly, we use the AdaTape without adaptive length as platform. We sweep the number of tape tokens T over {5, 10, 20, 40) and summarize the results in 6 We can see an obvious improvement when we increase the T from 5 to 10. However, the model saturated after that. Since using longer sequence means more computation budget, we select to use 10 tape tokens as the default choice.
63.0
ImagoNet 10 Shot
62.8
62.6
62.4
62.2
10a
10a
Bank Size
Figure 7:
We sweep the bank size over (le+2, let3, let4, le+5} for AdaTape with learnable bank, and found le+4 is the sweep point.
Bank Size We also sweep the bank size over {1e+2, le+3, le+4, le+5} for AdaTape with learnable bank. As shown in F the AdaTape performs best when we have tape tokens in the bank. As we fix the number of recurrences 10 in ATR algorithm by default, larger bank will only increase the computation cost linearly.
A. ABLATION ON COMPUTATION COST
Although Ada only increases the computation cost slightly, to further verify the improvement is from more reasonable design and adaptive abilities instead of more computation, we conduct ablation experiments on computation cost. First, as shown in Figure g even if smaller patch size can improve ViT. AdaTape is still outperforming ViT with less computation. In addition, we report the quality-cost comparison in terms of throughput in Figure We can see AdaTape can outperform VIT significantly with smaller latency.
A.11 ABLATION ON NUMBER OF TRAINABLE PARAMETERS
Since we have two FENs in every transformer block, AdaTape has more trainable parameters than VIT. To validate that the improvement is not just caused by having more parameters, increase the
17

---

Under review as a conference paper at ICLR 2023
Cost of Base models
Quality-Cost Large models
A ape B/16
Ada 11p
70
76
VTT-U/4
Imageteel 10-shot (%6
AdaTape-8
VV-11/14
AdaTape-L
VIT-L16
68
VVTT-8/16
ImageNeel to-hot
66
74
64
AdaTapp-8/32
72
AdaTape-U/52
62
VIT.BB/28.
60
VVTTBB332
VIT--U32
10
20
GFLOPS
GFLOPS
Figure 8: Ablation Study on the quality-cost trade-off. We scale the standard transformer (i.e. ViT)
smaller patch size, e.g., ViT-B/14 and ViT-I/14, and compare it with AdaTape.
Quality- Comparison
AdaTape--U16
76
VIT-L/14
AdaTape
VIT-L/16
ImageNet 10 shot
74
72
AdaTase B/16
70-
VTT-8/14
250
200
150
100
50
Figure 9: report the quality-cost comparison in terms of throughput to verify that AdaTape is better than baselines with comparable computation cost.
trainable parameters in ViT by adding more FFN layers to process different tokens. Then, ViT has 3 FFNs in total. The first FFN used to handle [CLS] token. The second FFN and the third FFN fed by half of the patch tokens, respectively. The results are summarized in Table 7 We can Ada Tape outperforms ViT with FFNs using less computation and fewer parameters. For instance, Ada AdaTape-BBl Tape-B/16 surpasses ViT-B/14-3FFN by 1.3% in terms of top-1 accuracy on ImageNet 10-shot.
A. VISUALIZATION OF TOKEN SELECTION DISTRIBUTION
Token Selection Distribution
Token Selection Distribution
5000
Frequency
4000
Frequency
4500
3500
4000
3500
3000
3000
2500
2500
2000
2000
200
400
600
800
200
400
600
800
Token Index
Token Index
Figure 10: We visualize the tape token selection distribution in AdaTape-B/32 (left) and AdaTape- B/16 (right). The index id is sorted by the value of the selected frequency
18

---

Under review as a conference paper at ICLR 2023
Table 7: Ablation study the number of trainable parameters.
Model
GFLOPs
Throughput
#Param
IN 10-shot
ViT-B/28
5.730
661.1
100.9
61.9
VII-B/28-3FFN
5.744
517.0
200.1
62.4
AdaTape-B/32
5.585
431.8
185.6
63.0
ViT-B/14
23.254
155.1
99.7
68.4
ViT-B/14-3FFN
23.239
148.8
198.9
69.0
AdaTape-B/16
18.837
167.1
192.5
70.3
we collect the token selection results JFT-300M validation set. We sort the tape token index by the frequency and visualize the token selection distribution Figure [0] We can observe the token selection decision obey long-tail distribution. That shows our AdaTape prefers the tape tokens at some specific positions, which is similar to our observation in Figure Le., central patches are frequently selected.
Similar to Section 3.5
19


Reviews:
[{"review_id": "tVAkk65IJ4", "reviewer": "Reviewer_pZjZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Paper proposes a new method called AdaTape that enables adaptive computation method. It is applied to vision transformers to achieve dynamic sequence length that will make compute adaptive to the input image. First, a set of learnable or predefined tokens are stored in the Tape bank, then Adaptive tape reader queries tape tokens from it. The method is inspired and built upon ACT proposed for RNNs. The AdaTape is tested on image classification task.", "summary_of_the_review": "Paper attacks a n important problem of enabling dynamic compute in neural networks. The method is novel and technically sounds. However, have main results on not public dataset limits its contributions. The absence of comparisons to other dynamic compute methods for CNNs/Transformers makes it hard to understand the impact. The code is not available and reproduction of results is difficult. Hopefully, these limitations will be addressed during the rebuttal, I am happy to reconsider my score. ", "strength_and_weaknesses": "Strength:\n\n- The motivation of the paper is great. Compute should be adaptive with respect to the complexity of input sequence. \n- The method, AdaTape seems to be novel and differences to ACT are clearly mentioned and sufficient. \n- Analysis of the effect of LayerNorm on ACT is insightful.\n- Large scale image classification task results are great to test methodology. \n\nWeaknesses:\n- Comparison to other adaptive networks/methods is missing.\n- Main results are presented with the model trained on JFT-300M that is not a public dataset (to my knowledge) and will make comparison in the future hard. Running main experiments on ImageNet-1K (not few shot) will make the work more valuable for future comparisons. \n- Not clear if the code will be available for reproducibility. \n- More clarification will be helpful for the points below.\n\n\nQuestions:\n- How to make sure that elements in $q$ are sorted by importance? Selecting the first $h$ implies that elements are ranked. Algorithm . \n- Can the compute be controlled by user? For example, during the inference compute is adapted to the current hardware load. Probably changing the stoping criteria will work.\n- Why not to do selection of input tokens instead?  \n- In 2.1 authors mention that ACT uses a linear component $q$ to compute halting score. I am not sure it is linear as it usually has sigmoid at the output to limit the score in 0..1. \n- There works that enable dynamic compute for neural networks. For example, [1], ConvNet-AIG, Dynamic-ViT[3], Avit[2], Adavit[5]. Even more papers are mentioned in [5]. Comparing to these works will help to understand the contribution better. \n\nReferences:\n\n[1] Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. In CVPR, 2017.\n\n[2] Yin, Hongxu, et al. \"A-ViT: Adaptive Tokens for Efficient Vision Transformer.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Rao, Yongming, et al. \"Dynamicvit: Efficient vision transformers with dynamic token sparsification.\" Advances in neural information processing systems 34 (2021): 13937-13949.\n\n[4] Veit, Andreas, and Serge Belongie. \"Convolutional networks with adaptive inference graphs.\" Proceedings of the European Conference on Computer Vision (ECCV). 2018.\n\n[5] Meng, Lingchen, et al. \"AdaViT: Adaptive Vision Transformers for Efficient Image Recognition.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[6] Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey. TPAMI, 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "is9XuSih9k", "reviewer": "Reviewer_V6XW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors investigate the problem of adaptive computation time and propose AdaTape, which can adaptively select different function types and computation budgets for different inputs. It incorporates an adaptive tape reader (ATR) to learn or extract variable-length tape tokens for dynamic computation in neural networks. The authors conducted experiments on the parity task and the image classification task. Experimental results show that the proposed AdaTape algorithm outperforms several baselines such as ViT, Universal Transformers, and their variants, while maintaining the same training efficiency.", "summary_of_the_review": "The writing (except the introduction section) needs to be significantly improved to enhance readability. It is recommended to add more clarification on the purpose of the tape token and compare with other possible methods that can achieve the same goal. There is no separate related work section. It may be helpful to add that section.", "strength_and_weaknesses": "Strength:\n1. The proposed AdaTape not only adaptively selects different parameters to handle different inputs, but also adjusts the computational budget by using variable-length storage tapes.\n2. The introductory part is well written and engaging.\n\nWeaknesses: \n1. There are quite a few grammatical errors starting from Section 2.2 that make the paper difficult to understand.\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "deu12VA7RP", "reviewer": "Reviewer_vn6P", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a novel way to add adaptivity to transformer models. By storing or generating a bank of tokens for each input, the input can be adaptively extended with a variable number of tokens. The method is compared to transformers without adaptivitiy as well as the Universal Transformer. The benchmarks for comparison include computing parity of bit stings as well as image classification. ", "summary_of_the_review": "I find this paper well below the threshold for acceptance. The experiments seem out of tune with the related work. Although the exact results support the very narrow claim that this model is better than one existing model, the contribution to the field is limited by the scope of comparison.", "strength_and_weaknesses": "This paper proposes a memory bank that can be queried adaptively for a variable number of tokens to append to the input sequence. As a way to add adaptivity to existing neural architectures, this is an alternative to recurrence or weight sharing. There is novelty in this choice, although it isn't adequately contextualized among the existing adaptive methods (listed below). Additionally, the choice of benchmarks is quite odd. While the first task considered, parity, is popular in some early adaptive compute papers, it strikes me as outdated. The latest work -- largely branded as algorithmic reasoning, but with the same focus -- includes methods that can compute prefix sums of binary strings (arguably more work and including the parity task) of many more bits than considered here (listed below). The second task is image classification which is not considered in most other adaptive compute/algorithmic reasoning papers. \n\nIf the goal of this work is to study, explore, and improve adaptivity in neural networks, then the choice of benchmarks and competing models is lacking. If the goal is to augment transformer architectures to improve their performance it aught to be better framed in the text. \n\nStrengths: \n- Creative and novel method for added adaptivity to transformer models\n\nWeaknesses: \n- Missing related work \n    - Adpative methods that use recurrence and solve algorithmic tasks like parity:\n        - [1] Schwarzschild, Avi, et al. \"Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks.\" Advances in Neural Information Processing Systems 34 (2021): 6695-6706.\n        - [2] Veličković, Petar, et al. \"The CLRS Algorithmic Reasoning Benchmark.\" arXiv preprint arXiv:2205.15659 (2022).\n        - [3] Ibarz, Borja, et al. \"A Generalist Neural Algorithmic Learner.\" arXiv preprint arXiv:2209.11142 (2022).\n    - Adaptive model that use recurrence can compute prefix sums (and therefore parity) of inputs with 512 bits. \n        - [4] Bansal, Arpit et al. End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking. Neural Information Processing Systems (NeurIPS), 2022. \n- Some minor writing issues (not affecting score, only a few examples here): \n    - Middle of Page 4: The following excerpt is not full sentences and has incorrect punctuation. \"For instance, using a smaller patch size for generating the bank can be seen as dynamic\nmulti-scale processing of the input where we can select what fine-grained information from the input\nis useful which is much more efficient than using all the small patches. , which would consume a\nlarge amount of computation.\" \n    - Bottom of Page 4: \"The of-the-shelve ACT ...\" should probably be \"...off-the-shelf...\".\n    - Bottom of Page 4: \"For instance, as shown in algorithm 1...\" where \"algorithm\" should be capitalized.\"\n    - Bottom of Page 5: \"In order to incentives shorter sequences...\" should probably be \"...incentivise...\".\n    - The Appendix has bad references to Figures that are not there.\n\n\n___\n\nIn their reply, the authors addressed all my concerns. I have changed my score accordingly.\n \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DV0LQV-dFxS", "reviewer": "Reviewer_rSKm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel method to add \"on demand\" computation to transformers by selecting additional tokens from a token bank. In particular the authors use a variation of the adaptive computation time algorithm used in universal transformers to add additional tokens in the input sequence. Experiments with image classification shows that the proposed method can indeed make use of the extra computation to improve the classification results.", "summary_of_the_review": "The paper seems a great relatively novel idea, however the experimental evaluation is lacking. The comparisons show that using the proposed method is actually significantly slower albeit better in terms of final accuracy. However, not using the adaptive tape reading algorithm actually performs better and would be possibly faster in modern accelerators.", "strength_and_weaknesses": "Strengths\n-----------\n\n- Adaptive computation is a very interesting and important subject, both in terms of saving computation by only computing as much as it is necessary as well as mimicking the human thought process of considering a problem again and again and collecting more information.\n- Adding extra tokens from a token bank is a very intuitive way to inject variable sized pieces of information to a transformer model.\n\nWeaknesses\n---------------\n\n- AdaTape is significantly slower than the baselines often being 2 times slower. This significantly reduces the usefulness of the method since the baselines with a few extra layers would probably perform as well.\n- The experimental evaluation is not very clearly written. In particular the information about finetuning the datasets except for JFT-300 is missing.\n- The usefulness of the adaptive computation abilities of AdaTape is put in question from the experiments in section 3.4 . In particular we see that in most cases, not using the adaptive length algorithm actually yields better results. It would also be significantly faster on GPUs and TPUs.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject