Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel method to add "on demand" computation to transformers by selecting additional tokens from a token bank. In particular the authors use a variation of the adaptive computation time algorithm used in universal transformers to add additional tokens in the input sequence. Experiments with image classification shows that the proposed method can indeed make use of the extra computation to improve the classification results.
strength_and_weaknesses: Strengths
-----------

- Adaptive computation is a very interesting and important subject, both in terms of saving computation by only computing as much as it is necessary as well as mimicking the human thought process of considering a problem again and again and collecting more information.
- Adding extra tokens from a token bank is a very intuitive way to inject variable sized pieces of information to a transformer model.

Weaknesses
---------------

- AdaTape is significantly slower than the baselines often being 2 times slower. This significantly reduces the usefulness of the method since the baselines with a few extra layers would probably perform as well.
- The experimental evaluation is not very clearly written. In particular the information about finetuning the datasets except for JFT-300 is missing.
- The usefulness of the adaptive computation abilities of AdaTape is put in question from the experiments in section 3.4 . In particular we see that in most cases, not using the adaptive length algorithm actually yields better results. It would also be significantly faster on GPUs and TPUs.
clarity,_quality,_novelty_and_reproducibility: The clarity of the paper can be improved. In particular, the experimental evaluation should contain more details regarding the training and evaluation on the datasets other than JFT-300M.
summary_of_the_review: The paper seems a great relatively novel idea, however the experimental evaluation is lacking. The comparisons show that using the proposed method is actually significantly slower albeit better in terms of final accuracy. However, not using the adaptive tape reading algorithm actually performs better and would be possibly faster in modern accelerators.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors investigate the problem of adaptive computation time and propose AdaTape, which can adaptively select different function types and computation budgets for different inputs. It incorporates an adaptive tape reader (ATR) to learn or extract variable-length tape tokens for dynamic computation in neural networks. The authors conducted experiments on the parity task and the image classification task. Experimental results show that the proposed AdaTape algorithm outperforms several baselines such as ViT, Universal Transformers, and their variants, while maintaining the same training efficiency.
strength_and_weaknesses: Strength:
1. The proposed AdaTape not only adaptively selects different parameters to handle different inputs, but also adjusts the computational budget by using variable-length storage tapes.
2. The introductory part is well written and engaging.

Weaknesses: 
1. There are quite a few grammatical errors starting from Section 2.2 that make the paper difficult to understand.
 
clarity,_quality,_novelty_and_reproducibility: There are quite a few grammatical errors starting from Section 2.2 that make the paper difficult to understand.

summary_of_the_review: The writing (except the introduction section) needs to be significantly improved to enhance readability. It is recommended to add more clarification on the purpose of the tape token and compare with other possible methods that can achieve the same goal. There is no separate related work section. It may be helpful to add that section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Paper proposes a new method called AdaTape that enables adaptive computation method. It is applied to vision transformers to achieve dynamic sequence length that will make compute adaptive to the input image. First, a set of learnable or predefined tokens are stored in the Tape bank, then Adaptive tape reader queries tape tokens from it. The method is inspired and built upon ACT proposed for RNNs. The AdaTape is tested on image classification task.
strength_and_weaknesses: Strength:

- The motivation of the paper is great. Compute should be adaptive with respect to the complexity of input sequence. 
- The method, AdaTape seems to be novel and differences to ACT are clearly mentioned and sufficient. 
- Analysis of the effect of LayerNorm on ACT is insightful.
- Large scale image classification task results are great to test methodology. 

Weaknesses:
- Comparison to other adaptive networks/methods is missing.
- Main results are presented with the model trained on JFT-300M that is not a public dataset (to my knowledge) and will make comparison in the future hard. Running main experiments on ImageNet-1K (not few shot) will make the work more valuable for future comparisons. 
- Not clear if the code will be available for reproducibility. 
- More clarification will be helpful for the points below.


Questions:
- How to make sure that elements in $q$ are sorted by importance? Selecting the first $h$ implies that elements are ranked. Algorithm . 
- Can the compute be controlled by user? For example, during the inference compute is adapted to the current hardware load. Probably changing the stoping criteria will work.
- Why not to do selection of input tokens instead?  
- In 2.1 authors mention that ACT uses a linear component $q$ to compute halting score. I am not sure it is linear as it usually has sigmoid at the output to limit the score in 0..1. 
- There works that enable dynamic compute for neural networks. For example, [1], ConvNet-AIG, Dynamic-ViT[3], Avit[2], Adavit[5]. Even more papers are mentioned in [5]. Comparing to these works will help to understand the contribution better. 

References:

[1] Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. In CVPR, 2017.

[2] Yin, Hongxu, et al. "A-ViT: Adaptive Tokens for Efficient Vision Transformer." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[3] Rao, Yongming, et al. "Dynamicvit: Efficient vision transformers with dynamic token sparsification." Advances in neural information processing systems 34 (2021): 13937-13949.

[4] Veit, Andreas, and Serge Belongie. "Convolutional networks with adaptive inference graphs." Proceedings of the European Conference on Computer Vision (ECCV). 2018.

[5] Meng, Lingchen, et al. "AdaViT: Adaptive Vision Transformers for Efficient Image Recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[6] Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey. TPAMI, 2021.

clarity,_quality,_novelty_and_reproducibility: Quality - great
Clarity - mostly clear, however, understanding Algorithm 1 is not easy, having a diagram will help
Novelty - great
summary_of_the_review: Paper attacks a n important problem of enabling dynamic compute in neural networks. The method is novel and technically sounds. However, have main results on not public dataset limits its contributions. The absence of comparisons to other dynamic compute methods for CNNs/Transformers makes it hard to understand the impact. The code is not available and reproduction of results is difficult. Hopefully, these limitations will be addressed during the rebuttal, I am happy to reconsider my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper describes a novel way to add adaptivity to transformer models. By storing or generating a bank of tokens for each input, the input can be adaptively extended with a variable number of tokens. The method is compared to transformers without adaptivitiy as well as the Universal Transformer. The benchmarks for comparison include computing parity of bit stings as well as image classification. 
strength_and_weaknesses: This paper proposes a memory bank that can be queried adaptively for a variable number of tokens to append to the input sequence. As a way to add adaptivity to existing neural architectures, this is an alternative to recurrence or weight sharing. There is novelty in this choice, although it isn't adequately contextualized among the existing adaptive methods (listed below). Additionally, the choice of benchmarks is quite odd. While the first task considered, parity, is popular in some early adaptive compute papers, it strikes me as outdated. The latest work -- largely branded as algorithmic reasoning, but with the same focus -- includes methods that can compute prefix sums of binary strings (arguably more work and including the parity task) of many more bits than considered here (listed below). The second task is image classification which is not considered in most other adaptive compute/algorithmic reasoning papers. 

If the goal of this work is to study, explore, and improve adaptivity in neural networks, then the choice of benchmarks and competing models is lacking. If the goal is to augment transformer architectures to improve their performance it aught to be better framed in the text. 

Strengths: 
- Creative and novel method for added adaptivity to transformer models

Weaknesses: 
- Missing related work 
    - Adpative methods that use recurrence and solve algorithmic tasks like parity:
        - [1] Schwarzschild, Avi, et al. "Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks." Advances in Neural Information Processing Systems 34 (2021): 6695-6706.
        - [2] Veličković, Petar, et al. "The CLRS Algorithmic Reasoning Benchmark." arXiv preprint arXiv:2205.15659 (2022).
        - [3] Ibarz, Borja, et al. "A Generalist Neural Algorithmic Learner." arXiv preprint arXiv:2209.11142 (2022).
    - Adaptive model that use recurrence can compute prefix sums (and therefore parity) of inputs with 512 bits. 
        - [4] Bansal, Arpit et al. End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking. Neural Information Processing Systems (NeurIPS), 2022. 
- Some minor writing issues (not affecting score, only a few examples here): 
    - Middle of Page 4: The following excerpt is not full sentences and has incorrect punctuation. "For instance, using a smaller patch size for generating the bank can be seen as dynamic
multi-scale processing of the input where we can select what fine-grained information from the input
is useful which is much more efficient than using all the small patches. , which would consume a
large amount of computation." 
    - Bottom of Page 4: "The of-the-shelve ACT ..." should probably be "...off-the-shelf...".
    - Bottom of Page 4: "For instance, as shown in algorithm 1..." where "algorithm" should be capitalized."
    - Bottom of Page 5: "In order to incentives shorter sequences..." should probably be "...incentivise...".
    - The Appendix has bad references to Figures that are not there.


___

In their reply, the authors addressed all my concerns. I have changed my score accordingly.
 

clarity,_quality,_novelty_and_reproducibility: The paper is clear, but contains many typos/grammatical errors. The methods are original, however there is no code or promise to release code. The details in the text might make reproducing the findings difficult, although I did not try to reproduce them myself.
summary_of_the_review: I find this paper well below the threshold for acceptance. The experiments seem out of tune with the related work. Although the exact results support the very narrow claim that this model is better than one existing model, the contribution to the field is limited by the scope of comparison.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new method AdaTape that enables adaptive computation in Transformers. Specifically, the method constructs a token bank with a set of learnable/predefined tokens, and selects tokens from the token bank with an adaptive tape reader to achieve dynamic sequence length according to the input sequences. The method is applied to vision Transformer for image classification and shows improved performance without added computation time. The work is well motivated. The method is sound. Reviewers had concerns on the lack of sufficient comparison with existing different adaptive methods, and the writing that would need more polishing and clarity on experimentation and empirical conclusions.
justification_for_why_not_higher_score: Reviewers had concerns on the lack of sufficient comparison with existing different adaptive methods, and the writing that would need more polishing and clarity on experimentation and empirical conclusions.
justification_for_why_not_lower_score:  The work is well motivated. The method is sound. Results show improved accuracy with the same training efficiency compared to certain baselines.
Decision: Reject