Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: This paper proposes L2 Adaptive Computation (LAC), a mechanism to dynamically control the computational budget of deep neural networks for each input. LAC keeps track of the difference in the network's activation and decides to halt the computation when it falls below a threshold. 
strengths_and_weaknesses: Strengths 
- Adaptive computation is an interesting research topic and the authors did a fine job in introducing the problem and motivating the proposed solution
- The proposed method, LAC, is generally well-motivated, simple, and achieves encouraging results.

Weaknesses
- My major concern of this work is the unclear description of LAC. This confusion arises because the authors used $n$ to denote **both the batch size and number of computation steps**. 
- The **number of computation steps** is also not well-defined for the two applications. On the image classification task, the authors stated that LAC was used to determine the number of layers to be applied. This is quite unclear as for ViT, during the forward pass, the input/feature must be forwarded until the last layer to obtain a classification prediction. 
- The overall results seem interesting and encouraging, but the authors are strongly advised to clearly explain how LAC was applied on the two applications provided.
suggested_changes: Please see "Weaknesses"
review_title: null

final_score: rating: High Potential (HP): a submission which meets the reviewing criteria and has potential to make an impact on the field
review_content: summary_of_contributions: This paper introduces L2 Adaptive Computation (LAC) that dynamically adjusts the computation budget by tracking changes in the L2 norm of a neural network’s hidden state. It eliminates the need for additional trainable modules or auxiliary loss terms to make halting decisions and shows promising results.
strengths_and_weaknesses: Strengths:
- LAC is a simple and clear method that does not require additional training parameters and loss terms.
- The image classification results seem to be promising.
- The writing is mostly clear and easy to follow.


Weaknesses:
- For per-example and per-batch halting, the authors say "depending on the hardware and framework, either could be preferred to efficiently introduce adaptivity". Can the authors further clarify what are the pros and cons of each method and when to use which? And what is the complexity of each method?
- For the parity experiment, in table 1, the accuracies of all of the tasks are 1 even with vanilla RNN. This makes the experiments meaningless although the steps are different - cannot spot the difference before and after LAC. The authors can show the ACC of RNN with 3 steps for comparison if it's not 1.0.
- How does the performance of LAC compare with other methods that require additional parameters or losses, such as Schuster et al, 2022 and Xue et al 2023? Is it better, worse, or close?
- Clarity: (1) in section 2, the authors did not define what is $x$ in $S(x, h_n) = h_{n+1}$, (2) section 3, parity experiment, "{1or − 1, or0}", is it a typo? (3) I think the threshold can be an important hyperparameter. What is the values of it in the experiments? Are they sensitive to different experiments?
suggested_changes: - I think it's important to compare with previous methods that require additional parameters or losses, as one of the major contributions the authors claimed is simplicity. It's necessary to see the trade-off between simplicity and effectiveness.
- Maybe I missed something, but I didn't really understand Fig 1. E.g., why do the computation steps of "ViT + LAC per-example" decrease first and then increase?
- Parity experiment: see weakness.
- Can be future work: as the authors identified, it would be good to show how LAC scales to larger models/datasets.
review_title: null

Meta-review: metareview: This paper is well structured and written, and promising numerical results are well presented.
It would nevertheless benefit from more carefully-designed and well-controlled experiments.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Invite to archive