Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes Confident Adaptive Language Modeling (CALM) to early-exit dynamically during the decoding process. The authors provide a theoretical guarantee that local confidence measures could lead to the confident generation of global outputs.
strengths_and_weaknesses: CALM is practically useful to speed up the inference where autoregressive decoding with large language models is quite expensive. The method is principled based on the learn-then-test framework.
questions: It is surprising that the oracle early exiting only uses less than 2 layers on average. How is it possible? 

There is still a large room between the oracle and the proposed methods. What are the potential directions to reduce this gap? Which assumptions can be the most easily relaxed for better efficiency?

I wonder whether the observed patterns in Figure 2 will repeat on any tasks.

Could you explain more about why the softmax-based measure gives the highest speedup compared to other measures?

limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces the "Confident Adaptive Language Modeling" (CALM) framework, which contains very detailed study about the source of errors of doing early exiting during the decoding process of Transformer based language model, and proposes a method to calibrate well about local and global predictions to overcome these errors. Several empirical results, on text summarization, machine translation and Squad demonstrated the usefulness of the proposed method.


strengths_and_weaknesses: Strengths:

- This paper is technically very rigorous. It tackles the problem in a step-by-step manner with clear very sound logic chain. Also, it is not a common practice now to pay attention to the statistical testing criteria and this paper does a good job here, by adopting a principled way to address that.  The writing, including the mathematical notation system, is also very accurate. 

- The technical works, especially in section 3.3,  are very detailed and (hence) insightful. For example, the headroom analysis of state propagation and local errors are comprehensive and convincing. 

Weakness:

- See the below question 2 in the "Questions" part: it seems good to discuss whether the technical proposal in section 4, to solve this problem, is really necessary to be so complicated. 
questions: 1. Would adding more softmax operators for each layer, for example listed in Equation (6), trigger more computational bottleneck? Understand that seemingly the trade-off here would be good since we can ignore a lot of full computations for the subsequent layers if we can early exit, but still Eqn (6) seems introducing the extra softmax operators for all the layers and all the timesteps, during training, and it would be good if there could be some ablation study here demonstrating the ROI;

2. It is definitely technically sound that the LTT framework is used. But this seems to be a double-sided sword: to determine the \lambda value in Eqn (5), is it really necessary to adopt such a complicated framework? Note supposedly doing complicated tech works might hurt the generalization of the proposed method in the real world application scenario. What if this problem is simply treated as a hyper-parameter selection one and use much simpler method to do so (e.g., some random search and early stopping methods used in AutoML)? 
limitations: See above. N/A for the "negative societal impact of their work" part.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: UPDATE: The authors have answered my concerned so I increased my score to an 8. 

Transformer language models such as T5 are the most important tool in the current NLP world. Typically the decoder has a set number of layers and all of these layers are executed for each token. In the past few years we've seen many papers that do 'early exit' for tokens for which the output is easy to compute- during these timesteps, the model exists early and does not compute all the layers. 

This paper first analyzes these types of model with an oracle to show what the best case speedups could be, and then proposes a few new ways to do early exit, for the T5 model, evaluated on a group of interesting downstream tasks such as summarization and translation. 

I felt like the paper wasn't very clear in comparing the final performance to the many prior works such as Elbayad et al or Schwartz et al. In fact Table 2 which is the main table that presents results only contains the 3 proposed models, it does not present the full model (no early exit) baseline and it does not present prior models such as Elbayad et al or Schwartz et al.

In addition, while these models have been proposed for the past couple of years, none of them have been adopted or used in production. I feel like this might be because of many issues with these methods that none of these papers really discuss. For example, you point out the "potential FLOP speedup", but what is the *actual* wall clock speedups of your methods? When doing batched inference, these methods can't really be used because every token in the batch might require a different amount of compute. It would be good to discuss these potential issues in the next version of the paper. 
strengths_and_weaknesses: Strengths:
1) The analysis in this paper is super interesting, the oracle analysis shows that these methods might have big potential.

Weaknesses:
1) No wallclock time. Sure- you might save some compute by not computing some layers. But you are also using more compute to decide when to stop running the layers. So what is the total running time? Providing wallclock times is the only way to show if these methods are actually efficient or not.
2) Can this method be used in batched inference? Is there any downside to that?
3) Table 2 should have comparisons with the full (no early-exit) model baseline and with models from previous papers. It's not clear where you stand in comparison to the prior literature. 

questions: My questions are in the prior sections. The most important question is how would table 2 look if you added numbers from the full (no early-exit) model baseline and from the Elbayad and Schwartz papers. 
limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: **The research problem:** The authors study the problem of accelerating transformer-based autoregressive inference from the perspective of early-exit strategies - approaches that terminate the forward pass before the final layer is reached based on whether a confidence criterion is satisfied. The primary goal of the proposed framework, CALM, is to find a method that allows for exiting the forward passes as early as possible while guaranteeing performance bounds on the consistency between the full model and early-exit model. 

**Analysis and proposed method:** After analyzing the sources of error that arise when running early-exit models, the authors propose a decaying confidence threshold for early exit that allow for earlier exits closer to the end of sequences. 

**Theoretical guarantees on performance:** The authors use tools from multiple hypothesis testing literature to pick confidence thresholds that provably satisfy performance bounds. 

**Empirical Justification:** The proposed method is shown to (with some caveats) result in potential speedup of up to 3x. 

**POST REBUTTAL UPDATE:** I thank the authors for their responses. I've raised my score. 

strengths_and_weaknesses: **STRENGTHS**
* Provable performance bounds is appealing: The distribution free approach to proving performance bounds is appealing. The fact that this approach doesn't yield too-conservative bounds is promising. 
* Approach considers local and global performance metrics: The proposed method considers the deterioration of quality both locally and globally when one employs early-exit strategies. 
* Error analysis provides insight about transformer inner-workings: Experiments in Section 3.3 are quite interesting - that one can get away with using only 1.53 layers per forward pass on average in a 8-layer transformer while (almost) not incurring a significant performance deterioration is surprising. 
* No labelled data is needed: The proposed method, when using textual consistency, doesn't require any labelled data. 
* Comparison of different confidence measures provides insight into early-exit strategies: The comparisons between softmax, state and classifier based early-exit confidence measures is quite interesting 
* Clear writing: The paper is easy to follow and well written, with very few typos. 


**WEAKNESSES**
* Comparison to baselines: While the contribution of this work is to facilitate early-exit while providing performance guarantees, it'd still be interesting to see comparisons with baselines to see how much (if any) cost one has to pay to obtain the provable guarantees. 
* Computational cost of some confidence measures: The best performing confidence measure (softmax-based one) is computationally heavy, potentially erasing the efficiency gains of early exit. (The authors do note that further parallelization can alleviate this issue, and other confidence measures also bring nontrivial improvements in efficiency)
* Practical benefit of performance guarantees in early-exit strategies: I believe the authors could spend some time justifying why one would be interested in provable guarantees on early-exit performance deterioration. What are some applications where this type of guarantee is desired/required? We already don't expect to have performance guarantees on the full model - what's the marginal benefit of using an early exit strategy that have theoretical guarantees instead of simply doing empirical validation to pick $\lambda$? This will probably be especially true in cases where full model (i.e. model without early-exit) performance is sub-optimal which will be the case for many difficult NLP tasks with even today's strongest models). (It's possible I'm missing something here). 
questions: * Could you elaborate on the meaning of the result reported on line 166? Am I right in thinking that if one uses only the first layer representations to attend to during decoding (as if all the previous tokens exited after the first layer), the performance basically doesn't take a hit? This seems highly counter-intuitive? 
* Does it ever happen that the early-exit prediction matches that of the full model at layer i < L, then starts disagreeing again at layer j where  i < j < L?
* Line 201: "Perfect local consistency implies global consistency". Could you clarify what is meant here? 



Nitpick: 
* $\mathcal{Y}$ doesn't seem to be defined anywhere. 
* Line 194: "Our goal" instead of "are goal"
limitations: The authors are upfront about the computational cost of using the softmax-based confidence measure (the best performing metric explored in the paper), which could potentially surpass the gains of the proposed method.
review_title: null

Meta-review: metareview: This paper studies the error of early exit in decoding Transformer Language models, and proposes a method CALM to calibrate and accelerate model inference. Experiments on a variety of tasks (summarization, MT, QA) show effectiveness of the proposed method. 
All reviewers find the paper solid and the author feedback convincing. 
confidence: Certain
Decision: Accept