Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes an improvement over PonderNet, an early-exit method that can help overcome long inference times and "overthinking" in a neural network. The specific improvement is the change of exit layer selection during inference-time, using a deterministic criterion instead of the probabilistic method proposed originally in PonderNet. The paper compares the improvements to both PonderNet (partially) and PABEE, a heuristic based early-exit method. The results are presented using ALBERT and a variety of GLUE tasks. Overall, the work shows that there are small but consistent gains over earlier methods, with a 1.2-1.3x speedup in inference compared to vanilla ALBERT.


strengths_and_weaknesses: Strengths

Consistent gains over previous methods 
Logical improvements over previous work (using two layers to study change across them) 
Evaluation of the method itself is done well (ablation studies for the sub-component across a few tasks, speed comparison with one other method)

Weaknesses

Paper is not very clearly written, and required several passes to fully grasp. 
It seems like the PonderNet results were added after the fact (e.g. they are missing from the speed vs performance graph, missing test set results)
The work does not shed much light on why PALBERT works better, and seems like a more empirically motivated and studied method. While there is nothing wrong in this in an itself, the improvements themselves are fairly incremental over PonderNet, and may not surpass the base quality of this particular venue.


questions: Table 1 shows the overall variance of accuracies across several tasks. While the variance of PonderNet is slightly higher than PALBERT, it does not justify the claim that there is "major variance" (L11, L45) in PonderNet's outputs. Do you have some quantitative evidence for this claim?

Line 159 mentions that the baselines were re-implemented, but PonderNet results are missing from the test set section in Table 2. What is the reason for these missing results?
Figure 3's caption says that the results are for two tasks, while the results for 4 tasks are presented, please fix this.

The paper repeatedly mentions that PALBERT "significantly outperforms" or "outperforms by a large margin", while the actual overall improvements are not qualified by this. Consider simplifying the language and/or presenting more evidence to support this claim.
L160 "Ablation" -> "Appendix"
limitations: Authors have addressed limitations adequately.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: Previous work found that the computation of a deep neural network can exit at earlier layers to prevent "overthinking". This improves upon previous method (PonderNet) of performing an early exit and proposes a new deterministic Q-exit criterion. This new method determines the exit layer index in a deterministic way by using the CDF of the layer exit distribution. Experiments of language understanding tasks (GLUE) and demonstrate improved performance over PonderNet.
strengths_and_weaknesses: **Strengths:**
1. The paper is relatively well written and proposes a simple and effective method to alleviate the high-bias issue of sampling based layer exit method.
2. Comprehensive analysis have been conducted to show the effects of the hyperparameter of threshold, the speedup and the distribution of examples that exit at different layers.

**Weaknesses:**
1. The original PonderNet paper which this paper is based on is evaluated on more complex tasks, e.g. QA and other multi-step reasoning tasks, which I find a more appropriate setup for evaluating this layer-exit approach, which is a better test bed for these approaches. With that being said, experiments on the GLUE task alone is not sufficient to demonstrate the effectiveness of the proposed method. Moreover, as discussed in the limitation section, the proposed method still lags behind the Albert model on some tasks, I think this paper still needs further improvements for a better contribution.
2.  This paper uses Albert model as the base model for executing the proposed method which is a limitation of the proposed method. I was wondering if there is a specific reason for the authors not to use other pre-trained models, especially deeper networks, to implement the idea. If this is because Albert uses shared weights for layers, then is the proposed method a model-specific trick but not a generic method?
3. The threshold p that decides when to exit seems to be a very important hyperparameter for the success of the proposed method and it is selected with the validation set. This directly contrasts with selecting the best exiting layer from 1 to 12 layers as a baseline. This simple baseline is not included in the paper. Could you also compare with that?
4. A suggestion on the presentation of the paper: it's better to describe Albert and PonderNet more clearly in a background section to make the paper more clear to the general audience.
questions: The questions are asked in the weaknesses section.
limitations: N/A.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper aims to improve the inference time efficiency of ALBERT, a pretrained large transformer, through the early exit. In particular, a dedicated binary classifier is trained to decide, at each layer, whether or not to use the current layer’s output as the final representation and skip further forward propagation. If the model exits at an early layer, significant efficiency gains can be achieved. This work heavily builds on a previous work along this line—the PonderNet—and improves its test-time exit criterion: instead of drawing from a Bernoulli, this work proposes to accumulate the early-exit classifier’s output probability throughout the layers and determine the early exit by comparing it to a prespecified threshold. The motivation is to reduce the variance. This method is combined with other techniques aiming to improve the capacity of the early-exit classifier. The model is evaluated on the GLUE benchmark, with extensive ablation study, tradeoff curves, and other analyses. The proposed model outperforms previous early-exit methods designed for ALBERT, and improves the efficiency with little or no accuracy drop.

I find the idea very intuitive and clearly presented. The detailed analysis and honest results are a big plus. However, I do have concerns about the technical contributions. It seems that most of the proposals heavily build on previous works and are rather engineering. Maybe the authors could help me and clarify their technical contributions in the response. Further, the method is presented in a way that can only be used in ALBERT. Exploring its applications in other pretrained models could improve its impact.

strengths_and_weaknesses: 
Strengths:
- Very clear writing
- Very detailed and informative analysis of the results.
- The method is intuitive.
- Honest and detailed discussion of limitations

Weaknesses:
- Early exit methods have been explored in other large pretrained models (e.g., [1]). Applying Ponder ALBERT in other pretrained models could improve its impact.
- The 1.29/1.26x speedup without sacrificing accuracy is marginal.

[1] https://arxiv.org/abs/2004.07453
questions: - It may happen that different instances in a mini-batch exit at different layers. How is this resolved? Would it affect the efficiency?
- The paper argues that a major drawback of PonderNet is that it has a higher variance. But the std is barely larger than ALBERT in Table 1. Could the authors comment on this? Also, it would be nice to include std too in Table 2.
- It would be interesting to see how the exit layers might correlate with the inputs’ lengths, labels, or some other metrics measuring their difficulty.
- Have the authors compared to a handcrafted exit strategy? One can dedicate that the model must exit after X layers to achieve the 1.29x speedup, and compare its accuracy with learned exit strategies.
limitations: The authors have discussed the limitations in detail. I don't have any further concerns about the potential negative societal impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper focuses on early exit mechanism for encoder-only Transformer models. Specifically, it uses the ALBERT architecture that is similar to BERT but shares parameters across all layers (12 layers for base size). 

The authors adopt the PonderNet framework of Banino et al. that learns early-exit "lambda" classifiers for each layer, and suggest two modifications to it. First, instead of sampling from the geometric-like distribution that PonderNet emulates to decide if to exit, they compute the empirical CDF of the layers so far, and exit if the value is greater than some defined threshold q. Second, they change the module that computes lambda to be slightly larger and to use the last two hidden states instead of just the current one, and modify the learning rate.

Experiments are completed on the GLUE tasks, where PonderNet and PABEE (exiting if several consecutive layers predict the same) are used as baselines (authors reimplemented these baeslines to use the same ALBERT backbone).  Also, a grid-search over threshold values for q and ablation studies on hyper-parameters are reported.
strengths_and_weaknesses: Strengths:

1. The paper deals with an important and impactful task of improving the efficiency of pretrained Transformer-based NLP classifiers
2. Empirical results suggests that the proposed modifications to PonderNet could be effective, though I have concerns on the evaluation and comparisons (see below)
3. The writing is overall clear and easy to follow, though there are several unsupported claims (see below)
4. The ablation studies could help future studies

Weaknesses:


1. The authors claim that other methods are heuristics, but the suggested method is also a heuristic with no theoretical grounding (as the authors admit in section 5). In fact, sampling from the geometric-like distribution seems to me more grounded as it aims to simulate the probability of a "good exit" exactly at the current layer, compared to the CDF that computes the probability that one of the layers so far has been a "good exit" layer, but the prediction is taken from the current layer, so the chance of an earlier exit being good is not relevant anymore. Perhaps the authors can clarify this?
2. The authors main motivation for using CDF instead of sampling from the geometric-like distribution is having a deterministic exiting rule (e.g. see bolded span lines 39, and lines 118, 126-127). However, I wasn't convinced (1) why determinism is better, and (2) why the CDF is needed for achieving determinism. Assuming I had a calibrated predictor [1], I don't see a particular issue with sampling as the training and evaluation are anyway in expectation. Also, why not make a deterministic threshold-based decision directly on the lambda values like in [2]. Though, to obtain theoretical-backed guarantees on the marginal performance of the model, there should be some MHT correction, or other calibration process for the threshold value like in [3].
3. The main evaluation in table 2 presents a different tradeoff point between the models. The accuracy is bolded as being better, but the speedup for PALBERT is lower (also, how is the speedup computed is missing). It is impossible to compare the models this way as it is unclear how does the performance compare when tuned for the same speedup or vice versa. For example, figure 3 shows at least for SST-2 several points where PABEE is better than PALBERT, but in table 2 PALBERT is reported with higher accuracy.
4. No details on how the reported speedup value is computed. Does it take into account the computation of the lambda function (that is now larger)?

[1] On Calibration of Modern Neural Networks

[2] Depth-Adaptive Transformer

[3] Consistent Accelerated Inference via Confident Adaptive Transformers

**update post author response**: I appreciate the author's detailed response. I've decided to slightly increase my score. I'm still not entirely convinced about the motivation behind the "deterministic" exiting rule. But more importantly, I think that a table with only accuracies is insufficient for evaluating such early exiting models since there is some tradeoff with the actual speedup of each model. According to figure 3, the gains over PABEE are inconsistent across tasks.

questions: See points above, also:
* Why PonderNet in table 2 doesn't have test results?
* How were the hyper-parameters and threshold decided for each model in table 2?
limitations: I don't see potential negative societal impact of this work.

Limitations are described in the last section.
review_title: null

Meta-review: metareview: This is an interesting paper which improved during discussion and the authors in my opinion addressed many of the suggestions and concerns from all of the reviewers (two reviewers who engaged after significant discussion were supportive of acceptance).  I personally think that the paper will be an interesting addition to neurips but could be stronger if experiments were performed beyond just the GLUE/SuperGLUE setup and considered other complex inference tasks such as question answering (e.g. natural questions).
confidence: Less certain
Decision: Accept