Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The focus of this work is on so-called early exit methods in deep neural network training and in particular on increased efficiency facilitated by the authors' proposal to connect internal classifiers which sit at the crux of early exit approaches. The proposed ensemble based method is shown to improve on the accuracy/inference time trade-off.
main_review: The problem considered is topical and the authors' approach is methodologically sound, well justified, and convincing.  All of the theoretical aspects of the submission look correct to me.  The empirical part of the paper is reasonably comprehensive as well; I do not think that the authors can be faulted here.  The paper is also well written.  As far as the key contributions go, from the broad idea itself to the introduced concept of hindsight improvability, there are interesting things to be found here, no doubt, but as I have to judge every paper not merely on its merits but on its merits relative to the pool of submissions (or expected pool of submission, given that as a single reviewer I have limited view) I would say that the overall value of this paper is around the borderline, perhaps slightly below it.  Hence my recommendation. 

However, what would in my mind increase the impact of the work and make it more convincing, thus changing my recommendation towards acceptance, is the replication of the findings on data sets were time is really of crucial importance.  Time saving in general is good, of course, but in many applications it is neither here nor there - one just takes more time.  However, in some domains the times involved are really seriously limiting. This could for example be on medical images of the kind used in pathology and which are huge.  Demonstrating the ability to save time in this context, or equivalently, to use more training data within specified time, would be of major value.  Another example is that of various astronomical images.

--- Round 2
I am grateful to the authors for directly addressing my concerns. Their response does indeed make me feel more positive (or rather, even more since I liked the key ideas of the work already) towards the submission.  HI have also read with interest other reviewers' comments and the corresponding authors' responses, and in all cases I found that the authors made a good case for their work, clarifying points which may have needed clarification or elaboration.  Hence, I have upgraded my recommendation.
limitations_and_societal_impact: n/a
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes Zero Time Waste (ZTW), a novel approach to leverage previous intermediate classifiers (ICs) for early-exit neural networks. ZTW consists of two simple extensions of early-exits NNs. The first is to add cascaded connections from the result of previous ICs to the next IC. The second is to ensemble previous results for confidence measurement. The author conducts extensive experiments on image classification benchmarks and shows improved performance compared to vanilla early-exit neural networks and PABEE, a previous work investigating using results from previous ICs for confidence measurement.
main_review: - The paper is well written. The proposed method and experiments are clearly described.

- The paper conducts both experiments on supervised learning and reinforcement learning. This is good given most previous works only consider the supervised setting.

- The idea of ZTW is simple and intuitive, which is fine. And the empirical performance of ZTW seems to be promising. However, given the fact that PABEE already exploited the idea of using the results of previous ICs and their approach can be viewed as a voting-based ensemble scheme, the ensembling technique proposed in ZTW seems to be a direct extension to logits average based ensemble. The discussion about PABEE states that PABEE does not work well on these image classification settings, this is probably because the criterion in PABEE is tailored for NLP applications where the number of output classes is small. For the first technical contribution, I actually wonder why this would help a lot. Since the intermediate representation fed in the previous ICs is passed to the next ICs through residual connection, and the intermediate classifiers are only very small MLPs, the information gain by cascade connection seems to be marginal.

- The paper introduces two novel techniques but does not perform an ablation study to show the relative improvement yielded by each of them. This prevents readers from getting a comprehensive understanding of the proposed method. This kind of analysis is crucial and I strongly recommend adding this.

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a Zero Time Waste (ZTW) method with cascading and ensembling components to speed up neural network inference. The ZTW model attaches internal classifiers (ICs) to pre-trained models, and the model is evaluated in both image classification benchmarks (CIFAR-10, 100, mini-ImageNet) and reinforcement learning benchmarks (Atari-games). 

main_review: 1.  Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?

This paper studies the efficient inference task, which is not a new task. However, the analysis of information loss and wasted computation operations can be an interesting perspective. 

The proposed method follows the line of work in conditional computation and attaches independent classifiers to the network to enable early exit during inference.  As the authors stated in the paper (abstract & related work), this has been done in the literature. The claimed novelty comes from the combination of cascading and ensembling, and the proposed method also applies to pre-trained models (lines 98-100 in Related work).  However, I'm a bit concerned about the claimed novelty. 

Model cascade and early exit have rich literature which is not fully discussed in the paper.  For example,  IDK Cascades (wang et al.) attaches IDK classifiers to pre-trained models to form model cascades, claiming that "none of these methods are designed to work with pre-trained models" in lines 99-100 too bold.  Also, the authors listed a few related works in line 80 but did not make a clear contrast.  For example, the authors cite the SACT[29] in line 80 as "skipping intermediate layers," which is inaccurate.  SACT uses halting scores to enable early exit, and the lower layers are always executed (not skipping), which is closer to the cascading method.  

Given the existence of rich study on model cascades, it seems the novelty comes from ensembling which helps reuse the computation from the lower layers.  However, dynamic routing style methods should have effectively used the computation in the network. Unfortunately, the authors didn't provide a comparison with the dynamic routing methods. Also, they didn't provide an ablation study of the proposed ZTW methods with and without ensembling the predictions of the Internal Classifiers (ICs).

IMHO,  the clarity of the contribution and novelty, and discussion of previous work need improvement.

2. Quality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?

The submission is technically sound, and the experimental evaluation is somewhat thorough.  However,  for the image classification task, the authors only compared two recent works. They ignored the large body of work using related approaches like dynamic routing, cascades, anytime prediction, etc.  Also, the authors only apply the model to CIFAR-10/100 and mini-ImageNet.  It is common and more convincing to conduct a comparison in the full ImageNet dataset in the literature. 

Ablation study on ensembling, as mentioned in the reviews above, is missing. 

Results on wallclock speed up are recommended to include. 

The results in RL benchmarks are a plus. 


3. Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)

Overall the paper is well written. However, the clarity in the related work section can be further improved (check the concerns in Sec 1.). In addition, it would be clearer if the authors can explicitly describe how their work differs from the previous work and restate their approaches to guide the audience for comparison.  

Also, there are a few typos in the manuscript. 
- line 59. Hindisight or Hindsight. 
- line 60. "propose a instance of .." -> "propose an instance .."
- line 72. "reduce the inference speed".  The authors may mean the opposite. 

4. Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?


Yes. The results are important, and they may inspire other researchers in the field.  The results on RL benchmarks are interesting. However, more studies can be done on large-scale image classification benchmarks. 

limitations_and_societal_impact: The authors have addressed the limitations and social impact adequately.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents an approach to reuse predictions of intermediate classifiers in order to improve classification performance.
main_review: Originality: I found of great interest the idea of combining the ratings of intermediate classifiers using something well known in machine learning, that is, boosting using cascades and ensembles. The methodologies are then not new as this is "old machine learning", but I find the idea of applying them into deep learning a good contribution. To the best of my knowledge, related work is adequate.

Quality: I find methods to be appropiate and a complete piece of work. By combining such intermediate classifiers, the paper achieves quite an improvement in some well known datasets. Computation time is also evaluated in RL. Authors evaluate weaknesses which is a plus.

Clarity: I found the paper well written in general, although some English can be corrected (e.g. successes and others) and very clear to understand. In general terms I find the paper very interesting and the experimental evaluation is extensive and well analyzed. The methodology is also clearly stated and the equations seem correct. 

Significance: The paper is relevant to the field and contributes on extending deep learning through boosting. It advances the state of the art considering the extensive experimental evaluation.

Maybe some minor criticisms: Include some references about cascade connections (e.g. Viola and Jones, 2004 and others).
limitations_and_societal_impact: The paper makes a thorough analysis of the approach, including limitations, which is some wrong classifications and an analysis on why this would happen.
In any case, I found the paper interesting, well written and the methodology well evaluated.
review_title: null

Meta-review: This paper introduces an early exit strategy for pre-trained networks that allow computation to be reduced at prediction time using internal classifiers (ICs) that can terminate the required computation early.  Its distinction from previous work is in "recycling" information from previous ICs by a) feeding the output of their logits into the next IC, and b) using a geometric ensembling approach for the final predictions.

The review scores for the paper were quite split, with two reviewers (after discussions) in favour of acceptance and two reviewers feeling the paper falls marginally below the threshold.  The authors provided quite comprehensive responses to the original reviews, with all reviewers following these up and one deciding to increase their original score.  

Highlighted strengths of the paper include:
- The strength of its numerical results and the extensive number of experiments carried out
- Its overall quality, completeness, and polish
- Its clarity
- The fact that the algorithm itself is original (though there were varying opinions on quite how novel the approach is)
- The simplicity of the approach

Concerns raised by the reviewers include:
- The discussion of previous work (though the reviewers that raised this seem satisfied it can be corrected in the updated paper)
- Lack of comparisons to methods that are trained alongside the base network, rather than being applied post-hoc to an already trained network
- Lack of sufficient ablations (I believe this concern is unwarranted: as the authors point out, quite extensive ablations are actually provided in the supplement)
- Whether the paper is actually the first to use ensembling for early exit prediction (I also feel this is unwarranted as the paper clearly explains the differences and I do not feel it overclaims)
- Generality of the approach beyond image data settings

Given the high variance of scores, I have carefully also read the paper myself to form my own independent opinion, which is generally rather positive: I think this is high quality and useful work that deserves to be published (if I were a reviewer, I would have myself given the paper a "7").  In particular, I think it is quite unusually comprehensive in it considerations of issues and experiments, and that it is generally very well written and accessible.  I also feel that the authors did a very good job in addressing the concerns of the reviewers in their responses, and am inclined to take their side on most of the issues raised.  Specifically:
- I think the concerns of Reviewer Z6co were quite extensively addressed in the rebuttal and it is somewhat unclear what their remaining concerns about the technical contributions are as nothing major is raised in their review that is not addressed: their concerns about ablations are clearly unfounded; the focus on images instead of NLP would not prohibit publication and the new experiments show strong performance in the setting the reviewer is concerned with anyway; and the fact that the PABEE approach could be viewed as ensembling is not reasonable grounds for rejection in my opinion.
- Though I think the concerns of Reviewer AQfs are quite reasonable, I do not see them as sufficient grounds to reject the paper.  It should be first noted here that the reviewer themselves here also expressed in the private discussions that they were unsure if they had "set the bar too high" with their concerns and expressed that they were open to increasing their score, so I do not feel that I am completely at odds with their assessment here anyway.  To be more specific, I think they are correct to point out that comparisons between early exit approaches and those trained with efficient prediction in mind is important, but I also agree with the authors that this has not generally been done by previous work in the literature and that such comparisons are actually quite difficult to do fairly.  As such, I think it would be very harsh to reject the paper on these grounds, given that it is otherwise quite strong work.  I would strongly encourage the authors to see if they can add in any such comparisons for the camera-ready as it would noticeably strengthen the work and help correct what might be a reoccurring issue in the literature, but I do not think this should be a pre-requisite for publication (and I certainly would not suggest it is necessary to beat these baselines, so much as include them for context).

Given the above specific comments, my overall feeling is that the paper does not have any major flaws that warrant rejection by themselves, while the positives of the paper are sufficiently strong to overlook the more minor and fixable issues.  I thus recommend acceptance.

Some other comments from my own reading of the work to help make further improvements:
- The claim “Cascade connections primarily boost the accuracy of early IC” could do with some more careful explanation/discussion.  Because of the stop gradients, the cascading can never help the first IC (unless I've missed something) and at the moment the paper is a little misleading in that it is still the pushed forward information that is the thing which is helpful, but that this extra information is most important for relatively early ICs (but presumable not monotonically so: I would assume there is some sort of sweet spot, even if that is the 2nd IC?).
- There seems to be a typo in the definition of HI_m: should it not be the union of the previous C_n rather than the intersect so that it corresponds to *any* previous IC being correct?
- I presume all numbers in the tables are test accuracies?  This is never actually explicitly stated.  Information/details also seem to be missing on any use of held-out/validation data for hyper-parameter tuning.
Decision: Accept (Poster)