Venue: ICLR 2022 Conference

Reviews:

final_score: rating: null
review_content: The authors have addressed most of my concerns in their response. Also, after some discussions on the issue of lacking positive signals from wallclock time metric, I buy the arguments to position this work as a forward looking exploration in the alternative space to methods that are chosen in current hardware lottery. With this, I will raise my rating to positive inclined. 
review_title: Post-rebuttal review

final_score: rating: null
review_content: I have read the rebuttal, and I think the authors did a good job addressing most of my concerns. I still think that the paper is a bit technically incremental. However, given the authors' rebuttal response, and the positive comments of other reviewers, I'm willing to improve my initial rating to "accept" (as a poster).
review_title: Post-Rebuttal Comments

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents adaptive and anytime methods for semantic segmentation and pose recognition. Both of these are "pixel-level" tasks where the model is expected to output a prediction for each pixel in the input image. Their adaptive method, that performs variable amounts of computation depending on the input and the budget, with the amount of computation devoted to a given pixel varying from one part of the image to another. This is done by adding early exits to the base model architecture, and modifying the convolution layers to perform sparse computation on a subset of locations followed by interpolation. Experiments are done on Cityscapes semantic segmentation and MPII pose estimation benchmarks.
main_review: **Strengths**

i) Submission includes well-documented code.

The code is included in submission and the paper has an anonymous github link. README.md describes setup steps.  The code itself is reasonably well-organized.  Though I did not see many comments or docstrings, the names are reasonable and there are at least some logging messages that help make it clear what's going on in parts. Provided sample commands seem to work (at least as far as training).

Minor comment: Sample commands appear to use the authors' specific GPU setup, this seems unnecessary. It's good to have simple README sample commands targeting the least common denominator to debug environment setup and dependency installation. Though the minimal command may not be that simple due to calls to torch.distributed in the source.

ii) Description of adaptive/early exit method is overall very clear.

Figure 2 makes the high-level idea very clear, showing where the early exits and masking is done. The novel modules are described precisely in mathematical notation in the "Confidence Adaptivity" subsection.

iii) Good FLOPs/accuracy tradeoff experiments.

Experiments look at the pareto frontier of computational cost (as FLOPs) vs accuracy (as mIOU or probability of correct keypoint). They include a reasonable baseline, MDEQ that also varies in the tradeoff. Results are good, showing that the authors' method dominates the baseline.

Minor Point: Varying the HRNet architecture for mulitple tradeoff points also seems like a good baseline to illustrate here, rather than plotting only the single backbone used.

**Weaknesses**

iv) Relatively expensive early exits.

The early exits must have multiple layers of downsampling & upsampling, as described in the "Head Redesign" subsection. It does seem necessary to do this apply this to segmentation, and the authors give a good description of the reasoning. This "redesign" seems like an important novelty of the submission. The heavyweight early exists maybe make this harder to apply than the very simple early exits in classification work such as Veit & Belongie. The authors do still show some computational savings, though, so this extra cost is clearly not severe enough to invalidate the method.

v) Some unclear parts of interpolation.

The description of the interpolation, in the span between and including equations (2)-(3), does not make the handling of the mask that clear. Will zeroed-out features be included in the average for I(f_{out})? The weighting in (3) seems to only take distance into acount, not the value of the mask.

vi) Limited empirical handling of wall-clock time.

The submission describes the considerations behind the decision to report FLOPS at the bottom of page 7. The authors correctly note that reporting FLOPs is common, and that this is due to the need for lots of work on the acceleration of sparse computations in underlying libraries and hardware (see also "The Hardware Lottery" by Hooker). But the most recent work on this that they cite, such as Elsen et. al. 2020, may be worth considering in experiments: it is fairly well-optimized and has a mature implementations. So we'd expect that the best possible likely speedups in the near future are at least not that far from what would be achieved with the implementations of the past 1-2 years. In some experiments, such as the HRNet-W48 network on Cityscapes, the advantage of early exiting looks slim enough that it seems doubtful one would see wall-clock speedups even with fairly good acceleration of the sparse computations.

The video included in the submission also illustrates the real timings, and shows that the wall-clock time is probably at least reasonable: but this is less useful than quantitative experiments.

**Minor Comments**

  * "this strategy work well" -> "this strategy works well" at bottom of page 3.
  * It's usually better to number *all* equations, for instance on page 5. This makes it easier for readers (and reviewers!), or paper citing this one, to reference specific equations, even if that equation isn't cross-referenced in the paper itself.
summary_of_the_review: The method is interesting and presented well. Weaknesses in presentation and experiments are mostly lower-level and affect only parts of the method and its description or validation, and are not central to the overall paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work focuses on anytime pixel-level recognition (e.g., semantic segmentation). They propose to add intermediate exists in the architecture for anytime inference. They also consider spatial confidence adaptivity in their network, where they only execute subsequent layers on a small set of non-confidence pixels and obtain the features of other positions via interpolation. They apply the method to semantic segmentation and human pose estimation and demonstrate a reduction in FLOPs and good anytime performance for both tasks.  

main_review: The proposed method is thoughtful and demonstrates good empirical performance for anytime pixel-level recognition. This paper is also very well-written.

The proposed method is not friendly for hardwares like GPU. But I like the part where the authors implement their method and demonstrate speedup on the CPU. One small clarification: what’s the metric used here to evaluate the computation on CPU? Is that the end-to-end latency? 
 
My main concern is its technical novelty. Early exits have been explored in many previous work, e.g. Multi-scale DenseNet for anytime image classification (Huang et al., 2017). While pixel-level recognition is definitely more challenging than image classification, the solution proposed in this work is still similar to the spirit of previous work.

Also, it seems that most gain brought by the proposed method comes from the “confidence adaptivity” strategy. I find the interpolation idea really interesting and promising. But as mentioned by the authors, this strategy was originally proposed by Xie et al., 2020 (ECCV). So this downgrades the contribution of this work.

In Table 1, why do we need to show the average performance across exits? I feel the curve in Figure 3 would provide a more complete picture of the performance & efficiency. But the average performance is not really informative here.

Does the method generalize to other architectures other than HRNet? It would be helpful to see the proposed idea can work on more than one architecture family (even though HRNet is the current SOTA for segmentation).
summary_of_the_review: My main concern is that the technical novelty of this work is somewhat limited. Other than that, this work is solid and has comprehensive results and analysis. So, I feel this work is borderline and am slightly towards borderline accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an anytime method for pixel recognition tasks like semantic segmentation and human pose estimation. The key idea is to design a network with multiple "early exits", from which the network could make predictions using a corresponding prediction head. Thus, depending on criteria like budget or confidence, the network could decide where to exit and therefore leads to different accuracy-computation operating points (i.e. its anytime property). 

The technical contributions are mainly on two parts. The first contribution is a re-design of prediction heads (denoted as RH in the paper) at each early exits, mainly to tackle the difference of granularity for different intermediate feature maps. Another contribution is a confidence-based adaptive filtering mechanism that decides how to allocate computation budgets across spatial regions. Specifically, at each exit, the max prediction scores are used as a measure such that all spatial pixels that exceeds certain score threshold will not be processed in later stages. 

The authors evaluated their methods on two tasks (Cityscapes for semantics segmentation and MPII for human pose estimation) and demonstrated that their full-blown method yields a better accuracy-computation tradeoff, compared to variants of the proposed approach and several previous methods.  

------------------------- POST REBUTTAL -------------------------

The authors have addressed most of my concerns in their response. Also, after some discussions on the issue of lacking positive signals from wallclock time metric, I buy the arguments to position this work as a forward looking exploration in the alternative space to methods that are chosen in current hardware lottery. With this, I will raise my rating to positive inclined.




main_review: Strengths:
+ The task setup is interesting and of great practical value, as well-motivated in the paper. 
+ The proposed method is simple, and makes intuitive sense, also the results have shown that it maintains a better accuracy-computation tradeoff, at least with the theoretic FLOPs measure. 
+ The authors have open-sourced their implementation. 
+ The writing is very clear and easy to follow. 

Weaknesses:
- First, I appreciate that the authors discussed the matter of FLOPs vs runtime on page 7. Though, I'm a bit disappointmented about the fact that the proposed method does not achieve wallclock speedup with GPUs. As for author's argument about the potential of the proposed method with more development on sparse convolution, it holds with the assumption that it would be a cost-effective route to invest more into software/hardware support for sparse convolution, compared to investing more on developing models that are more friendly to batch computation and parallel hardware. In short, I'd be more convinced if I can see some wallclock acceleration from the proposed approach, even if it's far lower than its theoretical FLOPs speedup due to hardware constraints.  
- An important ablation that's missing is for the interpolating neighborhood size gamma, as I'd imagine it has a large effect on interpolating quality vs computation trade-off. In fact, I cannot find in the paper what's the default value used in the experiments. 
- It's not clear from the texts whether the confidence adaptivity mechanism is used in both training and testing? Or only during testing? It would be interesting to show results for both settings. 
- Though it's shown as a visualization on specific samples (Figure 4), it would be helpful to also show quantitatively the portion of pixels that exceed thresholds at each stage. 
- Cityscapes is relatively small and only with limited number of classes, it would make the results more convincing if the authors could repeat their experiments also on large datasets like ADE.  
- Table 1, when comparing "Early Exiting + RH (HRNet)" and "Early Exiting + RH + CA (HRNet)", the gain mainly comes from exit 1, in fact if you average exit 2, 3, 4, you will get almost the same accuracy (72.1 vs 72.2), does this mean that the utility of RH is mainly to provide more layers for exit 1 so that it can produce a high enough modeling capacity? 
- Page 7, "our model with confidence values as mask indicators also outperforms the feature-based mask sampling method in accuracy-computation tradeoff", when considering comparing to feature-based mask sampling method, shouldn't the correct comparison be "feature-based mask sampling" vs. "Early Exiting (HRNet)"? As my understanding is that "feature-based mask sampling" is not using an adaptive head for each exit and thus not a fair comparison to "Early Exiting + RH (HRNet)". 

Questions/concerns:
- For the re-designed head, the authors mention that only 1x1 conv is used for both the encoder and decoder, would this be too limiting? Especially for earlier exits where the receptive fields of features are not large enough. 
- Table 1, for the entry "Early Exiting (HRNet)", it's not clear what does the head design look like in this case? Are they with similar parameter counts? 
- Table 1, as I mentioned before, it's not clear whether the entry "Early Exiting + RH + CA (HRNet)" use CA both in training and testing? 

Typos:
- Page 2, "multiple predictors branch of from" --> "branch off" 


summary_of_the_review: The proposed approach is simple and has nice accuracy-FLOPs tradeoff. However, the proposed approach does not seem to yield actual wallclock speedup due to its incompatibility with current parallel chips (GPUs). I'm not an expert in HPC, but I'm not sure about the assumption that this will change in near future. Without much positive signal on that, I'm doubtful about this general direction.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new task called "anytime" prediction, which requires a model to make a progression of predictions, which might be halted at any time. The authors then also introduce an end-to-end model for this problem. The main two components behind the proposed model are: (1) a cascade of “exits” enabling the model to make progressive predictions while taking into account accuracy vs computational cost tradeoff; (2) Confidence Adaptivity, which allows the model to focus on the less confident pixel prediction. The authors implement their approach using HRNet baseline and demonstrate improved performance and efficiency on semantic segmentation and pose prediction tasks.
main_review: Strengths:
+ The paper is well written.
+ The motivation behind the proposed problem is clear and convincing.
+ The results are reasonable, i.e., the authors are able to match or improve the HRNet baseline while reducing the computational complexity.

Weaknesses:
- I'm not convinced of the novelty of the "anytime" prediction problem. The authors might be the first to formulate it as an actual research problem. However, the main requirement of this problem is to predict segmentation maps sequentially / progressively. It seems that there have already been several much older methods that do this, i.e., "Holistically-Nested Edge Detection" (ICCV 2015), which uses deep supervision to predict dense edge maps at every stage of the network, "DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs" (ICLR 2015), which also produces segmentation maps at every resolution level, etc. I'm sure there are many other methods that do this, but these are the first ones / most popular ones that come to mind. The authors in these papers don't explicitly discuss the "anytime" prediction problem, but technically one could adapt these methods to this problem quite easily.
- I'm also not convinced of the novelty of the proposed approach. The exit branches seem very similar to standard prediction branches attached to different stages of the network (as in the two papers listed above). Furthermore, the concept of confidence adaptivity shares lots of similarity with prior methods such as "PointRend: Image Segmentation as Rendering", and "Not All Pixels Are Equal: Difficulty-Aware Semantic Segmentation via Deep Layer Cascade", both of which also focus the computation of their model towards pixels that are not predicted confidently. Of course, there are subtle differences between the proposed approach and these prior methods. However, the main concepts are quite similar, which diminishes the technical novelty of the proposed approach.
- The experiments are lacking. Considering that the authors claim that they are the first ones to perform evaluations on this problem, I would expect more extensive experiments with different models/backbones (not just with HRNet, which is a bit outdated at this point). I'm not familiar with the latest state-of-the-art in semantic segmentation, but it would be useful to include at least several additional models to demonstrate the generality of the proposed approach.
- I'm confused why the accuracy in Table 1 improves when adding the  Confidence Adaptivity component. My intuition was that this should reduce the computational cost of the method, but that it should also decrease the accuracy (as we are only focusing on a subset of pixels in the future maps). Therefore, I'm confused why the accuracy would actually increase in this case.
- From Figure 5, it doesn't seem that the downsampling strategy at each exit matters that much, i.e., for three out four exits, the 3/3/3 strategy works as well as the proposed 3/2/1 scheme. 

summary_of_the_review: I'm not convinced that the proposed problem is novel. Furthermore, considering the conceptual similarity to several prior methods (see the discussion above), I believe that the technical novelty of the paper is also quite limited. Lastly, in my opinion, the current experiments are lacking. Therefore, I would recommend rejecting the paper.
review_title: null

Meta-review: This submission received 4 final ratings above the acceptance threshold: 6, 6, 6, 8. The reviewers mentioned limited novelty, but acknowledged practical importance of this work, and particularly appreciated thorough analysis provided by the authors. After a strong rebuttal, most of remaining concerns have been addressed.
The final recommendation is therefore to accept this submission as a poster.
Decision: Accept (Poster)