Anytime Dense Prediction with Confidence Adaptivity
Authors: Zhuang Liu
Zhiqiu Xu
Hung-Ju Wang
Trevor Darrell
Evan Shelhamer
Venue: ICLR 2022 Poster (2022)
Keywords: Efficient Inference
Anytime Inference
Semantic Segmentation
Dense Prediction
Computer Vision

Abstract: Anytime inference requires a model to make a progression of predictions which might be halted at any time. Prior research on anytime visual recognition has mostly focused on image classification.We propose the first unified and end-to-end approach for anytime dense prediction. A cascade of "exits" is attached to the model to make multiple predictions. We redesign the exits to account for the depth and spatial resolution of the features for each exit. To reduce total computation, and make full use of prior predictions, we develop a novel spatially adaptive approach to avoid further computation on regions where early predictions are already sufficiently confident. Our full method, named anytime dense prediction with confidence (ADP-C), achieves the same level of final accuracy, and meanwhile significantly reduces total computation. We evaluate our method on Cityscapes semantic segmentation and MPII human pose estimation: ADP-C enables anytime inference without sacrificing accuracy while also reducing the total FLOPs of its base models by 44.4% and 59.1%. We compare with anytime inference by deep equilibrium networks and feature-based stochastic sampling, showing that ADP-C dominates both across the accuracy-computation curve. Our code is available at https://github.com/liuzhuang13/anytime.

Published as a conference paper at ICLR 2022
ANYTIME DENSE PREDICTION WITH CONFIDENCE ADAPTIVITY
Zhuang Liu' Zhiqiu Xu' Hung-Ju Wang'
Trevor Darrell' Evan Shelhamer
University of California, Berkeley AAdobe Research
ABSTRACT
Anytime inference requires a model to make a progression of predictions which might be halted any time. Prior research anytime visual recognition has mostly focused on image classification. We propose the first unified and end-to- end approach for anytime dense prediction. A cascade of "exits" is attached the model to make multiple predictions. We redesign the exits to account for the depth and spatial resolution the features for each exit. reduce total computa- tion, and make full use of prior predictions, we develop a novel spatially adaptive approach to avoid further computation on regions where early predictions are al- ready sufficiently confident. Our full method, named anytime dense prediction with confidence (ADP-C), achieves the same level of final accuracy, and mean- while significantly reduces total computation. We evaluate ADP-C Cityscapes semantic segmentation and MPII human pose estimation: our method enables any- time inference without sacrificing accuracy while also reducing the total FL.OPs of its base models by 44.4% 4% and 59.1 %. We compare with anytime inference by deep equilibrium networks and feature-based stochastic sampling, showing that ADP-C dominates both across the accuracy-computation curve. Our code is available at https: wigghuu.bb..... com http//uuuuhhugnnn 3 /aateaannnng/a/iimmmmama....
INTRODUCTION
Deep convolutional networks (Krizhevsky et al., 2017; He et al., 2016) achieve high accuracy but significant computational cost. Their computational burden hinders deployment, especially for time-critical or low-resource use cases that for instance require interactivity or inference on mo- bile device. This efficiency problem is tackled by special-purpose libraries (Chetlur et al., 2014), compression by network pruning (Han et al., 2015; Li et al., 2016; Liu et al., 2019), quantization (Rastegari al., 2016; Jacob al., 2018), and distillation (Hinton al., 2015; Romero al., 2014 4). These solutions accelerate network computation but the entire network must still be computed; how- ever, a prediction may be needed sooner. Time constraints vary, but the inference time of a standard deep network does not.
Anytime inference (Fig. 1) mitigates this issue by bringing flexibility to model computation. An anytime algorithm (Dean Boddy, 1988) grad- ually improves its results more computation time is given. It can be interrupted at any point during its computation tem or user requirements demand. In this way, the time the first output is reduced while the quality of the last output is preserved.
Iuput
Final Output
Anytie Model
Output t= Output at 1
return result as sys-
Anytime inference produces a pro- gression of outputs.
Figure 1:
An anytime model makes progression of predic- tions between the first and last. This progression continues if time remains, or halts if either already satisfactory or out of time. For example, consider a user on a mobile device: an approximate result could be returned earlier if there is ur- gency, or the user could monitor sequence of predictions as time goes by and stop the model
Part of work done during an internship at Adobe Research.
Work done Adobe Research; the author is now DeepMind.

---

Published as a conference paper at ICLR 2022
once it is good enough. Note that anytime inference differs from adaptive or dynamic inference (Veit & Belongie, 2018; Wu et al., 2018; Wang et al., 8a) where the model decides how much to compute instead of external decision.
Prior research has explored anytime inference by feature selection (Karayev et al., 2014) or ensem- bling models through boosting (Grubb & Bagnell, 2012). For end-to-end neural network models, research has focused on classification for anytime inference or adaptive inference. In particular, the multi-scale dense network (Huang et al., 2017) is an architecture for resource-efficient classifica- tion. The attraction of anytime inference is not limited to classification however, and the additional computation required for dense prediction tasks makes even more desirable. For instance, an autonomous driving system may demand swifter reaction time for safety in the presence of pedes- trians, and so an anytime semantic segmentor might sooner recognize their presence. addition to urgency, an anytime segmentor could help efficiency, by not further processing already confident predictions of street pixels and therefore save power.
In this work, we develop the first single-model anytime approach for dense prediction tasks. We adopt an early exiting framework, where multiple predictors branch off from the intermediate stages of the model. The exits are trained end-to-end (both the original exit and intermediate exits), and during inference each provides a prediction in turn. To compensate for differences in depth and spatial dimensions across stages, we redesign the predictors for earlier exits. For each exit, choose an encoder-decoder architecture to enlarge receptive fields and smooth spatial noise.
Exits might suffice for anytime image classification, but dense prediction tasks have spatial struc- tures. Simple regions may need less processing while complex ones need more. Standard inference applies an equal amount of computation at every pixel without taking advantage of spatial structure. Our spatially adaptive anytime inference scheme decides whether or not to continue computation each exit and position. We mask the output of each exit by thresholding the confidence of its predictions: the remaining computation for sufficiently confident pixels is then reduced (Fig. 2). For each masked pixel, its prediction will be persisted in the following exits, as it is already sufficiently confident. In the following layers, the features for the masked pixel will be interpolated, rather than convolved, and therefore reduce computation. The confidence measure can depend on the task, e.g., in segmentation, it could be the entropy of class predictions. This confidence adaptivity can substantially reduce the total computation while maintaining accuracy.
We experiment with two dense prediction tasks: Cityscapes semantic segmentation and MPII human pose estimation. Redesigning the exits and including confidence adaptivity significantly improves across accuracy-efficiency operating points. Our full approach, named anytime dense prediction with confidence (ADP-C), not only makes anytime predictions, but its final predictions achieve the same level of accuracy as the base model, with 40-60% less total computation. For analysis, visualize predictions and confidence adaptivity across exits, and ablate design choices for the exits and masking.
APPROACH
Anytime Setting In an anytime inference setting, the user can stop the inference process based on the input current event. Thus the computation budget for each instance x could be time or input-dependent. We use B(x, to denote the computation budget assigned for instance x at time where the time variable models events that can change the budget. B(x, t) could be independent of x, i.e. the budget only depends on the time f, for example if a model on server is asked to make predictions with less budget during high-traffic hours; B(x, t) can also be independent of t, meaning the budget is only decided by input x, regardless of external events. The output of the anytime model depends the task loss and y is the ground truth, the per-instance loss is L( (I, B (z,, L)) y). This leads to the expected training loss be E(x.g)>-(X.) (x- [L[ (x. B (ar, ()) v)], where (X, Y) is the input- output joint distribution and T is the distribution modeling the time or event variable.
the budget given, and we denote it as [(x B(x, ()). Assuming L is
Early Exiting. Next, we introduce the early exiting framework which has been used in prior works (Huang et al., 2017; Teerapitaaynnon et al., 2016) for anytime prediction. Standard convolutional networks only have one prediction "head" at its final stage. The network takes the input æ, forwards it through intermediate layers, and finally outputs the prediction at its head. The concrete form of
2

---

Published as a conference paper at ICLR 2022
belween
Mask between exit
Input Image
Fealures for exit
Fealures
Final Prediction
Exit
Exit
Downsamping module
Upsampling
Prediction
Prediction 2
Predictions confident enough early exit
Active positions compuled convolution
Inaclive posiliens that are interpolated
Figure 2: Our anytime dense prediction with confidence (ADP-C) approach. We equip the model with intermediate exits for anytime inference. We redesign each exit with encoder-decoder archi- tecture to compensate for spatial resolution across model stages. At each exit's output, sufficiently confident predictions (green squares) are identified skip further computation in the following lay- ers.
the head depends on the task. For dense prediction, the head is usually one or multiple convolutions that output spatial maps representing pixel-wise predictions.
To obtain an anytime model, we attach multiple heads to the network, branching from its interme- diate features (Fig. 2). We call these additional heads early exits, since they allow the network to give early predictions and stop the inference at the current layer. Suppose add & early exits at intermediate layers with layer indices .,... 4x- We denote the intermediate features at these layers Fi, (x) .... Fi, (zz), and the functions represented by the early exits Eg Eg. Note that É,s may be of the same form but they do not share weights. The early prediction maps can be denoted as (F,, (x)), 1...k. k. Together with the original final prediction ūk+1, the total loss is:
k+1
Taotal Σ wy, ((b..))
where WP is the weight coefficient at exit i. The original network, together with the added exits, will be trained end-to-end to optimize this total loss function. In experiments, set all weights equal to 1. This corresponds to the minimization of the expected loss in Sec. 2 when the exiting probabilities all exits equal. We find this be simple yet effective scheme.
For anytime inference, as the network propagates features through its layers, if the computation budget reached the user asks the model stop, will output the latest d that already computed. Similar early exiting strategies have been used in resource-efficient image classification (Teerapittayanon al., 2016; Huang al., 2017). but dense prediction tasks require further steps detailed in the following subsections.
Head Redesign. Typical convolutional networks have a hierarchical structure that begins with shallow, fine, and more local features and ends with deep, coarse, and more global features. These deeper features represent more image content by their larger receptive fields. For dense prediction, upsampling done within the network restore lost resolution during downsampling, and ensure precise spatial correspondence between the input and the output. This upsampling can be accom- plished few (Long et al., 2015) or many (Zhao al., 2017) layers, but no matter the architecture, the network learns its most local features in its carliest layers. This presents challenge for the car- liest exits, since these features are limited in depth and receptive field. Making direct predictions at these exits with the typical 1xI convolution head produces spatially noisy and inaccurate results.
To compensate for these lacking early features, we redesign the prediction heads for the exits E,- Each E, first downsamples its input features Fl, (z). through series of pooling and 1 convolution layers. Each pooling operation halves the spatial resolution, increasing its output's receptive fields. The following convolution provides the opportunity to learn coarser-level features, specifically

---

Published as a conference paper at ICLR 2022
for that exit's prediction. After several (denoting this number as D) "pool-conv" layers, we upsample the features back to the original output resolution, with an equal number (D) of bilinear interpolation and 1 1 convolution layers. The output of this "interpolate-con" sequence will be the prediction Ds at this exit. This is important for ensuring spatial accuracy for pixel-level dense prediction tasks. Our redesigned exits essentially small "encoder-decoder" modules (Fig. 2), where the encoder downsamples the features, the decoder upsamples them back.
The downsampling ratio each exit determined by D. the number of consecutive "pool-conv" layers. Intuitively, features at earlier layers are more fine-level, and the exit branching from them can potentially benefit from more downsampling. In experiments, we use an encoder with D N downsampling operations at exit i, where N is the total number of exits, including the original last exit. Empirically we find this strategy works well, and alternative strategies are compared Sec. 4.
Finally, in all early exits, the first convolution will transform the number of channels to a fixed number for all exits. By setting the channel width relatively small, we can still save computation while adding layers with this redesigned encoder-decoder head structure.
Confidence Adaptivity. For dense prediction tasks, any early prediction úi is a spatial map con- sisting of pixel-wise predictions at each position. While most convolution networks spend an equal amount of computation at each input position, it is likely that recognition at some regions are easier than others, where the network can make predictions with high confidence even at earlier exits. For instance, the inner part of large sky segment may be easy to recognize, whereas the boundary between the bicycle and the person riding it may need more careful delineation.
Once early prediction is made, we can inspect the "confidence" at each position. As an example, for semantic segmentation, the maximum probability over all classes can serve as confidence measure. If the confidence has passed a pre-defined threshold at certain positions (green squares on predictions in Fig. 2), we may decide these predictions are likely to be correct, and not continue the computation of further layers this position. Suppose the pixels of the early prediction ğ are indexed by p. we form a mask M,:
0, if (onfiidence((.(p)) 2 Threshold
(p))) 1 otherwise
For any convolution layer between exit (E,) and the next exit + 1 (Es+1). we could choose whether to perform or skip computation at position p based on the mask (Fig. 2). Assuming C' is convolution layer with input Sin, then by applying the mask, the output sout at position p becomes:
(C(fm)))))
(p) = 1, if (p) 0.
Sone(p) = 10,
If O's output and the mask M, do not share the same spatial size, we interpolate 9: in Eqn. to the size of C"s output, so that the mask M, is compatible with C' in Equ. 3.
The output four could be sparse, with many positions being O. This could potentially harm further convolutional computation. To compensate for this, we spatially interpolate these positions from their neighbors across all channels, using a similar approach as in (Xie et al., 2020). Denoting the interpolation operation as I, the final output feature fonr is
Soue(p) = (ou) if (p) (p),
if M.) 1,
(4)
= 0.
Here, the value of (p) is a weighted average of all the neighboring pixels centered at p within a radius r:
((for)(() ExErg) W (on.)))
(5)
ExERgp) W(p.s)
where s indexes p's neighboring pixels and k(p) [s[[|s -pls> Sr, p), the neighborhood of We set radius r = 7 in our experiments. W(p.s) is the weight assigned to point s for interpolating p. for which we use the RBF kernel, a distance-based exponential decaying weighting scheme:
W ((p.e) = exp ( (->21pp -112)
4

---

Published as a conference paper at ICLR 2022
with À being a trainable parameter. This indicates that the closer is to p. the larger its assigned weight will be. Note that masked-out features (M) = 0) still participate in the interpolation process as inputs with values 0.
Replacing filtering by interpolation at these already confident spatial locations (M) = 0) could potentially save a substantial amount of computation. The mask M, will be used for all convolutions between exit i and + 1, including the convolutions inside exit + 1. Once the forward pass arrives the next exit, to make the prediction 1i+1, the last prediction at positions where M.(p) 0 will be carried over, having already been deemed confident enough at the last exit and having been skipped during further computations. This means:
(p) (a (a)),,
if M. = 1, if M:(p) 0.
(x((p);
The network then calculates a new mask M1+1 based on ii+1, and uses it to skip computation going forward. The process continues until reach the final exit.
In summary, we incorporate spatial confidence adaptivity into the early exiting network, by not fil- tering at spatial locations that are already sufficiently confident in the latest prediction. At these positions interpolation is used instead, at much reduced computational cost, to avoid excessive spar- sity. Unless otherwise specified, confidence adaptivity is used in both training and inference. We dub our full approach as anytime dense prediction with confidence (ADP-C).
3 EXPERIMENTS
We evaluate ADP-C with two dense prediction tasks: semantic segmentation and human pose esti- mation. Our experiments implemented using PyTorch (Paszke et al., 2019).
Architectures. We use the High- Resolution Network (HRNet) (Wang et al., 2020) architecture as our base model. HRNet is a multi-stage architecture, where each stage adds lower-resolution/laaer scale features. Specifically, HRNet-W W18 model. HRNet-W W48 is state-of-thh-an for semantic segmentation and HRNet-Wis W32 is suitable for pose estimation. HRNet-W18 is highly efficient and has been shown outperform other efficient networks (Zhao et al., 2018; Sandler et al., 2018) () in its accuracy-efficiency tradeoff. The 48/32/18 denotes the number of channels in the bottleneck of the first stage. The original head for HRNet before our redesigning is two consecutive x 1 convolutions. We attach three exits, one at the end of each stage before the final prediction. We follow the training/evaluationn protocol and hyperparameters of the reference HRNet implementation at (Sun et al., 2019; Wang et al., 2020) (except that our models include a loss each exit). Please see Appendix C for more training details.
adopt the standard HRNet- (W48, W32) models and the smaller
we compare with a standard HRNet that has only one (final) exit with the same backbone architecture. The standard HRNet not anytime, so we focus on comparing it with our anytime model's final exit. 2. MDEQ (Bai et al., 2020) is recent deep implicil model, which achieves competitive performance on vision tasks without stacking explicit layers, but rather solves an optimization problem for inference. Its representation x" is an equilibrium point of its learned transformation f2;;2), i.e., f(2"; = x* where x is the input. The representation is obtained by iteratively solving the equation x==;x) 2, for which the quality of the solution improves with more iterations. The converged representation is then decoded into a prediction. We examine anytime prediction with the MDEQ by decoding intermediate iterates of the representation. To the best of our knowledge, this is the first study of anytime implicit modeling, as (Bai et al., 2020) only reports the predictions of implicit models equilibrium, and does not produce or inspect intermediate predictions. We use the "small" version of the MDEQ (Bai et al., 2020) and the 4th, 6th, 8th, and 10th iterations of its equilibrium optimization to bound the amount of computation and align its iterations with our architecture's stages. 3. Feature-Basse Stochastic Sampling: we follow Xie et al. (2020) in using internal features to predict masking positions, with the Gumbel- Softmax trick (Jang et al., 2016) used for sampling. We use 3 x 3 convolution upon the features for exits (Fig. 2) for mask prediction until the next exit. During training, an L1 sparsity regularization is applied the stochastic continuous mask values, and only during inference the mask values discretized. The interpolation procedure is the same as in our method. This baseline is evaluated on HRNet-WIss
Baselines. 1. HRNet:
5

---

Published as a conference paper at ICLR 2022
Accuracy (mloU)
Computation (GFLOPs)
Method Output
2 3 4
Avg
2
4
Avg
One-exit
HRNet- W48 (Wang ct al., 2020)
80.7
696.2
MDEQ-Small (Bai ct al., 2020)
17.3 38.7 65.5 72.4 48.5 521.6
717.9 914.2
1110.5 816.0
Baselines
EE (HRNet)
34.3 59.0 76.9 80.4 62.7 48.4 113,4
388.9
722.2
318.2
EE + RH (HINNe)
44.6 60.2 76.6 79.9 65.3
41.9
105.6 368.0 701.3 304.2
Ours
ADP-C: EE + RH + CA (HRNet) 44.3 60.1 76.8 81.3 65.7 41.9 93.9 259.3 387.1
195.6
Table Accuracy (mloU) and inference computation (GFL.Os)) for Cityscapes semantic segmenta- tion with four exits. Our approach achieves higher accuracy in less computation than the HRNet and MDEQ baselines across exits. Early exiting (EE) makes progressive predictions. Redesigned heads (RH) improve early predictions (exits and 2). Confidence Adaptivity (CA) reduces computation.
3.1 SEMANTIC SEGMENTATION
The Cityscapes dataset (Cordts et al., 2016) consists of 2048 1024 images of urban street scenes with segmentation annotations of 19 classes. We train the models with the training set and report results on the validation set. 'he accuracy metric is the standard mean intersection-over-iiin (mloU %), and the computation metric the number of foating-poiie operations (FI.Oss.. Anytime inference improves with higher accuracy, less computation, and more predictions. We evaluate HRNet-WAa and HRNet- MiNNet-W18 for this task.
Redesigned heads (RH) use our encoder-decoder structure for exits. Since we have 4 exits in total, we repeat the downsampling operation 3/2/1 times at exit 1/2/3 to generate larger-scale features for carlier exits, as described in Sec. 2. We set the number of channels at all exits to 128/64 for HRNet-W48/W155 W48/W18. For confidence adaptivity (CA), we use the maximum probability among all classes the confidence measure, and set the confidence threshold in Eqn. 2 to be 99.8% based on cross-validation. For CA, the computation for each input can differ, so we report the average FL.OPs across all validation images at each exit.
The results for HRNet-W48 are shown in Table 1. We observe that our early exiting model based on HRNet-W48 outperforms the MDEQ model by large margin, with significantly less FLOPs at each exit. With RH, we achieve notable accuracy gain in early predictions, especially at the first exit (more than 10%). with roughly the same computation. With CA added, we arrive our full ADP-C method (RH CA), which maintains roughly the same accuracy as the RH model but reduces the total computation exits and 4. Interestingly ADP-C has slightly higher mIoU at the final exit (81.3 vs. 80.7) with 44.4% less total computation (387.1 vs. 696.2 GFLOPs) compared to the base HRNet. This is possibly due to a potential regularization effect of confidence adaptivity: computing fewer intermediate features exactly may prevent overfitting.
The same results are plotted in Fig. 3 (left). The plot shows accuracy (y-axis) and computation (a- axis) tradeoffs: points to the upper left indicate better anytime performance. The baseline HRNet is represented by a red cross, while anytime models are plotted as curves with a point for each predic- tion. We plot the results for the smaller HRNet-WIs. model in Fig. 3 (middle). RH improves early prediction accuracy from the basic early exiting model, and CA substantially reduces computation at later exits. The full model reaches the same-level of accuracy as the baseline HRNet with much less total computation. We note that our model with confidence values as mask indicators also outper- forms the feature-based mask sampling method in the accuracy-computatin tradeoff, demonstrating that confidence is effective at filtering out redundant computation despite its simplicity.
Our experiments measure computation by FLOPs rather than time. Reporting FLOPs is common (Figurnov et al., 2017; Xie et al., 2020; Huang et al., 2017; Wang et al., 2018a; Liu et al., 2019) and meaningful because it hardware independent. However, similarly to spatially adaptive com- putation methods (Figurnov et al., 2017; Xie et al., 2020), our model does not achieve wall clock speedup at this time due to the lack of software/hardware support for sparse convolution with current frameworks and GPU devices. To approximate CPU speedup, we conduct profiling experiment on a theeadithrediigg processor (specifically we measure computation time on a Linux machine with Intel Xeon Gold 5220R CPUs using 16 threads). We replace all convolutions with implemen- tations following (Xie et al., 2020). ADP-C on HRRNet-Wag achieves 1.48x x speedup compared to the non-anytime baseline, measured in end-to-end latency (wall-clock time). There is gap between

---

Published as a conference paper at ICLR 2022
HRNel-W18 Cityscapes Segmentation
HRNee--W188 on Cityscapes Segmentation
MPII Estimation
15
60-
80
POKWgga
mou
HRNet
50
40
-
Eary Exitin (HRNee)
Earr RH (HRNN)))
20
Eary Exitni RH CA (HRNet)
30
Reeture Based Stochentc Sampl ng (IRNN)
600
50
70
GFLOPs
GFLOPs
GFLOPs
Figure 3: Accuracy (v) and computation (x) at four exits across architectures (HRNet- W48/W32/W18) and tasks (semantic segmentation and pose estimation). Anytime performance improves with higher y (more accuracy) and lower z (less computation). Redesigned heads (RH) boost the accuracy at early exits, while confidence adaptivity (CA) reduces computation by up to more than half. Our method outperforms baseline methods (MDEQ and Feature-based Stochastic Sampling) across the accuracy- conerracctcomtaatiionnn tradeoff curve.
this measured time and the theoretical 1.80x speedup measured by FLOPs. ADP-C and others can benefit from ongoing and future work on efficient sparse convolutions (Graham van der Maaten, 2017: Choy et al., 2019; Verelst & Tuytelaars, 2020; Elsen et al., 2020). We also refer readers to the Hardware Lottery (Hooker, 2021) for discussion on how hardware compatibility affects progress in Al research. Please see the supplement for an anytime inference video where each exit is timed to the computation requires.
3.2 HUMAN POSE ESTIMATION
For human pose estimation, we evaluate on the MPII Human Pose dataset (Andriluka et al., 2014) of image crops annotated with body joints collected from everyday human activities. The positions of 16 joint types are annotated for the human-centered in each crop. We report the standard metric (Andriluka et al., 2014) for MPII, the PCKh (head-nords neremlnoomalized probability of correct keypoint) score, on its validation set. We use HRRNet-W32 for this task and follow the reference settings from (Sun et al., 2019). The standard head for this task is x 1 convolution. As in segmentation, our redesigned heads encoder-decoded structures. The number of channels for all exits is 64.
Pose estimation task is formulated as regression. The HRNet model outputs 16 spatial feature maps, each regressing the corresponding body joint. The only positive target for each type is coded as 1; all other points are negatives coded as Unlike segmentation, the output at each pixel is not a probability distribution,
the maximum value across channels as the confidence measure. A pixel is masked out if the maximum value at that position is smaller than the threshold, marking it unlikely be joint prediction. We choose 0.002 the threshold by cross-validation, value makes the mask too sparse and hurts learning. The RH + CA model adopts adaptivity after 10 epochs of normal training, because nearly all outputs are too close zero in the beginning.
larger
PCKh@0.5
GFLOPss
Method / Output
Last
Avg
Last
Avg
HRNet-W23 W32 (Wang et al., 2020)
90.33
9,49
EE (HRNet)
90.31
74.60
9.51
4.73
EE + RH (HRNet)
90.26
79.16
9.55
4.76
ADP-C: EE + RH + CA (HRNet)
90.20
79.04
3.88
2.44
Table 2: Accuracy (PCKh@0.5) and computation (GFLOPs) on MPII pose estimation the last exit and averaged for all exits. Early exits (EE) make progress predictions, redesigned heads (RH) improve accuracy, and confidence adaptivity (CA) reduces computation.
Fig. 3 (right) and Table 2 show the results. We observe similar trend segmentation: RH improves accuracy and CA reduces FLOPs. In this case, ADP-C reduces computation by 59.1% (9.49 to 3.88 GFLOPs) while accuracy only drops by 0. 13% relative to the baseline HRNet.
7

---

Published as a conference paper at ICLR 2022
Exit
Exit
Ext
Prediction
Confidance
Masked Points
Figure 4: Top: prediction results all exits. Middle: confidence maps, lighter color (yellow) indicates higher confidence. Bottom: correct/wrong predictions at the exit drawn as white/black. The confident points selected for masking are in green. Confidence adaptivity excludes calculation on already confident pixels (green) in early exits, mostly located at inner parts of large segments.
4 ANALYSIS
Visualizations. To inspect anytime predictions and masking Cityscapes, visualize ADP-C exit results on a validation image with HRNet- W488 Fig. 4 shows the predictions, confidence maps, and computation masks across exits. With each exit, the prediction accuracy improves, especially in more detailed areas with more segments. The confidence maps are shown with high lighter/yellow and low darker/green. Most unconfident points lie around segment boundaries, and the interior of large stuff segments (road, vegetation) are already confident at early exits. This motivates the use of confidence adaptivity to avoid unnecessary further computations on these areas. For computation masks, the correct/incorrect predictions at each exit are marked white/black. Pixels surpassing the confidence threshold (99.8%) are masked and marked green. Many pixels can be masked out in this way, and each exit masks more. Most of the masked pixels are found in the inner parts of large segments or already correct areas. fact, the masked pixels are 100% correct at all exits for this instance, which partly justifies their exclusion from later computation. The predictions at these positions are already confident and correct at early exits, and so the only potential harm of skipping their computation later is their possible effect at less confident positions. See Appendix E for more visualizations. On average, 19.3%, 38.4%, 63.0% pixels are masked out at exit 1, 2, 3, respectively.
Downsampling at Early Exits. In Sec. 2, we described how many consecutive downsampling operations
D 3/2/1 consecutive "pool- conv" layers for downsampling at exit 1/2/3. Here we compare this strategy with D 1/1/1 and 3/3/3, /3, where the same level of downsampling and hence the same head structure is used at all exits, on HRNet-We W18. Fig. 5 (left) shows that our adopted D = 3/2/1 strategy obtains the highest accuracy at all exits among these choices.
each exit, by D N i, which means
Downsampling times at three early exits
Masking Criterion
70
3/211
+
D 1/1/1
Average mioU (all exits)
-
3/3/3
:00
60
Entropy Max Prob.
mloU
50
Max Prob. (Ratic)
Random
40
30
30
30
40
50
60
70
25
30
35
40
45
GFLOPs
Average GFLOPs (all exits)
Figure 5: Left: comparing downsampling strategies. D = 33/2/1 means downsampling the features 3/2/1 times at exit 1/2/3. Right: comparison between different masking criteria.
8

---

Published as a conference paper at ICLR 2022
Masking Criterion. We used the max probability as the confidence measure and fixed threshold for masking. Here we consider a few alternatives. One is to mask out the top k% (by max prob.) of the pixels at each exit, regardless of their values. We also consider thresholding probability distribution. In addition, we compare them with random masking. We use HRNet-WIN W18, change the ratio or threshold for wide range, and present the average mloU vs. GFLOPs all exits at Fig. 5 (right). For this ablation, adaptivity is only applied during inference.
the entropy of the
First, we notice all three confidence criteria largely outperform random masking. For max proba- bility, using a threshold performs slightly better than a fixed ratio, possibly because this gives the flexibility for different exits mask out different amounts of points. Finally, we observe using en- tropy as the confidence measure performs similarly as using max probability, but we stick to max probability in our methods because is trivial to compute.
5 RELATED WORK
Anytime Inference. Anytime algorithms (Zilberstein, 1996; Dean & Boddy, 1988) can be inter- rupted at any point during computation to return a result, whose quality improves gradually with more computation time. In machine learning, anytime inference has been achieved by boost- ing (Grubb & Bagnell, 2012), reinforcement learning (Karayev et al., 2014), and random forests (Fröhlich al., 2012). Anytime deep networks have been brought to bear on image classification, but not dense prediction. Branching architectures have been a common strategy (Amthor et al., 2016; Teerapittayanon et al., 2016) along with other techniques such as adaptive loss balancing (Hu et al., 2019). While there is work on the tasks of person re- re-identification ( (Wang et al., 2019) and stereo depth (Wang et al., 2018b), these techniques are task-specific, while our method applies to multiple dense prediction tasks, as we show with semantic segmentation and pose estimation. Liu et al. (Liu & He, 2016) learn a hierarchy of models for anytime segmentation, but its multiple models complicate training and testing, and require more memory. Our work instead augments the base model architecture for simplicity and efficiency. The PointRend method (Kirillov et al., 2020) out- puts initial dense prediction first and then refine it adaptively, but the predictions are all made full depth. Its majority of the computation is spent before the first output and thus cannot be practi- cally "anytime". Our method is the first to selectively update anytime predictions across space and layers.
Adaptive Computation. An adaptive model adjusts its computation to each specific instance dur- ing inference. For deep networks, this is often done by adjusting which layers to execute, that is, choosing which layers to run or skip. This can be done by supervised controller (Veit & Belongie, 2018; Liu & Deng, 2017), a routing policy optimized by reinforcement learning (Wang et al., 201 8a; Wu et al., 2018; Lin et al., 2017a), or other training strategies (McGill & Perona, 2017). Rather than choosing layers. spatial adaprivity chooses where to adjust the amount of computation across differ- ent spatial positions in the input. For example, the model could infer spatial masks for feature maps and skip computation on masked areas (Shomron et al., 2019; Dong et al., 2017: Lin et al., 2017b; Ren et al., 2018; Cao et al., 2019). Figurnov et al. (2017) maintains a halting score at each pixel and once it reaches a threshold the model will stop inference at those positions for spatially coarse tasks like classification or bounding box detection. Xie et al. (2020) stochastically sample positions for computation from an end-to-end learned sampling distribution. Li et al. (2017) convert deep net- work into a difficult--warr cascade, where earlier steps handle easier regions and later steps tackle harder regions. These spatially adaptive models reduce computation, but are not anytime: they do not make a series of predictions and cannot be interrupted.
CONCLUSION
We propose ADP-C, the first single-model anytime approach for dense visual prediction. Based on an early-exiting framework, our redesigned exiting heads and confidence adaptivity both improve the
tradeoff. On Cityscapes semantic segmentation and MPII pose estimation, ADP-C achieves 40%-60% FLOPs reduction with the same-level final accuracy, compared against the baseline HRNet. We further analyze confidence adaptivity with visualizations and ablate key design choices justify our approach anytime inference with confidence.
9

---

Published as a conference paper at ICLR 2022
This work was supported in part by DoD including DARPA's XAL, LwLL, and/or SemaFor programs, as well as BAIR's 's industrial alliance programs.
Acknowledgement.
REFERENCES
Manuel Amthor, Erik Rodner, and Joachim Denzler. Impatient dnns-deep neural networks with 2016.
dynamic time budgets. arXiv preprint
Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estima- tion: New benchmark and state of the analysis. In CVPR, 2014.
Shaojie Bai, Vladlen Koltun, and Zico Kolter. Multiscale deep equilibrium models. arXiv preprint (00::20066066556, 2020. 5, 6
Shijie Cao, Lingxiao Ma, Wencong Xiao, Chen Zhang, Yunxin Liu, Lintao Zhang, Lanshun Nie, and Zhi Yang. Seernet: Predicting convolutional neural network feature-may map sparsity through low-bit quantization. In CVPR, 2019. 9
Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catan-
zaro, and Evan Shelhamer. cuDNN: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759, 2014.
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In CVPR, 2019. 7
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Thomas L Dean and Mark S Boddy, An analysis of time-dependent planning. In AAAI, 1988. 1,9
Xuanyi Dong, Junshi Huang, Yi Yang, and Shuicheng Yan. More is less: A more complicated network with less inference complexity. CVPR, 2017.
Erich Elsen, Marat Dukhan, Trevor Gale, and Karen Simonyan. Fast sparse convnets. In CVPR, 2020. 7
Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. In CVPR, 2017. 6,
Björn Fröhlich, Erik Rodner, and Joachim Denzler. As time goes by-anytime semantic segmenta- tion with iterative context forests. Joint DAGM (German Association for Pattern Recognition) and OAGM Symposium. Springer, 2012. 9
Benjamin Graham and I Laurens van der Maaten. Submanifold sparse convolutional networks. arXiv
preprint arXiv: 1ee.:177...3330000 30307, 2017. 7
Alex Grubb and Drew Bagnell. Speedboost: Anytime prediction with uniform near-optimality. In Artificial Intelligence and Statistics, 2012. 2.9
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. NeurIPS, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1 025331, 2015.
Sara Hooker. The hardware lottery. Communications of the ACM, 64(12):58-66 65, 2021.
HRNet Human Pose Estimation. Code for deep high-resolution representation learning for human pose estimation. https: /gght/itt.b///./// com/leoxiaobin/
deep-highh-resolute net.tooor 2019. 14
10

---

Published as a conference paper at ICLR 2022
HRNet Semantic Segmentation. Code for deep high-resolution representation learning for vi- sual recognition. ptttps//githin com/ Hw////NNN////H /HRNet Seweeettteeaaniiiiie pemmaatiic-eemmetttt..... 2019. 13
Hanzhang Hu, Debadeepta Dey, Martial Hebert, and Andrew Bagnell. Learning anytime predic- tions in neural networks via adaptive loss balancing. In AAAI, 2019. 9
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Wein- berger. Multi-scale dense networks for resource efficient image classification. arXiv preprint 17X3094703.09844, 2017. 3,
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient antege--arithmeticone
inference. In CVPR, 2018.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint 161i 1.0/ 144, 2016. 5
Sergey Karayev, Mario Fritz, and Trevor Darrell. Anytime recognition of objects and scenes. In CVPR, 2014. 2,9
Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. In CVPR, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 2017. 1
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.087100 2016.
Xiaoxiao Li, Ziwei Liu, Ping Luo, Chen Change Loy, and Xiaoou Tang. Not all pixels are equal: Difficulty-wware semantic segmentation via deep layer cascade. In CVPR, 2017.
Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In NeurlPS, 2017a.
Yingyan Lin, Charbel Sakr, Yongjune Kim, and Naresh Shanbhag. Predictivenet: An energy- efficient convolutional neural network via zero prediction. In 2017 IEEE international symposium on circuits and systems (ISCAS). IEEE, 2017b. 9
Buyu Liu and Xuming He. Learning dynamic hierarchical models anytime scene labeling. In ECCV. Springer, 2016. 9
Lanlan Liu and Jia Deng. Dynamic deep neural networks: Optimizing accuracy-efficieney trade-offs by selective execution. arXiv preprint arXiv:/701 771.00099, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. ICLR, 2019. 1.6
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Mason McGill and Pietro Perona. Deciding how to decide: Dynamic routing in artificial neural networks. arXiv preprint / 2017. 9
MDEQ. Code for multiscale deep equilibrium models. idda, 2020. 17
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmen- tation in the wild. In CVPR, 2014. 14
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurlPS, 2019.

---

Published as a conference paper at ICLR 2022
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV. Springer, 2016.
Mengye Ren, Andrei Pokrovsky, Bin Yang, and Raquel Urtasun. Sbnet: Sparse blocks network for fast inference. In CVPR, 2018.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv: 2014. 1
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Gil Shomron, Ron Banner, Moran Shkolnik, and Uri Weiser Thanks for nothing: Predict- ing zero-valued activations with lightweight convolutional neural networks. arXiv preprint arXiv:1909.07636, 2019.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. CVPR, 2019. 5,7
Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsurg Kung. Branchynet: Fast inference via early exiting from deep neural networks. In JCPR. IEEE, 2016. 2, 3,
Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In ECCV, 2018. 2, 9
Thomas Verelst and Tinne Tuytelaars. Dynamic convolutions: Exploiting spatial sparsity for faster inference. In CVPR, 2020. 7
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recogmition. TPAMI, 2020. 5, 6, 7, 14
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dy- namic routing in convolutional networks. In ECCV, 2018a. 2, 6,
Yan Wang, Lequn Wang, Yurong You, Xu Zou, Vincent Chen, Serena Li, Gao Huang, Bharath Hariharan, and Kilian Q Weinberger. Resource aware person re-identification across multiple resolutions. In CVPR, 2018b.
Yan Wang, Zihang Lai, Gao Huang, Brian H Wang, Laurens Van Der Maaten, Mark Campbell, and Kilian Q Weinberger. Anytime stereo image depth estimation mobile devices. In ICRA, 2019.
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, L. Davis, K. Grauman, and R. Feris. Blockdrop: Dynamic inference paths residual networks. CVPR, 2018. 2.
Zhenda Xie, Zheng Zhang, Xizhou Zhu, Gao Huang, and Stephen Lin. Spatially adaptive inference with stochastic feature sampling and interpolation. arXiv preprint arXiv:2003.08866 2020. 4, 5, 6,9
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi. Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
Hengshuang Zhao, Xiaojuan Qi. Xiaoyong Shen, Jianping Shi, and Jiaya Jia. Icnet for real-time semantic segmentation on high-resolution images. In ECCV, 2018. 5
Shlomo Zilberstein. Using anytime algorithms in intelligent systems. Al magazine, 1996. 9
12

---

Published as a conference paper at ICLR 2022
APPENDIX
A ABLATION ON INTERPOLATION RADIUS
In the main paper,
default radius of when interpolating masked-out features (Eqn. and following text), Here we demonstrate that the radius of is reasonable through an ablation experiment, whose results are listed in Table 3. The experiment is conducted with HRNet-W18 on Cityscapes semantic segmentation, with both redesigned head (RH) and confidence adaptivity (CA). We observe that a radius of outperforms lower ones (3 and 5) in the average mloU slightly, with minimal FLOPs addition, due to the fact that interpolation is done channel-wise. A further increase of radius to does not bring significant gain in final or average mloU.
Accuracy (mloU)
Computation (GFLOPs)
Radius
Avg
1
2
4
Avg
41.06
48.25
67.56
75.82
58.17
23.7
33.0
44.4
57.0
39.5
40.86
48.01
67.64
76.21
58.18
23.7
33.1
44.6
57.4
39.7
7 (default)
41.05
48.35
67.73
76.10
58.31
23.7
33.1
44.9
58.1
40.0
41.01
48.41
67.88
76.08
58.35
23.7
33.2
45.4
59.1
40.4
Table 3: Accuracy (mloU) and inference computation (GFLOPs) for Cityscapes semantic segmen- tation with four exits under different settings of interpolation radius. A minor increase in mloU and GFLOPs is observed with a larger radius.
B INFERENCE-ONLY CONFIDENCE ADAPTIVITY
Accuracy (mloU)
Computation (GFLOPs)
Adaptivity
2
3
4
Avg
2
3
4
Avg
No Adaptivity
44.61
60.19
76.64
79.89
65.33
41.9
105.6
368.0
701.4
304.2
Training and Inference
44.34
60.13
76.82
81.31
65.65
41.9
93.9
259.4
387.1
195.6
Inference-only
44.61
59.97
76.37
79.69
65.16
41.9
94.1
291.8
484.8
228.1
Table 4: Accuracy (mloU) and inference computation (GFLOPs) for Cityscapes semantic segmen- tation with four exits under different adaptivity settings.
In the experiment section of the main paper, confidence adaptivity is used in both training and inference. Here we compare this with the setting where adaptivity is only used for inference in Table 4. We use HRNet- W48 with redesigned heads (RH) on Cityscapes for this experiment. "No Adaptivity" corresponds to the "EE + RH" row in Table 1. We observe that using adaptivity only inference hurts the accuracy, compared with using it at both training and inference. It also increases the average FLOPs at exit and
C EXPERIMENTAL SETTINGS
For Cityscapes semantic segmentation, we follow the training settings at the official codebase (HR- Net Semantic Segmentation) of HRNet for semantic segmentation. The HRNet- W W18/48 models are pre-trained on ImageNet. During training, multi-scale and flipping data augmentation is used, and the input cropping size is 512 1024. The model trained for 484 epochs, with an initial learning rate of 0.01 and polynomial schedule of power 0.9, a weight decay of 0.0005, batch size of 12, optimized by SGD with 0.9 momentum. In evaluation, we use single-scale testing without flipping, with input resolution 1024 x 2048. For the Feature-Basee Based Stochastic Sampling baseline, modify the exit weights from (1, 1, 1, 1) to (0.5, 0.5, 0.5, 1) as we find it produces more stable masking val- ues during training. The L1 sparsity regularization
masks is set to 0.1. The mask outputs exit (1, 2, 3) have additional weight factors of (1/3,22 2/3, 1) to encourage more sparse features towards the end.
13

---

Published as a conference paper at ICLR 2022
For MPII human pose estimation, follow the training settings at the official codebase (HRNet Human Pose Estimation) of HRNet for pose estimation. The HRNet-32 model we use is also pre- trained on ImageNet. The image size for both training and evaluation is 256 x 256. The model is trained for 210 epochs, with an initial learning rate of 0.001, and decaying of 0. at epoch 170 and 200. The optimization is done by Adam with ÿ1 0.99, 72 momentum of 0.9. The batch size is 128. In evaluation, flipping test is used.
weight decay of 0.0001, and
D PASCAL-CONTEXT RESULTS
We present results with the PASCAL-Context semantic segmentation dataset (Mottaghi et al., 2014) in Table 5. It consists of 59 segmentation classes. For the Early Exiting baseline, use weights of (0.33, 0.33, 0.33, 1) for 4 exits. respectively, as we found increasing it to all would hurt the final performance too much. The confidence threshold is set to 99.5%. We observe RH improve the accuracy in most exits. ADP-C outperforms the Early Exiting baseline in average mloU, despite its final mloU worse. also saves more than 20% FL.OPs compared with the vanilla Early Exiting baseline.
Accuracy (mloU)
Computation (GFLOPs)
Method / Output
2
3
4
Avg
2
4
Avg
One-exit
FRRNet-W48 (Wang et al., 2020)
51.35
76.5
Baseline
Early Exiting (HRNet)
8.78 18.74
39.82 50. 10 29.36
5.4 12.6
43.1
80.1 35.3
EE + RH (HRNet)
14.55 19.42
39.30
50.32
30.90
4.5
11.4
40.2
77.2
33.3
Ours
ADP-C: EE + RH + CA (HRNet)
14.51 19.18 39.00
49.27 30.49
4.5
11.1 38.1 62.7 29.1
Table 5: Accuracy (mloU) and inference computation (GFLOPs) for PASCAL-Context semantic
segmentation.
E MORE VISUALIZATIONS
Figure 6: Input and ground truth of the example validation images visualized in Fig. 7 and Fig. 8
We present more visualizations of the same type as Fig. the main paper, Fig. and 8. Their input and ground truth are shown in Fig, 6 in the same order. We can see the same trend as discussed in the main paper still holds: the model will mask out confident points that are inside large segments (e.g., road, vegetable), which are mostly already predicted correctly in early exits.
14

---

Published as a conference paper at ICLR 2022
Exil
Cxit
Prediction
Confidence
Prediction
Masked
Exit
Exil
Exit 3
Ext4
Predicton
Map
Confidnce
Monked
Figure Top: prediction results exits. Middle: confidence maps, lighter color (yellow) indicates higher confidence. Bottom: correct/wrong predictions at the exit drawn as white/black. The confident points selected for masking are green. Confidence adaptivity excludes calculation on already confident pixels (green) in early exits, mostly located at inner parts of large segments.
15

---

Published as a conference paper at ICLR 2022
EXIt
Prediction
Exit
Ext
Exit 3
Ext4
Contidanca Prediction
Exi.
Confidence Prediction
Figure 8: Top: prediction results at all exits. Middle: confidence maps, lighter color (yellow) indicates higher confidence. Bottom: correct/wrong predictions the exit drawn as white/black. The confident points selected for masking are in green. Confidence adaptivity excludes calculation on already confident pixels (green) early exits, mostly located at inner parts of large segments.
16

---

Published as a conference paper at ICLR 2022
F MORE MDEQ RESULTS
In the main paper, we used MDEQ's "small" model's 4th, 6th, 8th, and 10th iterations' results align with model's 4 exits. In this section, we provide full results on Cityscapes semantic segmentation at all iterations for both the "small" and the "XL" (extra-large) models at Table 6. The original configuration (MDEQ) sets the number of iterations to 26 and 27 for MDEQ-small and MDEQ-XL.. In Fig. and 10, we also provide qualitative visualization of results for the same validation images from Fig. with MDEQ-small 4th, 6th. 8th, 10th, 14th, 18th, 22th, 26th iterations. With the progression of iterations, the predictions ger more and more accurate.
MDEQ-Small
MDEQ-XL
Iteration
mloU
GFLOPs
mloU
GFLOPs
1.1
227.2
1.9
1983.0
2
1.1
325.3
1.9
2861.3
9.0
423.5
8.0
3739.6
4
17.3
521.6
11.6
4617.9
38.7
619.7
34.1
5496.2
6
38.7
717.9
49.4
6374.5
7
61.0
816.0
58.6
7252.9
65.5
914.2
67.1
8131.2
9
70.1
1012.3
71.6
9009.5
10
72.4
1110.5
74.5
9887.8
11
73.8
1208.5
76.1
10766.
12
74.6
1306.7
77.3
11644.4
13
75.1
1404.8
77.9
12522.7
14
75.5
1503.0
78.7
13401.0
15
75.8
1601.1
79.1
14279.3
16
75.9
1699.3
79.3
15157.6
17
76.1
1797.4
79.5
16036.0
18
76.2
1895.5
79.6
16914.3
19
76.2
1993.7
79.6
17792.6
20
76.2
2091.8
79.7
18670.9
21
76.2
2190.0
79.8
19549.2
22
76.1
2288.1
79.9
20427.5
2386.2
79.9
21305.8
24
76.3
2484.4
79.9
22184.2
76.4
2582.5
79.9
23062.5
26
76.5
2680.7
79.8
23940.8
27
76.5
2778.8
79.8
24819.1
Table 6: Accuracy (mloU) and computation (GFLOPs) on Cityscapes semantic segmentation for MDEQ models, at different iterations.
17

---

Published as a conference paper at ICLR 2022
Ieration
Teration 6
lleration 8
Ieraaion 10
Neertton 4
lteration 8
lleration 8
lleration 10
Iteration 14
Iteration 18
Iteration 22
lieration 26
ntertion 4
Ieertion 6
Iteration 8
Itenstion 10
Iteration 14
Iteration 18
Iteration 22
lleration 26
Figure 9: Cityscapes prediction results for MDEQ -mall all various iterations. Input and ground truth are in Fig.
18

---

Published as a conference paper at ICLR 2022
Neration
Teralion
lleration 8
Ieration 10
Neration 4
lteration 8
lleration 8
lleration 10
Iteration 14
Iteration 18
Iteration 22
lieration 26
ntrrtion 4
Ieertion 6
Iteration 8
Itenstion 10
Iteration 14
Iteration 18
Iteration 22
lleration 26
Figure 10: Cityscapes prediction results for MDEQ-Smal all various iterations. Input and ground truth are in Fig.
19


Reviews:
[{"review_id": "fCehCT2kzUJ", "reviewer": "Reviewer_GYrC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is thoughtful and demonstrates good empirical performance for anytime pixel-level recognition. This paper is also very well-written.\n\nThe proposed method is not friendly for hardwares like GPU. But I like the part where the authors implement their method and demonstrate speedup on the CPU. One small clarification: what’s the metric used here to evaluate the computation on CPU? Is that the end-to-end latency? \n \nMy main concern is its technical novelty. Early exits have been explored in many previous work, e.g. Multi-scale DenseNet for anytime image classification (Huang et al., 2017). While pixel-level recognition is definitely more challenging than image classification, the solution proposed in this work is still similar to the spirit of previous work.\n\nAlso, it seems that most gain brought by the proposed method comes from the “confidence adaptivity” strategy. I find the interpolation idea really interesting and promising. But as mentioned by the authors, this strategy was originally proposed by Xie et al., 2020 (ECCV). So this downgrades the contribution of this work.\n\nIn Table 1, why do we need to show the average performance across exits? I feel the curve in Figure 3 would provide a more complete picture of the performance & efficiency. But the average performance is not really informative here.\n\nDoes the method generalize to other architectures other than HRNet? It would be helpful to see the proposed idea can work on more than one architecture family (even though HRNet is the current SOTA for segmentation).", "summary_of_the_paper": "This work focuses on anytime pixel-level recognition (e.g., semantic segmentation). They propose to add intermediate exists in the architecture for anytime inference. They also consider spatial confidence adaptivity in their network, where they only execute subsequent layers on a small set of non-confidence pixels and obtain the features of other positions via interpolation. They apply the method to semantic segmentation and human pose estimation and demonstrate a reduction in FLOPs and good anytime performance for both tasks.  \n", "summary_of_the_review": "My main concern is that the technical novelty of this work is somewhat limited. Other than that, this work is solid and has comprehensive results and analysis. So, I feel this work is borderline and am slightly towards borderline accept.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JRf1WPov73l", "reviewer": "Reviewer_hS3f", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\ni) Submission includes well-documented code.\n\nThe code is included in submission and the paper has an anonymous github link. README.md describes setup steps.  The code itself is reasonably well-organized.  Though I did not see many comments or docstrings, the names are reasonable and there are at least some logging messages that help make it clear what's going on in parts. Provided sample commands seem to work (at least as far as training).\n\nMinor comment: Sample commands appear to use the authors' specific GPU setup, this seems unnecessary. It's good to have simple README sample commands targeting the least common denominator to debug environment setup and dependency installation. Though the minimal command may not be that simple due to calls to torch.distributed in the source.\n\nii) Description of adaptive/early exit method is overall very clear.\n\nFigure 2 makes the high-level idea very clear, showing where the early exits and masking is done. The novel modules are described precisely in mathematical notation in the \"Confidence Adaptivity\" subsection.\n\niii) Good FLOPs/accuracy tradeoff experiments.\n\nExperiments look at the pareto frontier of computational cost (as FLOPs) vs accuracy (as mIOU or probability of correct keypoint). They include a reasonable baseline, MDEQ that also varies in the tradeoff. Results are good, showing that the authors' method dominates the baseline.\n\nMinor Point: Varying the HRNet architecture for mulitple tradeoff points also seems like a good baseline to illustrate here, rather than plotting only the single backbone used.\n\n**Weaknesses**\n\niv) Relatively expensive early exits.\n\nThe early exits must have multiple layers of downsampling & upsampling, as described in the \"Head Redesign\" subsection. It does seem necessary to do this apply this to segmentation, and the authors give a good description of the reasoning. This \"redesign\" seems like an important novelty of the submission. The heavyweight early exists maybe make this harder to apply than the very simple early exits in classification work such as Veit & Belongie. The authors do still show some computational savings, though, so this extra cost is clearly not severe enough to invalidate the method.\n\nv) Some unclear parts of interpolation.\n\nThe description of the interpolation, in the span between and including equations (2)-(3), does not make the handling of the mask that clear. Will zeroed-out features be included in the average for I(f_{out})? The weighting in (3) seems to only take distance into acount, not the value of the mask.\n\nvi) Limited empirical handling of wall-clock time.\n\nThe submission describes the considerations behind the decision to report FLOPS at the bottom of page 7. The authors correctly note that reporting FLOPs is common, and that this is due to the need for lots of work on the acceleration of sparse computations in underlying libraries and hardware (see also \"The Hardware Lottery\" by Hooker). But the most recent work on this that they cite, such as Elsen et. al. 2020, may be worth considering in experiments: it is fairly well-optimized and has a mature implementations. So we'd expect that the best possible likely speedups in the near future are at least not that far from what would be achieved with the implementations of the past 1-2 years. In some experiments, such as the HRNet-W48 network on Cityscapes, the advantage of early exiting looks slim enough that it seems doubtful one would see wall-clock speedups even with fairly good acceleration of the sparse computations.\n\nThe video included in the submission also illustrates the real timings, and shows that the wall-clock time is probably at least reasonable: but this is less useful than quantitative experiments.\n\n**Minor Comments**\n\n  * \"this strategy work well\" -> \"this strategy works well\" at bottom of page 3.\n  * It's usually better to number *all* equations, for instance on page 5. This makes it easier for readers (and reviewers!), or paper citing this one, to reference specific equations, even if that equation isn't cross-referenced in the paper itself.", "summary_of_the_paper": "The paper presents adaptive and anytime methods for semantic segmentation and pose recognition. Both of these are \"pixel-level\" tasks where the model is expected to output a prediction for each pixel in the input image. Their adaptive method, that performs variable amounts of computation depending on the input and the budget, with the amount of computation devoted to a given pixel varying from one part of the image to another. This is done by adding early exits to the base model architecture, and modifying the convolution layers to perform sparse computation on a subset of locations followed by interpolation. Experiments are done on Cityscapes semantic segmentation and MPII pose estimation benchmarks.", "summary_of_the_review": "The method is interesting and presented well. Weaknesses in presentation and experiments are mostly lower-level and affect only parts of the method and its description or validation, and are not central to the overall paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GJ4ucR8aG_E", "reviewer": "Reviewer_2wqr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The paper is well written.\n+ The motivation behind the proposed problem is clear and convincing.\n+ The results are reasonable, i.e., the authors are able to match or improve the HRNet baseline while reducing the computational complexity.\n\nWeaknesses:\n- I'm not convinced of the novelty of the \"anytime\" prediction problem. The authors might be the first to formulate it as an actual research problem. However, the main requirement of this problem is to predict segmentation maps sequentially / progressively. It seems that there have already been several much older methods that do this, i.e., \"Holistically-Nested Edge Detection\" (ICCV 2015), which uses deep supervision to predict dense edge maps at every stage of the network, \"DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs\" (ICLR 2015), which also produces segmentation maps at every resolution level, etc. I'm sure there are many other methods that do this, but these are the first ones / most popular ones that come to mind. The authors in these papers don't explicitly discuss the \"anytime\" prediction problem, but technically one could adapt these methods to this problem quite easily.\n- I'm also not convinced of the novelty of the proposed approach. The exit branches seem very similar to standard prediction branches attached to different stages of the network (as in the two papers listed above). Furthermore, the concept of confidence adaptivity shares lots of similarity with prior methods such as \"PointRend: Image Segmentation as Rendering\", and \"Not All Pixels Are Equal: Difficulty-Aware Semantic Segmentation via Deep Layer Cascade\", both of which also focus the computation of their model towards pixels that are not predicted confidently. Of course, there are subtle differences between the proposed approach and these prior methods. However, the main concepts are quite similar, which diminishes the technical novelty of the proposed approach.\n- The experiments are lacking. Considering that the authors claim that they are the first ones to perform evaluations on this problem, I would expect more extensive experiments with different models/backbones (not just with HRNet, which is a bit outdated at this point). I'm not familiar with the latest state-of-the-art in semantic segmentation, but it would be useful to include at least several additional models to demonstrate the generality of the proposed approach.\n- I'm confused why the accuracy in Table 1 improves when adding the  Confidence Adaptivity component. My intuition was that this should reduce the computational cost of the method, but that it should also decrease the accuracy (as we are only focusing on a subset of pixels in the future maps). Therefore, I'm confused why the accuracy would actually increase in this case.\n- From Figure 5, it doesn't seem that the downsampling strategy at each exit matters that much, i.e., for three out four exits, the 3/3/3 strategy works as well as the proposed 3/2/1 scheme. \n", "summary_of_the_paper": "The paper proposes a new task called \"anytime\" prediction, which requires a model to make a progression of predictions, which might be halted at any time. The authors then also introduce an end-to-end model for this problem. The main two components behind the proposed model are: (1) a cascade of “exits” enabling the model to make progressive predictions while taking into account accuracy vs computational cost tradeoff; (2) Confidence Adaptivity, which allows the model to focus on the less confident pixel prediction. The authors implement their approach using HRNet baseline and demonstrate improved performance and efficiency on semantic segmentation and pose prediction tasks.", "summary_of_the_review": "I'm not convinced that the proposed problem is novel. Furthermore, considering the conceptual similarity to several prior methods (see the discussion above), I believe that the technical novelty of the paper is also quite limited. Lastly, in my opinion, the current experiments are lacking. Therefore, I would recommend rejecting the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8m5bBAsi58I", "reviewer": "Reviewer_ohPA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The task setup is interesting and of great practical value, as well-motivated in the paper. \n+ The proposed method is simple, and makes intuitive sense, also the results have shown that it maintains a better accuracy-computation tradeoff, at least with the theoretic FLOPs measure. \n+ The authors have open-sourced their implementation. \n+ The writing is very clear and easy to follow. \n\nWeaknesses:\n- First, I appreciate that the authors discussed the matter of FLOPs vs runtime on page 7. Though, I'm a bit disappointmented about the fact that the proposed method does not achieve wallclock speedup with GPUs. As for author's argument about the potential of the proposed method with more development on sparse convolution, it holds with the assumption that it would be a cost-effective route to invest more into software/hardware support for sparse convolution, compared to investing more on developing models that are more friendly to batch computation and parallel hardware. In short, I'd be more convinced if I can see some wallclock acceleration from the proposed approach, even if it's far lower than its theoretical FLOPs speedup due to hardware constraints.  \n- An important ablation that's missing is for the interpolating neighborhood size gamma, as I'd imagine it has a large effect on interpolating quality vs computation trade-off. In fact, I cannot find in the paper what's the default value used in the experiments. \n- It's not clear from the texts whether the confidence adaptivity mechanism is used in both training and testing? Or only during testing? It would be interesting to show results for both settings. \n- Though it's shown as a visualization on specific samples (Figure 4), it would be helpful to also show quantitatively the portion of pixels that exceed thresholds at each stage. \n- Cityscapes is relatively small and only with limited number of classes, it would make the results more convincing if the authors could repeat their experiments also on large datasets like ADE.  \n- Table 1, when comparing \"Early Exiting + RH (HRNet)\" and \"Early Exiting + RH + CA (HRNet)\", the gain mainly comes from exit 1, in fact if you average exit 2, 3, 4, you will get almost the same accuracy (72.1 vs 72.2), does this mean that the utility of RH is mainly to provide more layers for exit 1 so that it can produce a high enough modeling capacity? \n- Page 7, \"our model with confidence values as mask indicators also outperforms the feature-based mask sampling method in accuracy-computation tradeoff\", when considering comparing to feature-based mask sampling method, shouldn't the correct comparison be \"feature-based mask sampling\" vs. \"Early Exiting (HRNet)\"? As my understanding is that \"feature-based mask sampling\" is not using an adaptive head for each exit and thus not a fair comparison to \"Early Exiting + RH (HRNet)\". \n\nQuestions/concerns:\n- For the re-designed head, the authors mention that only 1x1 conv is used for both the encoder and decoder, would this be too limiting? Especially for earlier exits where the receptive fields of features are not large enough. \n- Table 1, for the entry \"Early Exiting (HRNet)\", it's not clear what does the head design look like in this case? Are they with similar parameter counts? \n- Table 1, as I mentioned before, it's not clear whether the entry \"Early Exiting + RH + CA (HRNet)\" use CA both in training and testing? \n\nTypos:\n- Page 2, \"multiple predictors branch of from\" --> \"branch off\" \n\n", "summary_of_the_paper": "This paper proposes an anytime method for pixel recognition tasks like semantic segmentation and human pose estimation. The key idea is to design a network with multiple \"early exits\", from which the network could make predictions using a corresponding prediction head. Thus, depending on criteria like budget or confidence, the network could decide where to exit and therefore leads to different accuracy-computation operating points (i.e. its anytime property). \n\nThe technical contributions are mainly on two parts. The first contribution is a re-design of prediction heads (denoted as RH in the paper) at each early exits, mainly to tackle the difference of granularity for different intermediate feature maps. Another contribution is a confidence-based adaptive filtering mechanism that decides how to allocate computation budgets across spatial regions. Specifically, at each exit, the max prediction scores are used as a measure such that all spatial pixels that exceeds certain score threshold will not be processed in later stages. \n\nThe authors evaluated their methods on two tasks (Cityscapes for semantics segmentation and MPII for human pose estimation) and demonstrated that their full-blown method yields a better accuracy-computation tradeoff, compared to variants of the proposed approach and several previous methods.  \n\n------------------------- POST REBUTTAL -------------------------\n\nThe authors have addressed most of my concerns in their response. Also, after some discussions on the issue of lacking positive signals from wallclock time metric, I buy the arguments to position this work as a forward looking exploration in the alternative space to methods that are chosen in current hardware lottery. With this, I will raise my rating to positive inclined.\n\n\n\n", "summary_of_the_review": "The proposed approach is simple and has nice accuracy-FLOPs tradeoff. However, the proposed approach does not seem to yield actual wallclock speedup due to its incompatibility with current parallel chips (GPUs). I'm not an expert in HPC, but I'm not sure about the assumption that this will change in near future. Without much positive signal on that, I'm doubtful about this general direction.   \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received 4 final ratings above the acceptance threshold: 6, 6, 6, 8. The reviewers mentioned limited novelty, but acknowledged practical importance of this work, and particularly appreciated thorough analysis provided by the authors. After a strong rebuttal, most of remaining concerns have been addressed.
The final recommendation is therefore to accept this submission as a poster.
Decision: Accept (Poster)