Confidence Adaptive Anytime Pixel-Level Recognition

1 INTRODUCTION . Deep convolutional networks ( Krizhevsky et al. , 2017 ; He et al. , 2016 ) achieve high accuracy but at significant computational cost . Their computational burden hinders deployment , especially for time-critical or low-resource use cases that for instance require interactivity or inference on a mobile device . This efficiency problem is tackled by special-purpose libraries ( Chetlur et al. , 2014 ) , compression by network pruning ( Han et al. , 2015 ; Li et al. , 2016 ; Liu et al. , 2019 ) and quantization ( Rastegari et al. , 2016 ; Jacob et al. , 2018 ) , and adjusting architecture by distillation ( Hinton et al. , 2015 ; Romero et al. , 2014 ) . These solutions accelerate network computation but the entire network must still be computed ; however , a prediction may be needed sooner . Time constraints vary , but the inference time of a standard deep network does not . Anytime inference mitigates this issue by bringing flexibility to model computation . An anytime algorithm ( Dean & Boddy ) gradually improves its results as more computation time is given . It can by interrupted at any point during its computation to return a result as system or user requirements demand . In this way the time to the first output is reduced while the quality of the last output is preserved . An anytime model makes a progression of predictions between the first and last . This pro- gression continues if time remains , or halts if it is either already satisfactory or out of time . For example , consider a user on a mobile device : an approximate result could be returned earlier if there is urgency , or the user could monitor the sequence of predictions as time goes by and stop the model once it is good enough . Note that anytime inference differs from adaptive or dynamic inference ( Veit & Belongie , 2018 ; Wu et al. , 2018 ; Wang et al. , 2018a ) where the model decides how much to compute instead of an external decision . Prior research has explored anytime inference by feature selection ( Karayev et al. , 2014 ) or ensembling models through boosting ( Grubb & Bagnell , 2012 ) . For end-to-end models , research has focused on classification for anytime inference or adaptive inference . In particular , the multi-scale dense network ( Huang et al. , 2017 ) is an architecture for resource efficient classification . The attraction of anytime inference is not limited to classification however , and the additional computation required for pixel-level recognition tasks makes it even more desirable . For instance , an autonomous driving system may demand swifter reaction time for safety in the presence of pedestrians , and so an anytime semantic segmentor might sooner recognize their presence . In addition to urgency , an anytime segmentor could help efficiency , by not further processing already confident predictions of street pixels and therefore save power . In this work , we develop the first single-model anytime approach for pixel-level visual recognition tasks . We adopt an early exiting framework , where multiple predictors branch off from the intermediate stages of the model . The exits are trained end-to-end ( both the original exit and intermediate exits ) , and during inference each provides a prediction in turn . To compensate for differences in depth and spatial dimensions across stages , we redesign the predictors for earlier exits . For each exit we choose an encoder-decoder architecture that combines pooling , convolution , and interpolation to enlarge receptive fields and smooth spatial noise . Exits might suffice for anytime image classification , but pixel-level tasks have spatial structure . Simple regions may need less processing while complex regions need more . Standard inference applies an equal amount of computation at every pixel without taking advantage of spatial structure . Spatially adaptive networks ( Figurnov et al. , 2017 ; Xie et al. , 2020 ) improve efficiency by skipping computation in places , but unlike anytime models they only make one final prediction . We propose a spatially anytime network that makes predictions for different pixels at different points in time . Our spatially adaptive inference scheme decides whether or not to continue computation at each exit and position . We mask the output of each exit by thresholding the confidence of its predictions : the remaining computation for sufficiently confident pixels is then reduced ( Fig . 2 ) . For each masked pixel , its prediction will be persisted in the following exits , as it is already sufficiently confident . In following layers , the features for the masked pixel will be interpolated , rather than convolved , and therefore reduce computation . The confidence measure can depend on the task , e.g. , in segmentation it could be the entropy of class predictions . This confidence adaptivity can substantially reduce the total computation while maintaining accuracy . We experiment with two pixel-level visual recognition tasks : Cityscapes semantic segmentation and MPII human pose estimation . Our networks are based on the recent HRNet architecture ( Wang et al. , 2020 ) . Redesigning the exits and including confidence adaptivity significantly improves across accuracy-efficiency operating points . Our full model not only makes anytime predictions , but its final predictions achieve the same level of accuracy as the base model with 40-60 % less total computation . For analysis , we visualize predictions and confidence adaptivity across exits , and ablate design choices for the exits and masking . To summarize our contributions : ( 1 ) we propose the first general-purpose anytime inference approach for pixel-level visual recognition ; ( 2 ) our redesigned exit architecture and confidenceadaptive masking boost the accuracy-efficiency tradeoff of pixel-wise anytime inference ; ( 3 ) we experiment with semantic segmentation and human pose estimation to show efficacy across tasks ; and ( 4 ) we analyze our design choices by visualization and ablation . 2 RELATED WORK . Anytime Inference . Anytime algorithms ( Zilberstein , 1996 ; Dean & Boddy ) can be interrupted at any point during computation to return a result , whose quality improves gradually with more computation time . In machine learning , anytime inference has been achieved by boosting ( Grubb & Bagnell , 2012 ) , reinforcement learning ( Karayev et al. , 2014 ) , and random forests ( Fröhlich et al. , 2012 ) . Anytime deep networks have been brought to bear on image recognition , but not pixel-level recognition . Branching architectures have been a common strategy ( Amthor et al. , 2016 ; Teerapittayanon et al. , 2016 ) along with other techniques such as adaptive loss balancing ( Hu et al. , 2019 ) . While there is work on the tasks of person re-identification ( Wang et al. , 2019 ) and stereo depth ( Wang et al. , 2018b ) , these techniques are task specific , while our method applies to multiple pixellevel tasks , as we show with semantic segmentation and pose estimation . Liu et al . ( Liu & He , 2016 ) learn a hierarchy of models for anytime segmentation , but its multiple models complicate training and testing , and require more memory . Our work instead augment the base model architecture for simplicity and efficiency . The PointRend method ( Kirillov et al. , 2020 ) outputs a initial dense prediction first and then refine it adaptively , but the predictions are all made at full depth . Its majority of the computation is spent before the first output and thus can not be practically “ anytime ” . Our method is the first to selectively update anytime predictions across space and layers . Adaptive Computation . An adaptive model adjusts its computation to each specific instance during inference . For deep networks , this is often done by adjusting which layers to execute , that is , choosing which layers to run or skip . This can be done by a supervised controller ( Veit & Belongie , 2018 ; Liu & Deng , 2017 ) , a routing policy optimized by reinforcement learning ( Wang et al. , 2018a ; Wu et al. , 2018 ; Lin et al. , 2017a ) , or other training strategies ( McGill & Perona , 2017 ) . Rather than choosing layers , spatial adaptivity chooses where to adjust the amount of computation across different spatial positions in the input . For example , the model could infer spatial masks for feature maps and skip computation on masked areas ( Shomron et al. , 2019 ; Dong et al. , 2017 ; Lin et al. , 2017b ; Ren et al. , 2018 ; Cao et al. , 2019 ) . Figurnov et al . ( 2017 ) maintains a halting score at each pixel and once it reaches a threshold the model will stop inference at those positions for spatially coarse tasks like classification or bounding box detection . Xie et al . ( 2020 ) stochastically sample positions for computation from an end-to-end learned sampling distribution . Li et al . ( 2017 ) convert a deep network into a difficulty-aware cascade , where earlier steps handle easier regions and later steps tackle harder regions . These spatially adaptive models reduce computation , but are not anytime : they do not make a series of predictions and can not be interrupted . 3 APPROACH . Anytime Setting and the Early Exiting Framework . In an anytime inference setting , the user can stop the inference process based on the input or a current event . Thus the computation budget for each instance x could be time or input dependent . We use B ( x , t ) to denote the computation budget assigned for instance x at time t , where the time variable t models events that can change the budget . B ( x , t ) could be independent of x , i.e. , the budget only depends on the time t , for example if a model on a server is asked to make predictions with less budget during high-traffic hours ; B ( x , t ) can also be independent of t , meaning the budget is only decided by input x , regardless of external events . The output of the anytime model depends on the budget given , and we denote it as f ( x , B ( x , t ) ) . Assuming L is the task loss and y is the ground truth , the per-instance loss is L ( f ( x , B ( x , t ) ) , y ) . This leads to the expected training loss to be E ( x , y ) ∼ ( X , Y ) , t∼T [ L ( f ( x , B ( x , t ) ) , y ) ] , where ( X , Y ) is the input-output joint distribution and T is the distribution modeling the time or event variable . Next , we introduce the early exiting framework which have been used in prior works ( Huang et al. , 2017 ; Teerapittayanon et al. , 2016 ) for anytime prediction . Standard convolutional networks only have one prediction “ head ” at its final stage . The network takes the input x , forwards it through intermediate layers , and finally output the prediction at its head . The concrete form of the head depends on the task . For pixel-level recognition , the head is usually one or multiple convolutions that output spatial maps representing pixel-wise predictions . To obtain an anytime model , we attach multiple heads to the network , branching from its intermediate features ( Fig . 2 ) . We call these additional heads early exits , since they allow the network to give early predictions and stop the inference at the current layer . Suppose we add k early exits at intermediate layers with layer indices l1 . . . , lk . We denote the intermediate features at these layers Fl1 ( x ) . . . , Flk ( x ) , and the functions represented by the early exits E1 . . . , Ek . Note that Eis may be of the same form but they do not share weights . The early prediction maps can be denoted as ŷi = Ei ( Fli ( x ) ) , i = 1 . . . k. Together with the original final prediction ŷk+1 , the total loss is : Ltotal = k+1∑ i=1 wiL ( ŷi , y ) ( 1 ) where wi is the weight coefficient at exit i . The original network , together with the added exits , will be trained end-to-end to optimize this total loss function . In experiments , we set all weights equal to 1 . This corresponds to the minimization of the expected loss in Sec . 3 when the exiting probabilities at all exits are equal . We find this to be a simple yet effective scheme . For anytime inference , as the network propagates features through its layers , if the computation budget is reached or the user asks the model to stop , it will output the latest ŷi that is already computed . Similar early exiting strategies have been used in resource-efficient image classification ( Teerapittayanon et al. , 2016 ; Huang et al. , 2017 ) , but pixel-level recognition requires further steps detailed in the following subsections . Head Redesign . Typical convolutional networks have a hierarchical structure that begins with shallow , fine , and more local features and ends with deep , coarse , and more global features . These deeper features represent more image content by their larger receptive fields . For pixel-level tasks , upsampling is done within the network to restore lost resolution during downsampling , and ensure precise spatial correspondence between the input and the output . This upsampling can be accomplished in few ( Long et al. , 2015 ) or many ( Zhao et al. , 2017 ) layers , but no matter the architecture , the network learns its most local features in its earliest layers . This presents a challenge for the earliest exits , since these features are limited in depth and receptive field . Making direct predictions at these exits with the typical 1×1 convolution head produces spatially noisy and inaccurate results . To compensate for these lacking early features , we redesign the prediction heads for the exits Ei . Each Ei first downsamples its input features Fli ( x ) , through a series of pooling and 1×1 convolution layers . Each pooling operation halves the spatial resolution , increasing its output ’ s receptive fields . The following convolution provides the opportunity to learn new coarser-level features , specifically for that exit ’ s prediction . After several ( denoting this number as D ) “ pool-conv ” layers , we upsample the features back to the original output resolution , with an equal number ( D ) of bilinear interpolation and 1× 1 convolution layers . The output of this “ interpolate-conv ” sequence will be the prediction ŷi at this exit . This is important for ensuring the spatial accuracy of the prediction for pixel-level tasks . Our redesigned exits are essentially small “ encoder-decoder ” modules ( Fig . 2 ) , where the encoder downsamples the features , the decoder upsamples them back . The downsampling ratio at each exit is determined by D , the number of consecutive “ pool-conv ” layers . Intuitively , features at earlier layers are more fine-level , and the exit branching from them can potentially benefit from more downsampling . In experiments , we use encoder with D = N − i downsampling operations at exit i , where N is the total number of exits , including the original last exit . Empirically we find this strategy works well , and alternative strategies are compared in Sec . 5 . Finally , in all early exits , the first convolution will transform the number of channels to a fixed number for all exits . By setting the channel width relatively small , we can still save computation while adding layers with this redesigned encoder-decoder head structure . Confidence Adaptivity . For pixel-level recognition tasks , any early prediction ŷi is a spatial map consisting of pixel-wise predictions at each position . While most convolution networks spend equal amount of computation at each input position , it is likely that recognition at some regions are easier than others , where the network can make predictions with a high confidence even at earlier exits . For instance , the inner part of a large sky segment may be easy to recognize , whereas the boundary between the bicycle and the person riding it may need more careful delineation . Once an early prediction is made , we can inspect the “ confidence ” at each position . As an example , for semantic segmentation , the maximum probability over all classes can serve as a confidence measure . If the confidence has passed a pre-defined threshold at certain positions ( green squares on predictions in Fig . 2 ) , we may decide these predictions are likely to be correct , and not continue the computation of further layers at this position . Suppose the pixels of the early prediction ŷi are indexed by p , we form a mask Mi : Mi ( p ) = { 0 , if Confidence ( ŷi ( p ) ) ≥ Threshold 1 , otherwise ( 2 ) For any convolution layer between exit i ( Ei ) and the next exit i + 1 ( Ei+1 ) , we could choose whether to perform or skip computation at position p based on the mask ( Fig . 2 ) . Assuming C is a convolution layer with input fin , then by applying the mask , the output fout at position p becomes : fout ( p ) = { C ( fin ) ( p ) , if Mi ( p ) = 1 , 0 , if Mi ( p ) = 0 . ( 3 ) If C ’ s output and the mask Mi do not share the same spatial size , we interpolate ŷi in Eqn . 2 to the size of C ’ s output , so that the mask Mi is compatible with C in Eqn . 3 . The output fout could be sparse , with many positions being 0 . This could potentially harm further convolutional computation . To compensate for this , we spatially interpolate these positions from their neighbors across all channels , using a similar approach as in ( Xie et al. , 2020 ) . Denoting the interpolation operation as I , the final output feature f∗out is f∗out ( p ) = { fout ( p ) , if Mi ( p ) = 1 , I ( fout ) ( p ) , if Mi ( p ) = 0 . ( 4 ) Here , the value of I ( fout ) ( p ) is a weighted average of all the neighboring pixels centered at p within a radius r : I ( fout ) ( p ) = ∑ s∈Ω ( p ) W ( p , s ) fout ( s ) ∑ s∈Ω ( p ) W ( p , s ) ( 5 ) where s indexes p ’ s neighboring pixels and Ω ( p ) = { s|∥s− p∥∞ ≤ r , s ̸= p } , the neighborhood of p. We set radius r = 7 in our experiments . W ( p , s ) is the weight assigned to point s for interpolating at p , for which we use the RBF kernel , a distance-based exponential decaying weighting scheme : W ( p , s ) = exp ( −λ2∥p− s∥22 ) ( 6 ) with λ being a trainable parameter . This indicates that the closer s is to p , the larger its assigned weight will be . Note that masked-out features ( Mi ( p ) = 0 ) still participate in the interpolation process as inputs with values of 0 . Replacing filtering by interpolation at these already confident spatial locations ( Mi ( p ) = 0 ) could potentially save a substantial amount of computation . The mask Mi will be used for all convolutions between exit i and i+ 1 , including the convolutions inside exit i+ 1 . Once the forward pass arrives at the next exit , to make the prediction ŷi+1 , the last prediction at positions where Mi ( p ) = 0 will be carried over , having already been deemed confident enough at the last exit and having been skipped during further computations . This means : ŷi+1 ( p ) = { Ei+1 ( Fli+1 ( x ) ) , if Mi ( p ) = 1 , ŷi ( p ) , if Mi ( p ) = 0 . ( 7 ) The network then calculates a new mask Mi+1 based on ŷi+1 , and uses it to skip computation going forward . The process continues until we reach the final exit . In summary , we incorporate spatial confidence adaptivity into the early exiting network , by not filtering at spatial locations that are already sufficiently confident in the latest prediction . At these positions interpolation is used instead , at much reduced computational cost , to avoid excessive sparsity . Unless otherwise specified , confidence adaptivity is used in both training and inference .

Summary: This paper proposes an anytime method for pixel recognition tasks like semantic segmentation and human pose estimation. The key idea is to design a network with multiple "early exits", from which the network could make predictions using a corresponding prediction head. Thus, depending on criteria like budget or confidence, the network could decide where to exit and therefore leads to different accuracy-computation operating points (i.e. its anytime property).   The technical contributions are mainly on two parts. The first contribution is a re-design of prediction heads (denoted as RH in the paper) at each early exits, mainly to tackle the difference of granularity for different intermediate feature maps. Another contribution is a confidence-based adaptive filtering mechanism that decides how to allocate computation budgets across spatial regions. Specifically, at each exit, the max prediction scores are used as a measure such that all spatial pixels that exceeds certain score threshold will not be processed in later stages.   The authors evaluated their methods on two tasks (Cityscapes for semantics segmentation and MPII for human pose estimation) and demonstrated that their full-blown method yields a better accuracy-computation tradeoff, compared to variants of the proposed approach and several previous methods.    ------------------------- POST REBUTTAL -------------------------  The authors have addressed most of my concerns in their response. Also, after some discussions on the issue of lacking positive signals from wallclock time metric, I buy the arguments to position this work as a forward looking exploration in the alternative space to methods that are chosen in current hardware lottery. With this, I will raise my rating to positive inclined.