Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models
Authors: Xiaofang Wang
Dan Kondratyuk
Eric Christiansen
Kris M. Kitani
Yair Movshovitz-Attias
Elad Eban
Venue: ICLR 2022 Poster (2022)
Keywords: Ensemble
Cascade
Efficiency

Abstract: Committee-based models (ensembles or cascades) construct models by combining existing pre-trained ones. While ensembles and cascades are well-known techniques that were proposed before deep learning, they are not considered a core building block of deep model architectures and are rarely compared to in recent literature on developing efficient models. In this work, we go back to basics and conduct a comprehensive analysis of the efficiency of committee-based models. We find that even the most simplistic method for building committees from existing, independently pre-trained models can match or exceed the accuracy of state-of-the-art models while being drastically more efficient. These simple committee-based models also outperform sophisticated neural architecture search methods (e.g., BigNAS). These findings hold true for several tasks, including image classification, video classification, and semantic segmentation, and various architecture families, such as ViT, EfficientNet, ResNet, MobileNetV2, and X3D. Our results show that an EfficientNet cascade can achieve a 5.4x speedup over B7 and a ViT cascade can achieve a 2.3x speedup over ViT-L-384 while being equally accurate.

However, when com- Published as a conference paper at ICLR 2022
WISDOM OF COMMITTEES: AN OVERLOOKED APPROACH To FASTER AND MORE ACCURATE MODELS
Xiaofang Wangl
Dan Kondratyuk"
Eric Christiansen'
Kris M. Kitani
Yair Alon (prev. Movshovitz-Attias)
Elad Eban'
Google Research *Carnegie Mellon University
{xiaofan2, kkitani fccs. cmu. edd (dankondratyuk, eri ame,me, yairmov, eeodeeegoo
ABSTRACT
Committec-basce models (ensembles or cascades) construct models by combining existing pre-trained ones. While ensembles and cascades are well-known tech- niques that were proposed before deep learning, they are not considered core building block of deep model architectures and are rarely compared to in recent literature on developing efficient models. In this work, we go back to basics and conduct a comprehensive analysis of the efficiency of committee-baser models. We find that even the most simplistic method for building committees from exist- ing, independently pre-trained models can match exceed the accuracy of state- of-the-art models while being drastically more efficient. These simple committee- based models also outperform sophisticated neural architecture search methods (e.g., BigNAS). These findings hold true for several tasks, including image classi- fication, video classification, and semantic segmentation, and various architecture families, such as VIT, EfficientNet, ResNet, MobileNetV2, and X3D. Our results show that an EfficientNet cascade can achieve 5.4x speedup over B7 and ViT cascade can achieve 2.3x speedup over ViT-L-384 while being equally accurate.
INTRODUCTION
Optimizing the efficiency of neural networks important for real-world applications they can only use limited computational resources and often have requirements on response time.
There has been considerable work in this direc-
usc Pecorr 196
tion (Howard al.) [017]] /Zhang et al.) [2088] Tan & Le| 2019), but they mostly focus design- ing novel network architectures that can achieve favorable speed-accuracy trade-off. Here, we do not present any novel method or architecture de- sign. Instead, we focus analyzing the accuracy and efficiency of a simple paradigm: committee- based models. We the term "committee" refer to model ensembles or cascades, which in- dicates that they are built using multiple indepen- dent models.
Reelwe112
5igg Medkla Casrades
D2
Lupdee 50
23
25
10
66
Figure 1: Committee-based models achieve higher accuracy than single models on Ima- geNet while using fewer FLOPs. For example, although Inception-v4 ( ('Incep-v4") outperforms all single ResNet models, ResNet cascade can still outperform Incep-v4 with fewer FLOPs.
Committec-basse models have been extensively studied and used before deep learning (Breiman 1996 Schapire
1990. Freund Schapire 1997
Viola & Jones 2001
Work done during an internship at Google. Work done part of the Google AI Residency Program.

---

Published as a conference paper at ICLR 2022
paring the efficiency of deep models, committee-aase models are rarely considered in recent work (Howard et al.) 2017] Zhang et al., 2018: Tan & Le. 2019). There still lacks a systematic understanding of their efficiency comparison with single models models that only use one net- work. Such an understanding is informative for both researchers to push the frontier of efficient models and practitioners to select model designs in real-world applications.
To fill this knowledge gap, we conduct a comprehensive analysis of the efficiency of committee- based models. To highlight the practical benefit of committee-based models, we intentionally choose the simplest possible method, which directly uses off-the-shelf, independently pre-trained models to build ensembles or cascades. We ensemble multiple pre-trained models via a simple average over their predictions (Sec.B) 3). For cascades, we sequentially apply each model and use simple heuristic (e.g., maximum probability in the prediction) to determine when exit from the cascade (Sec. 4
We show that even this method already outperforms state-of-the-are architectures found by costly neural architecture search (NAS) methods. Note that this method works with off-the-shelf models and does not use specialized techniques. For example, it differs from Boosting (Schapire 1990) where each new model is conditioned on previous ones, and does not require the weight generation mechanism in previous efficient ensemble methods (Wen et al. 2020). This method does not require the training of an early exit policy (Bolukbasi et al. 2017; Guan et al. 2018) or the specially designed multi-scale architecture (Huang et al.. 2018) in previous work on building cascades.
To be clear, the contribution of this paper is not in the invention of model ensembles and cascades, as they have been known for decades, and is not in new proposed method to build them. Instead, it is in the thorough evaluation and comparison of committee-basse models with commonly used model architectures. Our analysis shows that committee-based models provide a simple complementary paradigm to achieve superior efficiency without tuning the architecture. One can often improve accuracy while reducing inference and training cost by building committees out of existing networks.
Our findings generalize to a wide variety of tasks, including image classification, video classifica- tion, and semantic segmentation, and hold true for various architecture families: ViT (Dosovitskiy 20 EfficientNet (Tan & Lc 2019), ResNet (He et al. 2016)) MobileNetV2 (Sandler et al.) and X3D (Feichtenhoter) 2020). We summarize our findings as follows:
et al.
2018
Ensembles are more cost-effective than single model in the large computation regime (Sec. 3 For example, an ensemble of two separately trained EfficientNet-BB B5 models matches B7 accuracy, a State-of-the-an ImageNet model, while having almost 50% less FLOPs (20.5B vs. 37B).
Cascades outperform single models in all computation regimes (Sec. 5 Our cascade matches B7 accuracy while using average 5.4x fewer FLOPs. Cascades over ViT-L-384, a Transformer architecture, while matching its accuracy on ImageNet.
also achieve a 2.3x speedup
We further show that (1) the efficiency of cascades is evident in both FLOPs and on-device latency and throughput (Sec. [:.I): (2) cascades can provide a guarantee on worst-case FLOPs (Sec. (522); (3) one can build self-cascades using a single model with multiple inference resolutions achieve a significant speedup ().
Committee- based models are applicable beyond image classification (Sec. and outperform single models on the task of video classification and semantic segmentation. Our cascade outper- forms X3D-XL by 1.2% on Kinetics-600 (Carreira al. 2018) while using fewer FLOPSs
2 RELATED WORK
Efficient Neural Networks. There has been significant progress in designing efficient neural net- works. In early work, most efficient networks, such as MobileNet (Howard et al.] 2017 Sandler et al. 2018) and ShuffleNet (Howard et al., 2019), were manually designed. Recent work started to use neural architectures search (NAS) automatically learn efficient network designs (Zoph al., 2018 Cao et al. 2019: Tan et al. 22119 Tan & Le 2019 Chaudhuri et al. 2020)). They mostly fcous on improving the efficiency of single models by designing better architectures, while we explore committee-basee models without tuning the architecture.
Ensembles. Ensemble learning has been well studied in machine learning and there have been many seminal works, such as Bagging (Breiman 1996), Boosting (Schapire 1990), and AdaBoost (Freund & Schapire 1997). Ensembles of neural networks have been used for many tasks, such as image
2

---

Published as a conference paper at ICLR 2022
classification (Szegedy al. 2015 Huang et al. 201 7a), machine translation (Wn et al. 2020 active learning (Beluch et al. 2018). and out-of-distributee
robustness @Lakshminarayanan et al, 2017 Fort et al. |019 Wenzel al. 2020). But the efficiency of model ensembles has rarely been systematically investigated. Recent work indicated that ensembles can be more efficient than single models for image classification (Kondratyuk et al., 2020 Lobacheva al. 2020). Our work further substantiates this claim through the analysis of modern architectures on large-scale benchmarks.
Cascades. A large family of works have explored using cascades to speed up certain tasks. For example, the seminal work from Viola & Jones (200T) built a cascade of increasingly complex classifiers speed up face detection. Cascades have also been explored in the context of deep neural networks. BBolubbasi et al.] (2017) reduced the average test-time cost by learning policy to allow easy examples to early exit from network. A similar idea was also explored by Guan et al. 2018). Huang et al. (2018) proposed specially designed architecture Multi-Scale DenseNet to better incorporate early exits into neural networks. Given pool of models, [Streeter modees, (2018) presented an approximation algorithm to produce cascade that can preserve accuracy while reducing FLOPs and demonstrated improvement over state-of-the-art NAS-based models on ImageNet. Different from previous work that primarily focuses developing new methods build cascades, we show that even the most straightforward method can already provide a significant speedup without training an early exit policy (Bolukbasi et al. 2017 Guan al. 2018) designing specialized multi-scale architecture (Huang et al. 2018).
Dynamic Neural Networks. Dynamic neural networks allocate computational resources based on the input example, i.e., spending more computation on hard examples and less on easy ones (Han) al., 2021). For example, [Saazerr et al. (2017) trained gating network to determine what parts in high-capacity model should be used for each example. Recent work (Wu et al.] 2018: Veit & Belongie, 2018: Wang et al. 2018) explored learning a policy to dynamically select layers or blocks to execute in ResNet based on the input image. Our analysis shows that cascades of pre-trained models are actually a strong baseline for dynamic neural networks.
3 ENSEMBLES ARE ACCURATE, EFFICIENT, AND FAST TO TRAIN
Model ensembles are usefuul for improving accuracy, but the usage of multiple models also intro- duces extra computational cost. When the total computation is fixed, which one will give a higher accuracy: single models ensembles? The answer is important for real-world applications but this question has rarely been systematically studied on modern architectures and large-scale benchmarks.
We investigate this question on ImageNet (Russakovsky et al. 2015) with three architecture families: EfficientNet (Tan & Le 2019), ResNet (He et al.] (016). and MobileNetV2 (Sandler et al., 2018). Each architecture family contains series of networks with different levels of accuracy and com- putational cost. Within each family, we train a pool of models, compute the ensemble of different combinations of models, and compare these ensembles with the single models in the family.
We denote an ensemble of n image classification models by [MI,..., Mn J. where Mj is the zth model. Given an image x, a; Mx) is vector representing the logits for each class. To ensemble the n models, we compute the mean of logit: [1] a = 1 Σ, a; and predicts the class for image by applying argmax ans The total computation of the ensemble FLOP ens Σ, FL.)s(M.). where FL.OPs()) gives the FI.OPs of a model.
We show the t accuracy on ImageNet and FLOPS of single models and ensembles in Figure 2 Since there are many possible combinations of models to ensemble, we only show those Pareto optimal ensembles in the figure. We see that ensembles are more cost-effective than large single models, e.g., EffficientNet-B//B//B3 and ResNet-152/200 But in the small computation regime, single models outperform ensembles. For example, the ensemble of B5 matches B7 accuracy while using about 50% less FLOPs. However, ensembles more FLOPs than MobileNetV2 when they have a similar accuracy.
Wee note that the mean of probabilities is more general choice since logits can be arbitrarily scaled. In our experiments, we observe that they yield similar performance with the mean of logits heing marginally better. The findings in our work hold true matter which choice is used.
3

---

Published as a conference paper at ICLR 2022
05
Inneroted Casedaa
Srearb
Caccadee
Imagee 1en) GcOrRCC
70
-R00p
22
012
648
131
8364051 Oa.
180
180 1866 B91MCM
153
631
Avanga F.005 (001)nn)
Average TLOOS (Bil
Ararage FLOPS 211lon
(a) EfficientNet
(b) ResNet
(c) MobileNetV2
Figure 2: Ensembles work well in the large computation regime and cascades show benefits in all computation regimes. These cascades are directly converted from ensembles without optimizing the choice of models (see Sec. 4. Black dots represent single models. Ensembles: Ensembles are more cost-effective than large single models, e.g.,
and Res2et-152/200. Cascades: Converting ensembles cascades significantly reduces the FLOPs without hurting the full ensemble accuracy (each star is on the left of square).
A possible explanation of why model ensembles are more powerful at large computation than at small computation comes from the perspective of bias-variance tradeoff.. Large models usually have small bias but large variance, where the variance term dominates the test error. Therefore, ensembles are beneficial at large computation as they can reduce the variance in prediction (Breiman [996). For small models, the bias term dominates the test error. Ensembles can reduce the variance, but this cannot compensate the fact that the bias of small models is large. Therefore, ensembles are less powerful at small computation.
Our analysis indicates that instead of using a large model, one should use an ensemble of multiple relatively smaller models, which would give similar performance but with fewer FLOPs. In practice, model ensembles can be easily parallelized (e.g., using multiple accelerators), which may provide further speedup for inference. Moreover, often the total training cost of an ensemble is much lower than that of an equally accurate single model (see appendix for more details).
4 FROM ENSEMBLES TO CASCADES
In the above we have identified the scenarios where ensembles outperform or underperform single models. Specifically, ensembles are not an ideal choice when only small amount of computation is allowed. In this section, we show that by simply converting ensemble to cascade, one can significantly reduce the computation and outperform single models in all computation regimes.
Applying ensemble is wasteful for easy examples where a subset of models will give the correct answer. Cascades save computation via early exit po- tentially stopping and outputting an- swer before all models are used. The to- tal computation can be substantially duced if we accurately determine when exit from cascades. For this pur- pose, we need function measure how likely a prediction is correct. This func- tion is termed confidence (more details in Sec. 4.1 A formal procedure of cascades is provided in Algorithm [ Note that our cascades also average the predictions of the models having been used so far. So for examples where all models are used, the cascade effectively becomes an ensemble.
Algorithm Cascades
Input: Models (M,). Thresholds {4.). Test image r. for 1 1,2, do
cas - O₃ D M(xx)
FLOPs = Σ
=i (LOPs(M,)
Early exit if confidence score g(a 2 tk
end for Return ans and FLOP
4.1 CONFIDENCE FUNCTION
Let g()): N R be the confidence function, which maps maps predicted logits to confi- dence score. The higher g(a) is, the more likely the prediction a is correct. Previous work (Huang al. 2017b Streeterr 2018) tried several simple metrics indicate the prediction confidence, such
4

---

Published as a conference paper at ICLR 2022
54.5
194
54.0
95
B6
35
$3.0
12.5
85
Upper Bound
Imagelee
Max Prop
12.0
B3+85 Cascade
Legit
144485 Cascade
Prob
$1.5
B4+06
Estropy
1
150
Acerage FLOPS (11iioess)
Figure 3: Different metrics for the confidence Figure 4: Cascades with different confidence function. For EfficientNet-BOO model, we se-
thresholds. Each black dot is single model lect the top-k% validation images with highest and each square is an ensemble of models. Each confidence scores and compute the classification colored dot represents cascade with a specific accuracy within the selected images. The higher
(1 S t S 1). As t increases from 0 to I, the cascade uses more and more computation and dence metric is. All the metrics perform similarly changes from single model (first model in the in estimating how likely a prediction correct.
the accuracy at a certain k; the better the confi-
cascade; 1 = 0) to the ensemble (t1 = 1).
as the the maximum probability in the predicted distribution, the gap between the top-2 logits or probabilities, and the (negative) entropy of the distribution. As shown in Figure 3 all the metrics demonstrate reasonably good performance on measuring the confidence of prediction, i.e. estimat- ing how likely a prediction correct (see appendix for more details and results). In the following experiments, adopt the maximum probability metric, i.e., g(a) max(softmax(a))
For a cascade of n models (M,), we also need (n 1) thresholds {ty} on the confidence score, where
l; decide whether prediction is confident enough exit after applying model M; (see Algorithm| T. As we define g()) as the maximum probability, 4 is in [0, 1]. smaller i; indicates more images will be passed to the next model M++1. A cascade will reduce an ensemble if all the thresholds }t:} are set to 1. fa is unneeded, since the cascade will stop after applying the last model M,, no matter how confident the prediction is.
We can flexibly control the trade-off between the computation and accuracy of cascade through thresholds {t,) J. To understand how the thresholds influence cascade, we visualize several 2 2-model cascades in Figure 4. For each cascade, we sweep / from O) and and plot the results. Note that all the curves Figure have plateau, indicating that we can significantly reduce the average FL.OPs without hurting the accuracy if & is properly chosen. We select the thresholds [t,} on held-out validation images according to the target FLOPs or validation accuracy. In practice, we find such thresholds via grid search. Note that the thresholds are determined after all models are trained. We only need the logits of validation images to determine [tz], so computing the cascade performance for specific choice of thresholds is fast, which makes grid search computationally possible.
4.2 CONVERTING ENSEMBLES TO CASCADES
For each ensemble in Figure we convert a cascade that uses the same set of models. During conversion, we set the confidence thresholds such that the cascade performs similar to the ensemble while the FLOPs are minimized. By design in cascades some inputs incur more FLOPss than others. So we report the average FI.OPs computed over all images in the test set.
We see that cascades consistently use less computation than the original ensembles and outperform single models in all computation regimes and for all architecture families. Taking 2 EfficientNet-B2 as an example (see Figure 2a), the ensemble initially obtains similar accuracy to B3 but uses more FLOPs. After converting this ensemble to cascade, successfully reduce the average FLOPs to 1.3B (1.4x speedup over B3) and still achieve B3 accuracy. Cascades also outperform small MobileNetV2 models in Figure 2c
2 As side observation, when analyzing the confidence function, we notice that models in our experiments are often slightly underconfident. This contradicts the common belief that deep neural networks tend to be overconfident (Guo al. 2017). Please see appendix for more details.
5

---

Published as a conference paper at ICLR 2022
87.5
85.0
+0.5%
Imagedin Arturayy 080
83.5
84.5
Acuwary
Speedop
5.4s Sptedup
82.0
R7
84.0
60.5
47%
800.0
79.5
1.3g
ELPP
Siiar FLOPs
Sinilar Acouracy
Simllar Acturacy
0.5
1.0
1.5
2.0
10
20
30
40
50
Average ELOPS (Billions)
Average FLOPS (Bllions)
(a) EfficientNet (small computation)
(b) EfficientNet (large computation)
Accuaayy
+..0%
11.5%
Samilar Theoughuu
Accuracy
Inn manseeii (%)
ImageeMes Acturacy (%)
Semilar Accurary
2.14 Speedap
Imageete Arcuraey
79.5
1.492244
VIT---36
79.0
1.8
Roshe:152
78.5
Mobilhhet22 104924
Similar L'PS
Resfit
1Ax
NobileNatV2 Similar FLOPs
Accuracy
2,3x Specdup
DenheetC1
Similar
VIR1-204
17.5
300
Average FLOPS usilions)
FLOPS (Bilionni
Throughout
(c) ResNet
(d) MobileNetV2
(c) ViT
Figure 5: Cascades EfficientNet, ResNet, MobileNetV2 or ViT models on ImageNet. Compared with single models, cascades obtain a higher accuracy with similar cost (red squares) achieve a significant speedup while being equally accurate (green squares; e.g. 5.4x speedup for B7). The benefit of cascades generalizes to all four architecture families and all computation regimes. Numerical results are also available in Table 13 in appendix.
MODEL SELECTION FOR BUILDING CASCADES
The cascades in Figure do not optimize the choice of models and directly use the set of models in the original ensembles. For best performance, we show that one can design cascades to match specific target FL.OPs or accuracy by selecting models be used in the cascade.
Let M be the set of available models, e.g., models in the EfficientNet family. Given target FLOPs 3, we select n models M- {M, M) and confidence thresholds {t,} by solving the following problem:
max Accuracy (C (M,T)) (M;EM).(tz)
(1)
FLOPs (M,T)) < B,
where C (M, is the cascade of models {M+ with thresholds {t,). Accuracy(-) gives the val- idation accuracy of a cascade, and FLOPs(-) gives the average FLOPs. Similarly, we can also build cascade match target validation accuracy 7 by minimizing FLOPPs (C (M, ()) with the constraint Accuracy (C (M, T") 2
Note that this optimization done after all models M were independently trained. The difficulty of this optimization depends on the size of M and the number of models in the cascade n. The problem will be challenging if [M| or n is large. In our case, |M and are not prohibitive, e.g., [M] 8 and n < 4 for EfficientNet family. We are therefore able to solve the optimization problem with exhaustive search. See appendix for more details.
5.1 TARGETING FOR SPECIFIC FLOPS OR ACCURACY
For each single EfficientNet, ResNet or MobileNetV2, we search for cascade match its FLOPs (red squares in Figure 5a 5d) or its accuracy (green squares in Figure 5d). Notably, in addition to convolutional networks, we also consider Transformer architecture ViT (Dosovitskiy et al. 2021]. We build cascade ViT-Base and ViT-Large to match the cost or accuracy of VirgLerge (Figure Seb. For VIT, we measure the speedup in throughput (more details on throughput below).

---

Published as a conference paper at ICLR 2022
Table 1: Online latency measured on TPUv3. Compared with Efficient-B6 or B7, our cas- cade achieves a 3.8x or 5.5x reduction in la- tency respectively.
Table 2: Offline throughput (images processed
per second) measured on TPUv3. Compared
with Efficient-B6 or B7, our cascade achieves a
3.0x or 3.5x increase in throughput respectively
Top-1 (%)
Latency (ms)
Speedup
Top-I (%)
Throughput (/s)
Speedup
B6
83.7
57.1
B6
83.7
138
Cascade"
83.7
15.1
3.8x
Cascade
83.7
415
3.0x
B7
84.1
126.6
B7
84.1
81
Cascade"
84.2
23.2
5.5x
Cascade
84.2
280
3.5x
The cuscade matches B6 or B7 accuracy in Fig. 5b
The cuscade matches B6 or B7 accuracy Fig. Sbb
When building cascades, we consider all networks in the same family as the set of available models. The same model type is allowed to be used for multiple times in cascade but they will be different models trained separately. For ImageNet experiments, the search is conducted on a small set of held-out training images and cascades are evaluated the original validation set. We provide more experimental details in appendix.
Results in Figure further substantiate our finding that cascades more efficient than single mod- els in all computation regimes. For small models, we can outperform MobileNetV2-11 11.22-11222 224 by 1.4% using equivalent FLOPs. For large models, 5.4x over EfficientNet-B7 while matching their accuracy.
obtain 2.3x speedup ViT-L-384 and
To better understand how cascade works, compute the percentage of images that exit from the cascade at each stage. The cascade above that matches B7 accuracy contains four models: [B3, B5, B5, B5]. In this cascade, 67.3% images only consume the cost of B3 and only 5.5% images use all four models. This saves large amount of computation compared with using B7 for all the images.
On-device Latency and Throughput. In the above, we mostly use average FLOPs to measure the computational cost. We now report the latency and throughput of cascades on TPUv3 in Table I to confirm that the reduction in FLOPs can translate the real speedup on hardware.
Cascades are useful for online processing with a fixed batch size 1. Using batch size 1 is sub-optima for hardware, but still happens in real-world applications, e.g., mobile phone cameras processing single image (Wadhwa et al. 2018) or servers that need to rapidly return the result without waiting for enough queries to form a batch. Table [hoows the average latency of cascades on TPUv3 with batch size 1. Cascades are up to 5.5x faster than single models with comparable accuracy.
Cascades are also useful for offline data processing, where work can be batched to fully utilize the hardware. We can apply the first model in the cascade to all examples, and then select only subset of examples to apply the second model and so forth. Table reepors the throughput (images processed per second) of cascades on TPUv3 via batch processing. Cascades have significantly higher throughput than comparable models. We provide more results in appendix.
Comparison with NAS. We also compare with state-of-the -hee---tthhhrrr art NAS methods, e.g., Big- NAS (Yu al., 2020), OFA (Cai al.) 2020) and Cream (Peng et al. 2020), which can find architectures better than EfficientNet. But as shown in Table 3 simple cascade of Effi- cientNet without tuning the architecture already outperforms these sophisticated NAS methods. The strong performance and simplicity of cas- cades should motivate future research to in- clude them as a strong baseline when proposing novel architectures.
Table 3: Comparison with SOTA NAS methods.
Cascades outperform novel architectures found by costly NAS methods.
Top-1 1 (%)
FLOPS (B)
BigNASModel-L YYu al. 2020
79.5
0.59
OFA Cai
80.0
0.60
Cream-1. (Peng et
80.0
0.60
Cascude
80.1
0.67
BigNASModel-Xt YYu et 2020 Cascade"
80.9
1.0
81.2
1.0
The cascade that matches Bt FLOPPs Figure
5.2 GUARANTEE ON WORST-CASE FLOPS
Up until now we have been measuring the computation of cascade using the average FLOPs across all images. But for some images, it is possible that all the models in the cascade need to be applied. In this case, the average FLOPs cannot fully indicate the computational cost of a cascade. For example, the cascade that matches B5 or B6 accuracy in Figure 5b has higher worst-case FLOPPs
7

---

Published as a conference paper at ICLR 2022
Table 4: Cascades can be built with a guarantee on worst-case FLOPs. We use 'with' 'w/o' to indicate whether a cascade can provide such a guarantee not. Cascades with such a guarantee are assured use fewer FLOPs than single models in the worst-case scenario, and also achieve considerable speedup in average-case FLOPs.
Average case FLOPs (B)
Average-case Speedup
Average-case FLOPs (B)
Warsi-case FLOPs (B)
Averuge-cane Speedup
Top-1 (%)
FLOPs (B)
Top-1 (%)
B6
wro"
83.4
3.4
14.2
3.0k
w/o
83.7
4.1
25,9
4.7x
with
2.9x
with
83.7
4.2
15.0
Cascades from Figure Sb
Table 5: Self-cascades. In the column of self-cascades, the numbers represent the two resolu- tions 11 and 12 used in the cascade. Self-cascades use fewer FLOPs than comparable single models.
EfficientNet
Top-1 (%)
FLOPs (B)
Self-cascades
Top-1 (%)
FL.OPs (B)
Speedup
B2
80.0
1.0
B1-240-300
80.1
0.85
1.2x
B3
81.3
1.8
B2-260-380
81.3
1.6
1.2x
82.5
4.4
R3-300-456
82.5
2.7
1.7x
83.3
10.3
B4-380-600
83.4
6.0
1.7x
B6
83.7
19.1
B5-456-600
83.8
12.0
1.6x
B7
84.1
37
B6-528-600
84.1
22.8
1.6x
than the comparable single models (see 'wlo' in Table 4). Therefore, we now consider worst-case FLOPs of cascade, the sum of FL.OPs of all models in the cascade.
We can easily find cascades with a guarantee on worst-case FLOPs by adding one more constraint: E, FLOPP((M)) S @weest where gworst is the upper bound on the worst-case FLOPs of the cascade. With the new condition, we re-select models in the cascades to match of accuracy of B5 or B6. As shown in Table compared with single models, the new cascades achieve significant speedup in average-case FLOPs and also ensure its worst-case FLOPs are smaller. The new cascades with the guarantee on worst-case FL.OPs are useful for applications with strict requirement on response time.
6 SELF-CASCADES
Cascades typically contain multiple models. This requires training multiple models and combining them after training. What about when only model is available? We demonstrate that one can convert single model into a cascade by passing the same input image at different resolutions to the model. Here, we leverage the fact that resizing image to higher resolution than the model is trained on often yields higher accuracy (Touvron et al. 2019) at the cost of more computation. We call such cascades as "self-cascades" since these cascade only contain the model itself.
Given a model M, we build a 2-model cascade, where the first model applying M at resolution ri and the second model is applying M at higher resolution r2(r2 r1). We build self-cascades using EfficientNet models. Since each EfficientNet is defined with specific resolution (e.g., 240 for B1), set F1 its original resolution and set 72 higher resolution. We set the confidence threshold such that the self-cascade matches the accuracy of single model.
Table shows that self-cascades easily outperform single models, i.e., obtaining similar accuracy with fewer FLOPs. Table 5 also suggests that if we want to obtain B7 accuracy, we can train a B6 model and then build self-cascade, which not only uses much fewer FLOPPs during inference, but also takes much shorter time to train.
Self-cascades provide way to convert one single model a cascade which will be more efficient than the original single model. The conversion is almost free and does not require training any additional models. They are useful when one does not have resources to train additional models or the training data is unavailable (e.g., the model is downloaded).
7 APPLICABILITY BEYOND IMAGE CLASSIFICATION
We now demonstrate that the benefit of cascades generalizes beyond image classification.
7.1 VIDEO CLASSIFICATION
Similar to image classification, a video classification model outputs a vector of logits over possible classes. We the same procedure above build cascades of video classification models.
8

---

Published as a conference paper at ICLR 2022
Table 6: Cascades of X3D models on Kinetics-600, We outperform X3D-XL by 1.2%.
Single Models
Cascades Similar FLOPs
Cascades Similar Accuracy
Top-1 (%)
FLOPs (B)
Top-1 (%)
FLOPs (B)
ATop-1
Top-1 (%)
FLOPs (B)
Speedup
X3D-M
78.8
6.2 x 30
80.3
5.7 x 30
1.5
79.1
3.8 x 30
1.6x
X3D-L
80.6
24.8 x 30
82.7
24.6 x 30
2.1
80.8
7,9 x 30
3.2x
X3D-XL
81.9
48.4 x 30
83.1
38.1 x 30
1.2
81.9
13.0 x 30
3.7x
architecture family for video classification, which is the state-of-the-art in terms of both the accuracy and efficiency. The X3D family contains series of models of different sizes. Specifically, we build cascades of X3D models match the FLOPs or accuracy of X3D-M, X3D-L or X3D-XL on Kinetics-600) (Carreira et al., 2018).
We consider the X3D (Feichtenhofer 2020)
The results are summarized in Table 6 where cascades significantly outperform the original X3D models. Following X3D Feichtenhofer (2020), 'x30' in Table 6 means we sample 30 clips from each input video during evaluation (see appendix for more details). Our cascade outperforms X3D- XI. fatt-oo-theeare video classification model, by 1.2% while using fewer average FLOPs. Our cascade can also match the accuracy of X3D-XI. with 3.7x fewer average FI.OPs.
7.2 SEMANTIC SEGMENTATION
Table 7: Cascades of DeepLabv3 models on Cityscapes.
In semantic segmentation, models predict a vector of logits for each pixel in the image. This differs from image classification, where the model makes single prediction for the en- tire image. We therefore revisit the confidence function definition to han- dle such dense prediction tasks.
mloU
FLOPs (B)
Speedup
ResNet-50 ResNet-101
77.1
348
78.1
507
Cascade full
78.4
568
0.9x
Cascade s 512 78.1
439
1.2x
Cascade s 128
78.2
398
1.3x
Similar to before, we use the maximum probability to measure the confidence of the prediction for a single pixel Le., g(ap) = max (nxxsofttmxx(x))) where Opp is the predicted logits for pixel p. Next, we need a function (.)m to rate the confidence of the dense prediction for an image, so that we decide whether apply the next model to this image based this confidence score. For this purpose, we define (.) as the average confidence score of all the pixels in the image: dense (R) [T7] H(p))( where R represents the input image.
In a cascade of segmentation models, we decide whether to pass an image R to the next model based on g (.ense (-). Since the difficulty to label different parts in one image varies significantly, e.g., roads are easier to segment than traffic lights, making a single decision for the entire image can be inaccurate and leads to a waste of computation. Therefore, in practice, we divide an image into grids and decide whether to pass each grid to the next model separately.
We conduct experiments on Cityscapes (Cordts et al. 2016) and use mean IoU (mloU) as the metric. We build a cascade of DeepLaby3-ResNet-- and DeepLaby3-RB -eee-aaay-ReeNee---lee 101 (Chen et al., 2017) and report the reults in Table 7 is the size of the grid. The full image resolution is 1024 x 2048 so s = 512 means the image is divided into grids. If we operate on the full image level ('full'), the cascade will use more FLOPs than ResNet-101. But if operating on the grid level, the cascade can successfully reduce the computation without hurting the performance. For example, the smaller grid size ('s 128') yields 1.3x reduction in FLOPS while matching the mloU of ResNet-101.
8 CONCLUSION
We show that committee- based models, i.e., model ensembles or cascades, provide a simple comple- mentary paradigm to obtain efficient models without tuning the architecture. Notably, cascades can match or exceed the accuracy of state-of-the-an models on variety of tasks while being drastically more efficient. Moreover, the speedup of model cascades is evident in both FLOPs and on-device la- tency and throughput. The fact that these simple committee-based models outperform sophisticated NAS methods, as well as manually designed architectures, should motivate future research to include them as strong baselines whenever presenting models outline a simple procedure to improve accuracy while maintaining efficiency that only needs off shel models.
architecture. For practitioners, committee-bar based

---

Published as a conference paper at ICLR 2022
AUTHOR CONTRIBUTIONS
Xiaofang wrote most of the code and paper, and most of the experiments. Elad and Yair advised on formulating the research question and plan. Dan generated the predictions of X3D on Kinetics- 600. Eric conducted the experiments about the calibration of models. Kris helped in writing the paper and provided general guidance.
ACKNOWLEDGMENTS
The authors would like to thank Alex Alemi, Sergey loffe. Shankar Krishnan, Max Moroz, and Matthew Streeter for their valuable help and feedback during the development this work. Elad and Yair would like to thank Solomonico 3rd for inspiration.
REFERENCES
William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The power of ensembles for active learning in image classification. In CVPR, 2018.
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In ICML, 2017.
Leo Breiman. Bagging predictors. Machine learning, 24(2):123-140) 1996.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In ICLR, 2020.
Shengcao Cao, Xiaofang Wang, and Kris M. Kitani. Learnable embedding space for efficient neural architecture compression. In ICLR, 2019.
Joao Carreira, Eric Noland, Andras Banki- Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600, 1808. 01340, 2018.
Shraman Ray Chaudhuri, Elad Eban, Hanhan Li, Max Moroz, and Yair Movshovitz-Atiaa, Fine- grained stochastic architecture search. (rXXi:2000600988. 2020.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. 1:Xiv:/706.05588 2017.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x 16 words: Transformers for image recognition at scale. 2021.
Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In CVPR, 2020.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape per- spective.
2019.
Yoav Freund and Robert E Schapire. A decision-theorees generalization on-line learning and an application to boosting. Journal of computer and system sciences, 55(1): 1997.
Jiaqi Guan, Yang Liu, Qiang Liu, and Jian Peng. Energy-efficient amortized inference with cascaded deep classifiers. IJCAI, 2018.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML, 2017.
Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey, arXiv:2 102, 04906, 2021.
10

---

Published as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In ICCV, 2019.
Andrew G Howard, Mengiong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. N.K.vv/700404866/ 2017.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hopcroft, and Kilian Q Weinberger. Snapshot ensembles: Train get for free. In ICLR, 2017a.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017b.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Weinberger. Multi-scale dense networks for resource efficient image classification. In ICLR, 2018.
Dan Kondratyuk, Mingxing Tan, Matthew Brown, and Boqing Gong. When ensembling smaller models is more efficient than single large models. arXiv:2005.00570. 2020.
Balaji Lakshminarayanan. Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NewrIPS, 2017.
Ekaterina Lobacheva, Nadezhda Chirkova, Maxim Kodryan, and Dmitry P Vetrov. On power laws in deep ensembles. NeurIPS, 2020.
Houwen Peng, Hao Du, Hongyuan Yu, Qi Li, Jing Liao, and Jianlong Fu. Cream of the crop: Distilling prioritized paths for one-shot neural architecture search. In NearlPS, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Robert E Schapire. The strength of weak learnability. Machine learning, 5(2):197-227, 1990.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In ICLR, 2017.
Matthew Streeter. Approximation algorithms for cascading prediction models. In ICML, 2018.
Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net-
works. In ICML, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang. Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platfor--aware neural architecture search for mobile. In CVPR, 2019.
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. In NeurlPS, 2019.
Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In ECCV, 2018.
11

---

Published as a conference paper at ICLR 2022
Paul Viola and Michael Jones. Rapid object detection using boosted cascade of simple features. In CVPR, 2001.
Neal Wadhwa, Rahul Garg, David E Jacobs, Bryan E Feldman, Nori Kanazawa, Robert Carroll,
Yair Movshovitz-Atias, Jonathan Barron, Yael Pritch, and Mare Levoy. Synthetic depth-of- field with a single-camera mobile phone. ACM Transactions on Graphics (TOG), 37(4):1-13, 2018.
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph Gonzalez. Skipnet: Learning dy- namic routing in convolutional networks. In ECCV, 2018.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: alternative approach efficient ensemble and lifelong learning. In ICLR, 2020. URL/https://ooenneeview wwt.:/.eeeffrrree
id=sklfllrYDrr
Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. In NeurIPS, 2020.
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. CVPR, 2018.
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural archi- tecture search with big single-stage models. ECCV, 2020.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018
12

---

Published as a conference paper at ICLR 2022
A ON-DEVICE LATENCY AND THROUGHPUT
We report the on-device latency and throughput of cascades to confirm that the reduction in FLOPs translate the real speedup hardware. The latency throughput of model is highly dependent on the batch size. So we consider two scenarios: (1) online processing, where we use a fixed batch size and (2) offline processing, where we can batch the examples.
Online Processing. Cascades are useful for online processing with a fixed batch size 1. Using hatch size 1 is sub-optimal for the utilization of accelerators like GPU TPU, but it still happens in some real-world applications, e.g., mobile phone cameras processing a single image (Wadhwa et al. 2018) or servers that need to rapidly return the result without waiting for enough queries to form a batch. We report the average latency of cascades on TPUv3 with batch size 1 in Table g Cascades achieve a similar accuracy to single models but use a much smaller average latency to return the prediction.
Offline Processing, Cascades are also useful for offline processing of large-scale data. For example, when processing all frames large video dataset, we can first apply the first model in the cascade to all frames, and then select subset of frames based on the prediction confidence to apply following models in the cascade. In this way all the processing can batched to fully utilize the accelerators. We report the throughput of cascades on TPUv3 in Table 2 which is measured the number of images processed per second. We use batch size 16 when running models on TPUv3 for the case of offline processing. As shown in Table cascades achieve much higher throughput than single models while being equally accurate. For clarification, only the throughput of ViT in Table 14 is measured on RTX 3090 while the throughput for other models is measured on TPUv3.
B DETAILS OF IMAGENET MODELS
When analyzing the efficiency of ensembles or cascades ImageNet, we consider four architecture
families: EfficientNet (Tan & Lc 2019). ResNet (He et al. 2016), MobileNetV2 (Sandler et al.) 2018), and ViT (Dosovitskiy et al. 2021). All the single models are independently trained with their original training procedure. We do not change the training schedule or any other hyper-parament parameters.
The EfficientNet family contains 8 architectures (EfficientNet-BO) to B7). We train each architec- ture separately for 4 times with the official open-source implementation provided by the authors. So, in total there are 32 EfficientNet models.
For ResNet, we consider architectures (ResNet-5001101/15222 and train each architecture for 2 times using open-source TPU implementationf There are 8 ResNet models in total.
For MobileNetV2, we directly download the pre-trained checkpoints from its official open- source implementation We use 5 MobileNetV2 models: MobileNetV2-00756 @ 160, 1.0@ 160, 1.0@ 192, 1.0@ 224, and 1.4@ 224. Each model is represented the form of w @r@r, where wo is the width multiplier and is the image resolution.
For VIT, we directly use the pre-trained checkpoints provided by the Hugging Face Team 6 We use 4 ViT models: ViT-B-224, ViT-L-224, ViT-B-384, and ViT-L-384.
Training each EfficientNet architecture for 4 times (in total 32 models) may sound computationally expensive. We note that it is unnecessary train each architecture for 4 times to find a well- performing ensemble or cascade. We train large pool EfficientNet models mainly for the pur- pose of analysis so that EfficientNet-B5. We analyze the influence of the size and diversity of the model pool in Sec. E.4
try diverse range of model combinations, e.g., the cascade of
efficientnet
https://qi http:://hhhhhaaaaeee/ereececeeee
lenilenet
Foor example, VIT-B-224:
http:s//wgginnnaae....///ooooo// 6-224 6-224
13

---

Published as a conference paper at ICLR 2022
Table 8: Average latency on TPUv3 for the case of online processing with batch size 1. Cascades are much faster than single models in terms of the average latency while being similarly accurate.
Top-1 (%)
Latency (ms)
Speedup
B1
79.1
3.7
Cascade"
79.3
3.0
1.2x
B2
80.0
5.2
Cascade"
80.1
3.7
1.4x
B3
81.3
9.7
Cascade
81.4
5.9
1.7x
B4
82.5
16.6
Cascade
82.6
9.6
1.7x
B5
83.3
27.2
Cascade
83.4
14.3
1.9x
B6
83.7
57.1
Cascade
83.7
15.1
3.8x
B7
84.1
126.6
Cascade
84.2
23.2
5.5x
"Tee cascade matches accuracy of EfficientNee- B1 to B7 in Figure the right column Table
Table 9: Throughput TPUv3 for the case of offline processing. Throughput is measured as the number of images processed per second. Cascades achieve a much larger throughput than single models while being equally accurate.
Top-1 (%)
Throughput (/s)
Speedup
B1
79.1
1436
Cascade"
79.3
1798
1.3x
B2
80.0
1156
Cascade
80.1
1509
1.3x
B3
81.3
767
Cascade
81.4
1111
1.4x
B4
82.5
408
Cascade"
82.6
656
1.6x
B5
83.3
220
Cascade
83.4
453
2.1x
B6
83.7
138
Cascade *
83.7
415
3.0x
B7
84.1
81
Cascade
84.2
280
3.5x
The cascade that matches the accuracy of EfficientNet-B) B7 in Figure|5or or the right column of Table 13
C ENSEMBLES ARE ACCURATE, EFFICIENT, AND FAST TO TRAIN
C.1 EXPERIMENTAL DETAILS
We analyze the efficiency of ensembles of EfficientNet, ResNet, MobileNetV2 on ImageNet. For EfficientNet. we consider ensembles of two to four models of either the same or different ar- chitectures. Note that only try different combinations of architectures used in the ensemble,
14

---

Published as a conference paper at ICLR 2022
Table 10: Training time (TPUv3 days) of EfficientNet.
B0
B1
B2
B3
B4
B5
B6
I37
12
24
32
48
128
160
Table 11: Training time (TPU v3 days) of ensembles. We the '+' notation indicate the models in enmsebles. Ensembles are faster than single models in both training and inference while achieve similar accuracy.
Top-1 (%)
FLOPs (B)
Training
B6
83.7
19.1
128
B3+B4+B4
83.6
10.6
88
B7
84.1
37
160
B5+B5
84.1
20.5
96
B5+B5+B5
84.3
30.8
144
but not the combinations of models. For example, when an ensemble contains EfficientNNr-B5, while we have multiple B5 models available, we just randomly pick one but do not try all possible choices. The FLOPs range of ResNet MobileNetV2 models is relatively narrow compared with EfficientNet, so we only consider ensembles of two models for ResNet and MobileNetV2.
C.2 TRAINING TIME OF ENSEMBLES
In Sec. show that ensembles match the accuracy of large single models with fewer inference FLOPs. We now show that the total training cost of an ensemble if often lower than an equally accurate single model.
We show the training time of single EfficinetNer models in Table 10 train BO to B5, and 128 TPUv3 cores train B6 B7. All the models trained with the public official implementation of EfficientNet. We choose the ensemble that matches the accuracy of B6 or B7 and compute the total training time the ensemble based on Table [O] As shown in Table II] the ensemble of 2 B5 can match the accuracy of B7 while being faster in both training and inference.
We use 32 TPUv3 cores to
D FROM ENSEMBLES TO CASCADES
D.1 CONFIDENCE FUNCTION
The higher the confidence score g(a)) is, the likely the prediction given by is correct. In Sec 4.1, we compare different choices for the confidence function in Figure 3 For specific con- fidence function, we select the top-k% images with highest confidence scores. Then compute the classification accuracy within the selected images, If a higher confidence score indicates that the prediction more likely to be correct, the accuracy should drop as as increases.
Figure B is generated with EfficientNet-BB BO model trained on ImageNet, where we sweep l from 0 100 and compute the accuracy within the selected top-&% images from the ImageNet vali- dation set. When k 100, all the images are selected so the accuracy is exactly the accuracy of EfficientNet-B0 1%). The 'Upper Bound' curve represents the best possible performance for the metric. It has 100% accuracy when & < 77.1, i.e., all the selected images correctly classified. The accuracy starts to drop when & becomes larger, since some misclassified images are inevitably chosen. We observe all the metrics demonstrate reasonably good performance in estimating how likely a prediction is correct, where the entropy performs slightly worse than other metrics.
We also compare the performance of the cascade of ViT-B-224 and ViT-L-224 ImageNet with different confidence functions in Table For each confidence function, we set the threshold such that the cascade has similar throughput when using different confidence functions (~4099 images
15

---

Published as a conference paper at ICLR 2022
Original Callbraton
Calibration
80
Sccuray (90
40
Acurszy 10
60
20
20
20
40
60
80
100
20
100
Confidence
Confidence
(a) EfficientNet-B3.
(b) ResNett-0
Original
Original
Calibration
Callbraden
80
80
ecuracy (8)
40
40
20
Confidence 1961
Confidence (8)
(c) MobileNet-1.0@ 192
(d) X3D-M
Figure 6: Confidence accuracy for EfficientNet-B3, ResNet-50, MobileNet- 1,0t 192, and X3D- M on their respective dataset. 'Original' refers to the original prediction and 'Calibration' refers to the prediction after calibration. We divide the confidence into several intervals. Then for each interval, we visualize the accuracy of images whose confidence is within this interval. The original prediction is slightly underconfident, i.e., the confidence p is slightly lower than the actual accuracy of images whose prediction confidence is After calibration, the confidence almost equals to the actual accuracy.
Table 12: Performance of the cascade of ViT-B-224 and ViT-L-224 on ImageNet with different confidence functions. For each confidence function, we the threshold such that the cascade has similar throughput when using different confidence functions (~~409 images per second). The table shows that different confidence functions give a similar accuracy.
Top-1 (%)
Max Prob
82.3
Logit Gap
82.2
Prob Gap
82.3
Entropy Gap
82.1
per second). We observe that the cascade achieves similar accuracy with different confidence functions.
D.2 CALIBRATION OF MODELS
As a side observation, when analyzing the confidence function, we notice that models our ex- periments often slightly undereonfident, i.e., the confidence p is slightly lower than the actual accuracy of images whose prediction confidence is (see the 'Original' curve Figure (). This observation contradicts the common belief that deep neural networks tend be overconfident (Guo)
16

---

Published as a conference paper at ICLR 2022
et al. 2017 7). We conjecture this is due that models are trained with label smoothing (Szegedy) et al. 2016). Here, the confidence of a prediction defined as the probability associated with the predicted class (Guo et al., 2017). which is equivalent to the maximum probability in the predicted distribution.
A model is considered calibrated if the confidence of its prediction correctly estimates the true correctness likelihood (neither overconfident nor underconfident). The calibration of models can influence the ensemble performance when we ensemble models via simple averaging. If models in an ensemble are poorly calibrated, overconfident models may dominate the prediction, making those underconfident models useless in the ensemble.
For models in our experiments, we also have tried calibrating them before computing the ensem- ble performance. To calibrate a model, we use Platt scaling via learned monotonic calibration network. As shown in the 'Calibration' curve in Figure O the calibration improves the connection between the prediction confidence and accuracy, i.e., the confidence after calibration almost equals to the actual accuracy. But we notice that calibrating models only has small influence the final ensemble performance in our experiments, which might be because these models are just slightly underconfident before calibration. Therefore, we do not calibrate any model when computing the ensemble in all our experiments.
D.3 DETERMINE THE THRESHOLDS
Given the n models [M;} a cascade, we also need (n 11) thresholds {4:} on the confidence score. We can flexibly control the trade-off between the computation and accuracy of cascade through thresholds {tz].
We determine the thresholds } based on the target FLOPs or accuracy on validation images. In practice, we find such thresholds via grid search, i.e., enumerating all possible combinations for {tt]. Note that the thresholds are determined after all models of validation images to determine } so computing the cascade performance for a specific choice of thresholds is fast, which makes grid search computationally possible. As 4 is a real number, make sure two trials of 4 are sufficiently different by only considering the percentiles of confidence scores as possible values. When n > 2, there might be multiple choices of {44 that can give the target FLOPs or accuracy. In that case, we choose } that gives the higher accuracy or fewer FLOPs. Many choices for { } can be easily ruled as the FLOPs or accuracy of cascade changes monotonically with respect to any threshold t,
trained. We only need the logits
In practice, we often want the accuracy of a cascade match the accuracy of single model. To do that, we determine the thresholds such that the cascade matches the accuracy of the single model on validation images. Such thresholds usually enable the cascade to have similar test accuracy to the single model.
For ImageNet, we randomly select 25k training images and exclude them from training. We use these held-out training images to determine the confidence thresholds. The final accuracy is computed on the original ImageNet validation set.
E MODEL. SELECTION FOR BUILDING CASCADES
E. TARGETING FOR A SPECIFIC FLOPS OR ACCURACY
We can build cascades to match a specific FLOPs or accuracy by optimizing the choice of models and confidence thresholds, e.g., solving Eq. | when targeting for specific FLOOs. Note that this optimization done after all models in M are trained. The optimization complexity exponential in JM| and n, and the problem will be challenging if [M| and n are large. In experiments, [M and n are not prohibitive. Therefore, we solve the optimization problem with exhaustive search. One can also use more efficient procedures such the algorithm described in (Streeter) 2018).
Same as the our analysis of ensembles, we do not search over different models of the same architec- ture. but only search combination of architectures. Therefore, for EfficientNet, |M| 8 and and we have total 4672 = (8+ + 83 + 82) possible combinations models. Note that the search cheap to do it is conducted after all the models independently trained. No GPU training is
17

---

Published as a conference paper at ICLR 2022
Cascades of EfficientNet, ResNet or MobileNett22 models on ImageNet. This table con- tains the numerical results for Figure 5d Middle: Cascades obtain higher accuracy than single models when using similar FLOPs. Right: Cascades achieve similar aceuracy to single models with significantly fewer FLOPs (e.g. 5,4x fewer for B7). The benefit of cascades generalizes to all three convolutional architecture families and computation regimes.
Table 13:
Single Models
Cascades Similar FLOPs
Cascades Similar Accuracy
Top-1 (%)
FLOPs (B)
Top-1 (%)
FLOPs (B)
TTop-
Top-1 (%)
FLOPs (B)
Speedup
EfficlentNet
B1
79.1
0.69
80.1
0.67
1.0
79.3
0.54
1.3x
B2
80.0
1.0
81.2
1.0
1.2
80.1
0.67
1.5x
81.3
82.4
1.8
81.4
1.7x
1.2
82.6
2.2x
83.3
10.3
84.4
10.2
1.1
83.4
3.0x
B6
83.7
19.1
84.6
17.5
0.9
83.7
4.1
4.7%
B7
84.1
37
84.8
39.0
0.7
84.2
6,9
5,4x
ResNet
R101
77.9
7.2
79.3
7.3
1.4
78.2
4.9
1.5x
R152
78.8
10.9
80.1
10.8
1.3
78.9
6.2
1.8x
R200
79.0
14.4
80.4
14.2
1.3
79.2
6.8
2.1x
MobileNetV2
1.0@160
68.8
0.154
69.5
0.153
0.6
69.1
0.146
1.1x
1.0@192
70.7
71.8
0.22
70.8
1.2x
1.0@2224
0.30
73.2
0.30
1.4
71.8
0.22
1.4x
1,4@2224
75.0
0.58
76.1
0.56
1.1
75.1
0.43
1.4x
Table 14: Cascades of VIT models on ImageNet. This table contains the numerical results for Figure 5el 224 or 384 indicates the image resolution the model is trained on. Throughput measured on NVIDIA RTX 3090. Our cascades can achieve 1.0% higher accuracy than ViT-L-384 with similar throughput or achieve 2.3x speedup over while matching its accuracy. The benefit of cascades generalizes to Transformer architectures.
Single Models
Cascades Similar Throughput
Cascades Similar Accuracy
Top-1 (%)
Throughput (1s)
Top-1 (%)
Throughput ('s)
TTop-1
Top-1 (%)
Throughput ('s)
Speedup
VIT-L-224
82.0
192
83.1
221
1.1
82.3
409
2.1x
VIT-L-384
85.0
1.0
2.3x
involved in the search. We pre-compute the predictions of each model held-out validation set before search. During the search, we try possible models combinations by loading their predictions. We can usually find optimal model combinations within a few CPU hours.
In practice, we first train each EfficientNet model separately for 4 times and pre-compute their predicted logits. Then for each possible combination of models, load the logits of models and determine the thresholds according to the target FL.OPs or accuracy. Finally, we choose the best cascade among all possible combinations. Similar above, we choose models and thresholds on held-out training images for ImageNet experiments. No images from the ImageNet validation set are used when we select models for a cascade.
For ResNet and MobileNetV2, only tried 2-model cascades due to their relatively narrow FLOPs range. Therefore, the number of possible model combinations is very small (< 20). For ViT, we only tried cascades: VIT-B-22 + ViT-L-224 and ViT-B-384 + ViT-L-384.
F.2 CASCADES CAN BE SCALED UP
One appealing property of single models is that they can be easily scaled up down based on the available computational resources one has. We show that such property is also applicable to cascades, i.c., the model selection procedure when designing cascades for different FLOPs, which is required for cascades in Table 13
scale up base cascade respect different FLOPs constraints. This avoids
18

---

Published as a conference paper at ICLR 2022
Table 15: A Family of Cascades CO to C7. CO C7 significantly outperform single EfficientNet models in all computation regimes, C1 and C2 also compare favorably with state-of-the-an NAS methods, such as BigNAS (Yu et al. 2020), OFA (Cai et al. 2020) and Cream (Peng et al. 2200). This shows that the cascades can also be scaled up or down to respect different FLOPs constraints
as single models do. This helpful for avoiding the model selection procedure when designing
cascades for different FLOPs.
Model
Top-1 (%)
FLOPs (B)
ATop-1
Model
Top-1 (%)
FLOPs (B)
ATop-1
C0
78.1
0.41
C3
82.2
1.8
EfficientNet-BO
77.1
0.39
1.0
EfficientNet-BB
81.3
1.8
0.9
C1
80.3
0.71
C4
83.7
4.2
EfficientNer-Il
79.1
0.69
1.2
EffiiientNet--l
82.5
1.2
BigNASModehL
0.59
C5
84.3
OFA
80.0
0.60
EfficientNet-B5
83.3
1.0
Cream-L
80.0
0.60
C6
84.6
C2
81.2
RfficientNet-BR
83.7
19.1
0.9
EfficientNet-B2
80.0
1.0
1.2
C7
84.8
32.6
BigNASModel-XL
80.9
1.0
0.3
EfficientNet-B7
84.1
37
0.7
Specifically, we build 3-model cascade to match the FLOPs of EfficientNet-BO. We call this cascade CO (see below for details building CO). Then, simply by scaling up the architectures in CO, we obtain a family of cascades CO to C7 that have increasing FLOPs and accuracy. The models in CO are from the EfficientNet family. The results of CO to C7 in Table show that simply scaling up CO gives regimes. This finding enhances the practical usefulness of cascades as one can select cascades from this family based available resources, without worrying about what models should be used in the cascade.
family of cascades that consistently outperform single models in all computation
Details of building CO. The networks in EfficientNet family obtained by scaling up the depth, width and resolution of BO. The scaling factors for depth, width and resolution are defined as a w 3 and r 7 respectively, where a 1.2, 3 1.1 and y 1.15, as suggested in Tan et al. Tan & Le 2019). One can control the network size by changing Φ. For example, ó 0 gives BO,
gives B2, and á gives B7.
We build 3-model cascade CO to match the FLOPs of EfficientNet-BO by solving Eq, on held-out training images from ImageNet. When building CO, we consider 13 networks from EfficientNet family. As we want CO to use similar FLOPs to BO. we make sure the 13 networks include both networks smaller than BO and networks larger than BO. Their á are set to -4.0, -3.0, -2.0, -1.0, 0.0, 0.25, 0.5, 0.75, 1.0, 1 1.25, 1.50, 1.75, 2.0, respectively.
The of the three models in CO are -2.0, 0.0 and 0.75. Then simply scaling up the architectures in CO, i.e., increasing the á of each model in CO, gives us family of cascades CO to C7 that have increasing FLOPs and accuracy. The thresholds CO to C7 are determined such that their FLOPs similar to BO to B7.
E.3
EXIT RATIOS
To better understand how cascade works, we compute the exit ratio of the cascade, i.e., the per- centage images that exit from the cascade at each stage. Specifically, we choose the cascades in Table 4 that match the accuracy of B1 to B7 and report their exit ratios in Table For all the cascades Table 16. most images only consume the cost of the first model in the cascade and only few images have to use all the models. This shows that cascades are able to allocate fewer resources to easy images and explains the speedup cascades over single models.
E.4 MODEL POOL ANALYSIS
E.4.1 NUMBER OF MODELS CASCADES
We study the influence of the number models in cascades on the performance. Concretely, we consider the EfficientNet family and follow the same experimental setup in Sec. E.1 We sweep
19

---

Published as a conference paper at ICLR 2022
Table 16: Exit ratios of cascades. We use the '+' notation indicate the models in cascades.
Exit Ratio (%) at Each Stage
Top-1 (%)
FLOPs (B)
Model
Model
Model 3
Model
B1
79.1
0.69
BO+B1
79.3
0.54
78.7
21.3
B2
80.0
1.0
BO+B1+B3
80.1
0.67
73.2
21.4
5,4
B3
81.3
1.8
BO+B3+B3
81.4
1.1
68.0
26.4
5.7
B4
82.5
4.4
B1+B3+B4
82.6
2.0
67.9
15.3
16.8
B5
83.3
10.3
B2+B4+B4+B4
83.4
3,4
67.6
21.2
0.0
11.2
B2+B4+B4"
83.3
3.6
57.7
26.0
16.3
B6
83.7
19.1
B2+B4+B5+B5
83.7
4.1
67.6
21.2
5.9
5.3
B3+B4+B4+B4"
83.7
4.2
67.3
16.2
10.9
5.6
B7
84.1
37
B3+B5+B5+B5
84.2
6.9
67.3
21.6
5.6
5.5
Cascades from Table
with guarantee worst-case FLOPs.
ImageNet Top-1 Accuracy (%)
84
B6
EfficientNet-B7
85
82
B4
|B3
80
B2
B1
78
2-model Cascades
3-model Cascades
80
4 4-model Cascades
10
15
20
25
30
35
40
Average FLOPS (Billions)
Figure Impact of the number of models cascades.
the target FLOPs from 1 to 40 and find cascades of 3 or 4 models. As shown in Figure 7 the performance of cascades keeps improving as the number of models increases. We see big gap between 2-model cascades and 3-model cascades, but increasing the number of models from demonstrates a diminishing return.
As mentioned above, for EfficientNet cascades, we tried total 4672 (8* + 8 8²) possible combinations of models. Since 3-model cascades can obtain very close performance to 4-model cascades, one could try much fewer combinations to obtain similar results.
E.5 SIZE OF THE MODEL POOL
As mentioned in Sec. we train each EfficientNet architecture for 4 times so that we can try diverse range of model combinations. We now empirically show that naively adding more models
20

---

Published as a conference paper at ICLR 2022
Table 17: Max, min, mean, and standard deviation of the performance of single B5 models, 28 possible 2-B5 ensembles, and 56 possible 2-B5 cascades.
max
min
mean
std
Single Model Accuracy (%)
83.40
83.29
83.34
0.04
2-B5 Ensembles Accuracy (%)
84.18
83.97
84.10
0.05
2-B5 Cascades
Accuracy (%)
84.17
83.96
84.09
0.05
FLOPs (B)
13.35
12.32
12.62
0.29
Table 18:
Cascades of models of same architectures vs. Cascades of models of different architec- tures. The '+' notation indicates the models used in cascades.
Top-1 (%)
FLOPPs (B)
Speedup
B4
82.5
4,4
B3+B3+B3
82.6
2.7
1.6x
B1+B3+B4
82.6
2.0
2.2x
B5
83.3
10.3
B4+B4
83.3
5.1
2
B2+B4+B4+B4
83.4
3.4
3.0x
B6
83.7
19.1
B4+B4+B4
83.8
6,0
3.2x
B2+B4+B5+B5
83.7
4.1
4.7x
B7
84.1
B5+B5
84.1
13.1
2.8x
B3+B5+B5+B5
84.2
6.9
5.4x
of the same architecture to the pool only has small influence on the performance of ensembles or cascades.
We train EfficientNet-Bi models separately and build 2-B5 ensembles or cascades using any two of these models. The FL.OPs of these 2-B5 ensembles are the same (20.5B). For each cascade, we tune the confidence threshold such that the cascade achieves similar accuracy to the full ensemble. We show the max, min, mean, and standard deviation the performance of these different ensembles cascades in Table 17 and observe that the performance variation is small. Therefore, we conclude that adding more models the same architecture only has modest influence on the performance.
E.5.1 DIVERSITY OF THE MODEL POOL
We study the influence the diversity of architectures in the model pool on the performance. We compare cascades of models of same architectures and cascades of models of different architectures Tables As shown T8 while cascades of same-architecture models can already signif- icantly reduce the FLOPs compared with similarly accurate single model, adding more variations in the architecture can significantly improve the performance of cascades.
F APPLICABILITY BEYOND IMAGE CLASSIFICATION
F.1 VIDEO CLASSIFICATION
We conduct video classification on Kinetics-600 (Carreira et al. 2018). Following X3D (Feichten- hofer 2020), sample 30 clips from each input video when evaluating X3I models on Kinetics-
21

---

Published as a conference paper at ICLR 2022
600. The 30 clips are the combination of 10 uniformly sampled temporal crops and 3 spatial crops. The final prediction is the mean all individual predictions.
F.2 SEMANTIC SEGMENTATION
We notice that many pixels are unlabeled in semantic segmentation datasets, e.g., Cityscapes (Cordts et al. 2016), and are ignored during training and evaluation. These unla- beled pixels may introduce noise when we average the confidence score of all the pixels. To filter out unlabeled pixels in the image, we only consider pixels whose confidence is higher than a preset threshold junlab So we update the definition of gdense (.) as follows: goanse (R)
Confidence Function.
Eperi R ((pp),
[
where R' {p g(ap))
junlab , E R).
Experimental Details. We conduct experiments on the Cityscapes (Cordts et al., 2016) dataset, where the full image resolution is 1024 We train DeepLabv3 (Chen et al. 2017) mod- els on the train set of Cityscapes and report the mean IoU (mloU) over classes on the validation The threshold gunlab filter out unlabeled pixels is 3 that the first 7x7 convolution is changed three 3x3 convolutions (see resnet b in the official DeepLab imeplmentation"" ).
0.5. For DeepLaby3-ResNet--00 or
we follow the original architecture of ResNet-50 or ResNet-101, except
core/reenet_vl betett_____bee_p____ py
22


Reviews:
[{"review_id": "xmGVr-jy24c", "reviewer": "Reviewer_62w1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strong points\nThe paper presents many experimental results and offers an interesting view on ensembling and cascades. \nThe paper is well written and understandable.\n\n\n## Weak points\nGiven that efficiency is of core concern it would be good to at least discuss the concept and define what is referred to in the paper as efficient.\nE.g. is it the amount of electric power used in training and inference? (Which it is not).\n\nGiven that ensembling is well-known (as correctly pointed out by the authors) and wanting to establish a new interpretation of a known concept I would expect more intuition on why the presented interpretation seems fit.\n\nI am lacking concrete guidelines for a researcher or practitioner that go beyond existing knowledge.\n\nWhile the performed experiments are rather broad in scope, the evaluation protocol is not.\n\nOther ensembling approaches could be easily added, such as majority or variation ratio. How to set the threshold is also not explored in similar detail.\n\nEnsembling is to me rather orthogonal to the approaches it is compared with in this paper (i.e. model scaling).\n\n\n## Detailed comments\nEnsembling for classification can be done in multiple different ways, see e.g. Beluch, William H., et al. \"The power of ensembles for active learning in image classification.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\n\nUsing a confidence threshold for generating the cascade is justified with the simplicity, i would argue the approach referenced from Streeter (2018) - which is \"Margin sampling\"\nsee T. Scheffer, C. Decomain, and S. Wrobel. Active hidden Markov models for information extraction. In Proceedings of the International Conference on Advances\nin Intelligent Data Analysis (CAIDA), pages 309–318. Springer-Verlag, 2001. - is also rather simple. Given the amount of experiments conducted i wonder why such a worthy comparison was skipped.\n\nThe cascades approach seems interesting, but is in my view rather is an approach how to generate an ensemble, whichs size is determined by an uncertainty measure of its first members.\nHere the question arises which uncertainty measure is to be chosen, as pointed out before.\n\nWhy do you report average FLOPs to compare cascade and ensemble? How about total FLOPs over the images in the test set?\n\nWould be good to compare the number of parameters as well.\n\nI feel it would help the paper to strip down a bit on the amount of results included in the main text in favor of more details on why specific choices were made and conclusions taken from the results.\nSome of this (also some points I raised) are hidden in the appendix, which feels suboptimal to me.\n\n\n## Questions\n* How was the speed-up in the tables calculated?\n* How were the confidence thresholds t_i tuned?\n* Sem. seg. why average confindence score? Did you try others?\n* Could be interesting to plot the size of the resulting cascade over threshold\n\n## Minor comments:\n\n* Would be easier to read if r specifying the grid would be a different letter than r_1 and r_2 for the resolution of the self-cascade.\n* Caption figure 2: typo: \"cost-effectiev\"", "summary_of_the_paper": "This paper introduces ensembles as an option the reduce the amount of FLOPs while increasing or keeping the accuracy.", "summary_of_the_review": "This paper offers a new and interesting view on the well-known ensembling of DNNs. Some details are not entirely clear to me, details see above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LqH8iWrfCEt", "reviewer": "Reviewer_KTja", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Using early-exit cascades to improve average efficiency, equipped with a large handful of models and means to evaluate their combinations, is a simple and practical approach.  This paper clearly demonstrates their effectiveness and profiles the degree of their effect in several scenarios, including comparisons with larger single models, NAS, and differences between small and large models.\n\nWhile this paper does a good job in profiling results of this approach, I would have liked to see more on variation in the search and search space.  The exhaustive search in cascade selection appears important, but I don't see any evaluation on how important it is.  What is the performance variation among cascades in the search (particularly same-size cascades)?  What if one starts with a smaller model pool --- will results be similar or worse, and by how much, depending on the number of models trained?  Or, if one trains just 2 same-size models and uses them in a cascade with searched exit threshold, what is the expected performance and its possible variation?  How many model instances are needed in the search pool for obtaining good results with high enough probability?\n\nThe results from this paper also appear to conflict with those in Huang et al 2018 \"Multi-scale dense networks for resource efficient image classification.\":  On p.8 and Fig 7, that paper says \"We also evaluate an ensemble of the five ResNets that uses exactly the same dynamic-evaluation procedure as MSDNets at test time ....\", describing a similar cascade to this paper, and see no improvement in the average case, though the models used appear to be smaller than those investigated in this work (which this work says is a factor in effectiveness of non-cascade ensembles).  Are there any other causes for this difference?\n\nOverall, the measurements provided here demonstrating the effectiveness of this simple and sometimes overlooked approach, are welcome evaluations.  I would have liked to see more on the importance of the search and variation between model configurations.  However, I think there is significant practical interest to see measurements that establish the effects of cascades from a candidate model pool, in the context of current classification models.\n\n\n\nMinor questions:\n\n\nsec 3 ensembles:  How many models were trained in the pool for each architecture, and how many combinations were evaluated?\n\nsec 3 ensembles:  Were evaluations of the multiple combinations are performed on the test set, and pareto-optimal points chosen and shown as final performance for the same test set?  This would result in potentially better perf than if a separate validation set were used.  (Though in the context of the overall paper, seems a relatively minor point, since a validation set was indeed used for the cascade).\n\n\nsec 7.2 semenatic segmentation:  With grid subdivisions, it's a little unclear what is the image resolution for each model, and if any models (just the first or any beyond the first) take the entire image as input to be able to incorporate larger context regions.  If 128x128 does not have enough context, for example, it seems possible the second stage of a cascade might only degrade performance compared to the initial model that used the full image.  Were there any effects like this?\n\n", "summary_of_the_paper": "This paper investigates the effectiveness of model cascades in computation/accuracy tradeoff improvement.  A straightforward procedure is used, where all combination-permutations of a handful of models are evaluated in a cascade, and exit thresholds are determined by choosing best computation work within accuracy degradation constraint (or, best accuracy given computation constraint).  The resulting cascades perform significantly better than larger single models (more accurate or fewer flops depending on comparison).  Most significantly, the paper provides extensive evaluations on the degree of these gains for three model families (EfficientNet, Resnet, and MobileNetV2).\n", "summary_of_the_review": "Overall, the measurements provided here demonstrating the effectiveness of this simple and sometimes overlooked approach, are welcome evaluations.  I would have liked to see more on the importance of the search and variation between model configurations.  However, I think there is significant practical interest to see measurements that establish the effects of cascades from a candidate model pool, in the context of current classification models.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AqiCPnMLhKv", "reviewer": "Reviewer_MpvP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n\n+ the study of the efficiency-accuracy trade-off in shallow network compositions is a relevant topic;\n+ the empirical study on ImageNet is thorough and convincing;\n+ nice extensions to NAS-based models and related tasks of video recognition and semantic segmentation;\n+ generally very thoughtful experimental setups.\n\n**Cons:**\n\n> Significance: While I appreciate the thoroughness of the study, I have difficulties recognising the significance of the new insight gauged w.r.t. prior work. Compared to (Kondratyuk et al., 2020; Lobacheva et al., 2020), this work simply augments those studies to larger-scale classification, video recognition and semantic segmentation, with ImageNet experiments being by far the dominant content (7 out of 8 pages).\n\n> Reproducibility: This work would be likely very hard to reproduce. The text does not provide any training details (e.g. the training duration, the learning rate and schedule, regularisation, etc.).\nPerhaps more importantly, how were the ensemble models trained? Was there any bootstrapping used (e.g. see Ilg et al., (2018)), diversity-encouraging loss terms (e.g., Lee et al., (2016)) or snapshot ensembling? Why or why not? \nRelated to both the significance and reproducibility: I wonder how much deviation is there on the evaluation metrics (e.g., of the top-1 accuracy) in ensemble/cascade models?\n\n> Choice of the baseline and metrics: 1) I wonder how cascades/ensembles fare against the more efficient types of temporal ensembling (e.g. SWA from Izmailov et al., (2019))? 2) I’m curious what happens to uncertainty estimates. The choice of the max-based confidence function is interesting, but why is this a reasonable choice? The observed improvement in accuracy that it provides may come at a cost of deteriorated calibration; this needs to be verified. \n\n> Clarity. I find it rather unusual to seek ensemble configurations satisfying a particular value of accuracy or the number of FLOPs. It seems only to introduce additional experimental setups that deliver little new insight; there is unnecessary complexity of constraining the ensemble/cascade set, and it’s not always clear which models end up in the composition. I may be overlooking something, but the simple accuracy-efficiency plots (like Fig. 1 or Fig. 2) are already sufficient to draw the same conclusions and are simpler to interpret.\n\nMinor:\n- E1 the missing x30 factor in Tab. 8 is important, even though it does not change the ranking.\n- Semantic segmentation: Why is it reasonable to change 7x7 convolution to 3x3?\n- A number of typos (e.g. “cost-effectiev” on p. 4; “This methods does not reqire” on p. 2).\n- “we only show those Pareto optimal ensembles in the figure” needs more explanation. Which models/configurations were evaluated but were not Pareto optimal?\n\n\n**Post-rebuttal comment**\\\nI thank the authors for the clarifications. I recognise that the work provides significant empirical evidence (indicated in some previous findings) that compositions of shallower models may be both more accurate and more efficient. I also acknowledge that the revision now includes important details for reproducibility. However, a few things could be improved:\n\n- I agree with the other reviewers that the clarity of presentation should be improved significantly. For example, I find it bizarre to refer to Fig. 2 without introducing the architectures (and justifying their choice), how they were trained and selected for composition (i.e. the cascades are introduced only much later), how the hyperparameters are chosen (e.g. the thresholds), etc. These points are discussed in the paper, but either much later or in different contexts.\n- given the available computational resources and that most of the empirical advantages come from ImageNet, it would be interesting to verify these results on another classification dataset (e.g. Open Images, Pascal 2012 for multi-label classification, or at least CIFAR-100).\n- model/ensemble calibration needs more analysis. As highlighted in the paper (and in the rebuttal), the calibration of the models used contradicts the view on the networks being overconfident. However, the work makes no effort to explain this finding. Is it an artefact of the dataset? The training schedule? How does it affect the calibration of the cascades/ensembles?\n- minor: there are still quite a number of typos.\n\nNevertheless, I upgrade my score, since the remaining points can be addressed in the camera-ready.", "summary_of_the_paper": "The work provides an empirical accuracy-efficiency comparison of model ensembles and cascades of shallow models against single deeper models. The main finding, which supports previous results, is rather interesting: compositions of shallow models tend to provide better efficiency-accuracy trade-off than single deep models. This finding has been extended here to the ImageNet classification task with three architecture families (ResNet, EfficientNet and MobileNet), with further examples on video recognition and semantic segmentation.", "summary_of_the_review": "I find that this work provides a thorough study of the efficiency-accuracy trade-off offered by ensembles of shallow models w.r.t. single deeper models. However, I have difficulties recognising the significance of the new insights this study delivers in light of the previous work (e.g. Lobacheva et al., (2020))", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Nice paper, providing a thorough investigation of a simple idea that may be useful to a wide range of practitioners. All reviewers are positive, and the discussion has led to significant improvements in exposition and overall in the quality of the submission.
Decision: Accept (Poster)