Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models

1 INTRODUCTION Optimizing the efficiency of neural networks is important for real-world applications as they can only use limited computational resources and often have requirements on response time . There has been considerable work in this direction ( Howard et al. , 2017 ; Zhang et al. , 2018 ; Tan & Le , 2019 ) , but they mostly focus on designing novel network architectures that can achieve a favorable speed-accuracy trade-off . Here , we do not present any novel method or architecture design . Instead , we focus on analyzing the accuracy and efficiency of a simple paradigm : committeebased models . We use the term “ committee ” to refer to model ensembles or cascades , which indicates that they are built using multiple independent models . Committee-based models have been extensively studied and used before deep learning ( Breiman , 1996 ; Schapire , 1990 ; Freund & Schapire , 1997 ; Viola & Jones , 2001 ) . However , when com∗Work done during an internship at Google . †Work done as part of the Google AI Residency Program . paring the efficiency of deep models , committee-based models are rarely considered in recent work ( Howard et al. , 2017 ; Zhang et al. , 2018 ; Tan & Le , 2019 ) . There still lacks a systematic understanding of their efficiency in comparison with single models – models that only use one network . Such an understanding is informative for both researchers to push the frontier of efficient models and practitioners to select model designs in real-world applications . To fill this knowledge gap , we conduct a comprehensive analysis of the efficiency of committeebased models . To highlight the practical benefit of committee-based models , we intentionally choose the simplest possible method , which directly uses off-the-shelf , independently pre-trained models to build ensembles or cascades . We ensemble multiple pre-trained models via a simple average over their predictions ( Sec . 3 ) . For cascades , we sequentially apply each model and use a simple heuristic ( e.g. , maximum probability in the prediction ) to determine when to exit from the cascade ( Sec . 4 ) . We show that even this method already outperforms state-of-the-art architectures found by costly neural architecture search ( NAS ) methods . Note that this method works with off-the-shelf models and does not use specialized techniques . For example , it differs from Boosting ( Schapire , 1990 ) where each new model is conditioned on previous ones , and does not require the weight generation mechanism in previous efficient ensemble methods ( Wen et al. , 2020 ) . This method does not require the training of an early exit policy ( Bolukbasi et al. , 2017 ; Guan et al. , 2018 ) or the specially designed multi-scale architecture ( Huang et al. , 2018 ) in previous work on building cascades . To be clear , the contribution of this paper is not in the invention of model ensembles and cascades , as they have been known for decades , and is not in a new proposed method to build them . Instead , it is in the thorough evaluation and comparison of committee-based models with commonly used model architectures . Our analysis shows that committee-based models provide a simple complementary paradigm to achieve superior efficiency without tuning the architecture . One can often improve accuracy while reducing inference and training cost by building committees out of existing networks . Our findings generalize to a wide variety of tasks , including image classification , video classification , and semantic segmentation , and hold true for various architecture families : ViT ( Dosovitskiy et al. , 2021 ) , EfficientNet ( Tan & Le , 2019 ) , ResNet ( He et al. , 2016 ) , MobileNetV2 ( Sandler et al. , 2018 ) , and X3D ( Feichtenhofer , 2020 ) . We summarize our findings as follows : • Ensembles are more cost-effective than a single model in the large computation regime ( Sec . 3 ) . For example , an ensemble of two separately trained EfficientNet-B5 models matches B7 accuracy , a state-of-the-art ImageNet model , while having almost 50 % less FLOPs ( 20.5B vs. 37B ) . • Cascades outperform single models in all computation regimes ( Sec . 4 & 5 ) . Our cascade matches B7 accuracy while using on average 5.4x fewer FLOPs . Cascades can also achieve a 2.3x speedup over ViT-L-384 , a Transformer architecture , while matching its accuracy on ImageNet . • We further show that ( 1 ) the efficiency of cascades is evident in both FLOPs and on-device latency and throughput ( Sec . 5.1 ) ; ( 2 ) cascades can provide a guarantee on worst-case FLOPs ( Sec . 5.2 ) ; ( 3 ) one can build self-cascades using a single model with multiple inference resolutions to achieve a significant speedup ( Sec . 6 ) . • Committee-based models are applicable beyond image classification ( Sec . 7 ) and outperform single models on the task of video classification and semantic segmentation . Our cascade outperforms X3D-XL by 1.2 % on Kinetics-600 ( Carreira et al. , 2018 ) while using fewer FLOPs . 2 RELATED WORK . Efficient Neural Networks . There has been significant progress in designing efficient neural networks . In early work , most efficient networks , such as MobileNet ( Howard et al. , 2017 ; Sandler et al. , 2018 ) and ShuffleNet ( Howard et al. , 2019 ) , were manually designed . Recent work started to use neural architectures search ( NAS ) to automatically learn efficient network designs ( Zoph et al. , 2018 ; Cao et al. , 2019 ; Tan et al. , 2019 ; Tan & Le , 2019 ; Chaudhuri et al. , 2020 ) . They mostly fcous on improving the efficiency of single models by designing better architectures , while we explore committee-based models without tuning the architecture . Ensembles . Ensemble learning has been well studied in machine learning and there have been many seminal works , such as Bagging ( Breiman , 1996 ) , Boosting ( Schapire , 1990 ) , and AdaBoost ( Freund & Schapire , 1997 ) . Ensembles of neural networks have been used for many tasks , such as image classification ( Szegedy et al. , 2015 ; Huang et al. , 2017a ) , machine translation ( Wen et al. , 2020 ) , active learning ( Beluch et al. , 2018 ) , and out-of-distribution robustness ( Lakshminarayanan et al. , 2017 ; Fort et al. , 2019 ; Wenzel et al. , 2020 ) . But the efficiency of model ensembles has rarely been systematically investigated . Recent work indicated that ensembles can be more efficient than single models for image classification ( Kondratyuk et al. , 2020 ; Lobacheva et al. , 2020 ) . Our work further substantiates this claim through the analysis of modern architectures on large-scale benchmarks . Cascades . A large family of works have explored using cascades to speed up certain tasks . For example , the seminal work from Viola & Jones ( 2001 ) built a cascade of increasingly complex classifiers to speed up face detection . Cascades have also been explored in the context of deep neural networks . Bolukbasi et al . ( 2017 ) reduced the average test-time cost by learning a policy to allow easy examples to early exit from a network . A similar idea was also explored by Guan et al . ( 2018 ) . Huang et al . ( 2018 ) proposed a specially designed architecture Multi-Scale DenseNet to better incorporate early exits into neural networks . Given a pool of models , Streeter ( 2018 ) presented an approximation algorithm to produce a cascade that can preserve accuracy while reducing FLOPs and demonstrated improvement over state-of-the-art NAS-based models on ImageNet . Different from previous work that primarily focuses on developing new methods to build cascades , we show that even the most straightforward method can already provide a significant speedup without training an early exit policy ( Bolukbasi et al. , 2017 ; Guan et al. , 2018 ) or designing a specialized multi-scale architecture ( Huang et al. , 2018 ) . Dynamic Neural Networks . Dynamic neural networks allocate computational resources based on the input example , i.e. , spending more computation on hard examples and less on easy ones ( Han et al. , 2021 ) . For example , Shazeer et al . ( 2017 ) trained a gating network to determine what parts in a high-capacity model should be used for each example . Recent work ( Wu et al. , 2018 ; Veit & Belongie , 2018 ; Wang et al. , 2018 ) explored learning a policy to dynamically select layers or blocks to execute in ResNet based on the input image . Our analysis shows that cascades of pre-trained models are actually a strong baseline for dynamic neural networks . 3 ENSEMBLES ARE ACCURATE , EFFICIENT , AND FAST TO TRAIN . Model ensembles are useful for improving accuracy , but the usage of multiple models also introduces extra computational cost . When the total computation is fixed , which one will give a higher accuracy : single models or ensembles ? The answer is important for real-world applications but this question has rarely been systematically studied on modern architectures and large-scale benchmarks . We investigate this question on ImageNet ( Russakovsky et al. , 2015 ) with three architecture families : EfficientNet ( Tan & Le , 2019 ) , ResNet ( He et al. , 2016 ) , and MobileNetV2 ( Sandler et al. , 2018 ) . Each architecture family contains a series of networks with different levels of accuracy and computational cost . Within each family , we train a pool of models , compute the ensemble of different combinations of models , and compare these ensembles with the single models in the family . We denote an ensemble of n image classification models by { M1 , . . . , Mn } , where Mi is the ith model . Given an image x , αi = Mi ( x ) is a vector representing the logits for each class . To ensemble the n models , we compute the mean of logits1 αens = 1n ∑ i αi and predicts the class for image x by applying argmax to αens . The total computation of the ensemble is FLOPsens = ∑ i FLOPs ( Mi ) , where FLOPs ( · ) gives the FLOPs of a model . We show the top-1 accuracy on ImageNet and FLOPs of single models and ensembles in Figure 2 . Since there are many possible combinations of models to ensemble , we only show those Pareto optimal ensembles in the figure . We see that ensembles are more cost-effective than large single models , e.g. , EfficientNet-B5/B6/B7 and ResNet-152/200 . But in the small computation regime , single models outperform ensembles . For example , the ensemble of 2 B5 matches B7 accuracy while using about 50 % less FLOPs . However , ensembles use more FLOPs than MobileNetV2 when they have a similar accuracy . 1We note that the mean of probabilities is a more general choice since logits can be arbitrarily scaled . In our experiments , we observe that they yield similar performance with the mean of logits being marginally better . The findings in our work hold true no matter which choice is used . A possible explanation of why model ensembles are more powerful at large computation than at small computation comes from the perspective of bias-variance tradeoff . Large models usually have small bias but large variance , where the variance term dominates the test error . Therefore , ensembles are beneficial at large computation as they can reduce the variance in prediction ( Breiman , 1996 ) . For small models , the bias term dominates the test error . Ensembles can reduce the variance , but this can not compensate the fact that the bias of small models is large . Therefore , ensembles are less powerful at small computation . Our analysis indicates that instead of using a large model , one should use an ensemble of multiple relatively smaller models , which would give similar performance but with fewer FLOPs . In practice , model ensembles can be easily parallelized ( e.g. , using multiple accelerators ) , which may provide further speedup for inference . Moreover , often the total training cost of an ensemble is much lower than that of an equally accurate single model ( see appendix for more details ) .

Summary: This paper investigates the effectiveness of model cascades in computation/accuracy tradeoff improvement.  A straightforward procedure is used, where all combination-permutations of a handful of models are evaluated in a cascade, and exit thresholds are determined by choosing best computation work within accuracy degradation constraint (or, best accuracy given computation constraint).  The resulting cascades perform significantly better than larger single models (more accurate or fewer flops depending on comparison).  Most significantly, the paper provides extensive evaluations on the degree of these gains for three model families (EfficientNet, Resnet, and MobileNetV2).