Hybrid Cloud-Edge Networks for Efficient Inference

1 INTRODUCTION . Deep Neural Networks ( DNNs ) achieve state-of-the-art ( SOTA ) performance on challenging tasks such as image recognition ( Tan & Le , 2019 ; Howard et al. , 2019 ) , language modelling ( Devlin et al. , 2018 ) , and machine translation ( Wu et al. , 2016b ) . High accuracy on such tasks often comes at a high memory and compute cost , making DNN deployment on low resource edge hardware like microcontroller units ( MCUs ) very challenging ( Banbury et al. , 2021 ; Fedorov et al. , 2019 ; Lin et al. , 2020 ; Fedorov et al. , 2020 ; Gural & Murmann , 2019 ) . Edge-Device and Cloud ML As such , two paradigms currently co-exist . Edge-device inference utilizes lightweight architectures and focuses on low-level tasks such as smart messaging and face recognition ( Google LLC , 2021 ) . In parallel , more complex and nuanced tasks are deployed in the cloud , which is a term we use to refer to over-provisioned hardware platforms like a GPU server ( Alemi , 2016 ) . The fundamental drawback of cloud ML , however , is the increased latency and energy consumption arising from communication , which can be prohibitive for many applications . Indeed , the meager cost , size , and power requirements of MCUs make them the platform of choice for a large number of applications , so that MCU shipments outnumber GPU shipments by roughly 50 to 1 ( Fedorov et al. , 2019 ; Lin et al. , 2020 ) . While methods like pruning ( Molchanov et al. , 2017 ; Han et al. , 2015 ) , quantization ( Jacob et al. , 2018 ) , knowledge distillation ( Hinton et al. , 2015 ) , and adaptive computation ( Bejnordi et al. , 2019 ) could be leveraged to reduce the size of cloud-based models , these strategies fundamentally limit the resulting achievable accuracy due to the reduced model capacities ( Fedorov et al. , 2019 ; Lin et al. , 2020 ) . For example , SOTA accuracy on ImageNet is 84.3 % ( Tan & Le , 2019 ) with 37B FLOPs ; whereas the best deployable model on an STM32F746 MCU under 5 frames per second constraint achieves 51.1 % accuracy with 12.8M FLOPs ( Lin et al. , 2020 ) . Hybrid Edge-Cloud Inference . Motivated by these emerging trends , we propose a best-of-both hybrid solution , which allows for deploying cloud-based AI tasks on edge devices like MCUs , while lowering the average total latency ( the sum of communication and computational latency ) . Our end-to-end hybrid approach involves architecting and training three distinct networks in concert , individually named ‘ base ’ , ‘ global ’ , and ‘ routing ’ ( Fig . 1 ) . The base model is compact and designed for devices with low-resource hardware constraints like latency and memory usage . In contrast , the global model has a large capacity and is deployed in the resource-rich cloud environment . Finally , the routing model , which is very compact , is used to decide whether a query should be communicated to the global model , or handled entirely by the base model , thus enabling the two to work in tandem to maximise performance while controlling usage of computational or communication resources ( see Fig . 3 ) . Ideally , the global and base models are fine- tuned to be most accurate on the queries routed specifically to them , while the routing model in turn only issues queries to the global model when they are too hard to be processed by the base model . Technical Contributions .. Training Methodology . Learning an efficient hybrid model requires us to solve the challenging problem of discovering regions of ‘ easy ’ queries , where classification can reliably be performed by a simple model . In addition , training is challenging since a choice of base and global model affects the optimal routing , which cyclically affects the former . We propose an alternating-optimisation scheme that can be modularly executed , and design an efficient proxy supervision for the router , which together allow for simplified training . The result is a flexible scheme that can be used either to train all three models or only a subset . The routing model is further designed with a flexible assignment criterion that allows efficiently trading-off between a range of accuracy and resource usages , thus yielding a variety of operating points with a single training round . Neural Architecture Search ( NAS ) . Hybrid design also raises a novel architectural issue - while present DNN architectures are aligned towards a single model that performs end-to-end inference , a hybrid scheme may require a coupled design for the base and global models to best exploit the available resources . To discover efficient joint designs , we propose a NAS method that utilises an efficient proxy score to quickly determine fitness of a pair of base and global architectures , and performs an evolutionary search to optimise the accuracy at any given combined resource usage . Our approach is flexible and can also be used to , e.g. , adapt a base architecture to a given global model . Empirical Validation . Proposed above is a novel end-to-end methodology to design and train resourceefficient hybrid architectures and models that can perform at SOTA accuracy whilst satisfying hardware constraints . Extensive experimentation on the ImageNet dataset shows that the resulting scheme pareto dominates methods that learn a single efficient architecture , demonstrating 2− 3.5 % accuracy gains at any FLOP count when compared to prior efficient architecture designs such as MobileNetV3 and OFA ( Fig . 3 , 4 ) . Further , in settings where the global device has much higher compute capacity ( hence negligible inference cost ) , we show that whilst processing 70 % of queries at the base , our design can match the accuracy of these designs with up to 4.5× improvement in base FLOPs ( Fig . 2 ) . RELATED WORK . Efficient Architectures . Previous works have designed low complexity DNNs for mobile applications ( Iandola et al. , 2016 ; Gholami et al. , 2018 ; Sandler et al. , 2018 ; Howard et al. , 2019 ) using low-rank decomposition , separable convolutions , and hand-crafted feature blocks . Tan & Le ( 2019 ) study the effect of width , depth , channels , and input resolution on DNN memory and FLOP costs . In parallel , neural architectures searches under constraints such as FLOP ( Liu et al. , 2017 ; Zoph & Le , 2016 ; Dong & Yang , 2019 ; Elsken et al. , 2019 ) , latency ( Cai et al. , 2020 ) , memory ( Fedorov et al. , 2019 ) , etc have been carried out . These efforts are complementary to our hybrid scheme since we can leverage these improved architectures as base models and achieve similar gains in performance . Low Compute Transformations . Researchers have explored methods to obtain low-capacity models from SOTA DNNs , including compression and pruning ( Han et al. , 2016 ) , quantization ( Wu et al. , 2016a ) , hashing ( Chen et al. , 2015 ) , and knowledge distillation ( Hinton et al. , 2015 ) . Since these transformations are orthogonal to our proposal and can be leveraged post-hoc , we do not pursue these techniques in order to simplify our exposition . Adaptive Neural Networks . Han et al . ( 2021 ) present a comprehensive survey on designing dynamic neural networks that budget more computational resources for harder examples . These include ( a ) cascade-based early exit networks ( Park et al. , 2015 ; Bolukbasi et al. , 2017 ; Nan & Saligrama , 2017 ; Wang et al. , 2018 ) , where the constituent networks are independently designed and do not share intermediate features ; ( b ) early exit networks ( Teerapittayanon et al. , 2017 ; Dai et al. , 2020 ; Li et al. , 2019 ) where classifiers are introduced at intermediate layers ; and ( c ) multi-scale networks with early exits ( Huang et al. , 2018 ; Yang et al. , 2020 ) , which are allowed to operate at different input resolution , width or depth . Kang et al . ( 2017 ) splits network execution between device and cloud , resulting in higher communication as the features require more storage than input . Further , a high-performing model can neither be stored nor executed on a constrained MCU due to low RAM and Flash . In the context of our problem , much of the focus in these works is on scaling-up capacity without a proportional increase in inference time . Although our proposed hybrid approach bears resemblance to these works , we are focused on the opposite scenario , namely , how to overwhelmingly reduce resource usage ( FLOPs and communication latency ) to allow for deployment on edge devices without degrading accuracy achievable by a large model . As such , our perspective necessitates posing an end-to-end system-wide hybrid objective and requires systematic integration and optimization of all of the degrees of freedom ( architectures , routing & coverage , base and global networks ) . In contrast , prior works optimize these aspects in a decoupled and isolated manner . As a case in point , works focusing on architectures and early exit networks utilize simple entropy thresholding for routing . As we will show in our results , carefully designed routing schemes , which are jointly optimized along with base and global models , can result in substantial gains over entropy thresholding . Li et al . ( 2021 ) model a hybrid system with a similar design but missing crucial details , namely , ( a ) no coverage penalty in the train loss , ( b ) router is entangled with the base and global network , while we decouple it using the routing oracle , and ( c ) no evaluations on Imagenet ( see Appendix §B.7 for details ) . Learning with Abstention . Many researchers ( Liu et al. , 2019 ; Gangrade et al. , 2021 ; Geifman & El-Yaniv , 2019 ) have studied the problem of learning with a reject option , where a model can abstain prediction on some examples with the goal of minimizing the number of errors and abstentions . Although we get an abstaining classifier from the hybrid model by simply ignoring the global model , our main objective is to improve the performance of the hybrid system that includes the global model . 2 METHOD . Let X be a feature space and Y a set of labels . A hybrid design is composed of three models : • A base model b : X → Y , that can be deployed on an edge device . • A global model g : X → Y , that is deployed in the cloud and typically has high accuracy . • A routing model r : X → { 0 , 1 } , that is a very low resource model deployed alongside the base model , and routes hard queries to the global model . We will treat these models as soft classifiers , outputting |Y|-dimensional scores { by } and { gy } , and two scores r0 ( x ) and r1 ( x ) for the routing model . In this paper , r is realized by a 2-layer DNN with input by ( x ) . The default hard output for the base is the top entry b ( x ) = arg maxy by ( x ) , and similarly for g. By default r assigns x to the global model if r1 ( x ) > r0 ( x ) , i.e. , r ( x ) = 1 { r1 ( x ) > r0 ( x ) } , but this can be relaxed to r ( x ; t ) : = 1 { r1 ( x ) > t + r0 ( x ) } , where the hyper-parameter t allows a routing model to trade-off accuracy and resource usage in order to avoid separately training for each desired level . The decision produced by the system for an instance x ∈ X is ŷ ( x ) : = ( 1− r ( x ) ) b ( x ) + r ( x ) g ( x ) . ( 1 ) The coverage of the hybrid system is the fraction of instances that are processed by the base only , i.e . C ( r , b , g ) : = P ( r ( X ) = 0 ) , where P denotes the joint law over ( X , Y ) . The hybrid accuracy is A ( r , b , g ) = P ( ŷ ( X ) = Y ) = P ( r ( X ) = 0 , b ( X ) = Y ) + P ( r ( X ) = 1 , g ( X ) = Y ) . Architectures and Costs . The resources required to evaluate a DNN are mainly a function of its architecture - the number and arrangement of its layers and weights . We use α to denote a generic architecture , and say that a model f ∈ α if it is realizable by this architecture . To quantify the resource consumption , letR ( α ) denote the cost per inference for a model with architecture α . In our design , the base is always executed , with the output fed into the routing model . Therefore , the hybrid FLOP count of a hybrid model ( r , b , g ) , such that b ∈ αb and g ∈ αg , is R ( r , b , g ) : = Rr +R ( αb ) + ( 1− C ( r , b , g ) ) R ( αg ) ( 2 ) whereRr is a fixed , small quantity required to execute r. We can model many resources , including the FLOPs required to execute the model , and the latency on edge devices like MCUs ( Banbury et al. , 2021 ) . Additionally , for settings where the global model is on the cloud with a compute-rich environment , the resource costs are dominated by communication latency , modeled as R ( αg ) = τ , R ( αb ) = 0 ( where τ is the mean communication delay ) . We use coverage as a proxy metric for communication latency as it measures the data split between the base and global . In the following , we will focus on FLOP and coverage metrics , although we investigate inference latency in §3.4 .. Overall Formulation Let Ab and Ag be sets of base and global architectures , which may incorporate implementation restrictions , and % a target resource usage level . Our objective is max αb∈Ab , αg∈Ag max r , b∈αb , g∈αg A ( r , b , g ) s.t . R ( r , b , g ) ≤ % . ( 3 ) The outer maximisation over ( αb , αg ) in ( 3 ) amounts to an architecture search , while the maximisation over ( r , b , g ) in a fixed architecture corresponds to learning a hybrid model . The following sections describe our method for solving ( 3 ) . Briefly , we propose to decouple the inner and outer optimization problems in ( 3 ) for the sake of efficiency - hybrid models are trained by an empirical risk minimisation ( ERM ) strategy , whilst the architecture search is carried out using fast proxies for the accuracy attainable by a given pair of architectures without directly training hybrid models .

Summary: The paper proposes to combine the inference on edge and cloud together, taking advantage of the communication-free inference on edge and the high-accuracy model on cloud. It utilizes the OFA to obtain models with different resource requirement at low cost. It achieves higher accuracy on the ImageNet dataset compared to the edge-only inference.