Hybrid Cloud-Edge Networks for Efficient Inference

1 INTRODUCTION . Deep Neural Networks ( DNNs ) achieve state-of-the-art ( SOTA ) performance on challenging tasks such as image recognition ( Tan & Le , 2019 ; Howard et al. , 2019 ) , language modelling ( Devlin et al. , 2018 ) , and machine translation ( Wu et al. , 2016b ) . High accuracy on such tasks often comes at a high memory and compute cost , making DNN deployment on low resource edge hardware like microcontroller units ( MCUs ) very challenging ( Banbury et al. , 2021 ; Fedorov et al. , 2019 ; Lin et al. , 2020 ; Fedorov et al. , 2020 ; Gural & Murmann , 2019 ) . Edge-Device and Cloud ML As such , two paradigms currently co-exist . Edge-device inference utilizes lightweight architectures and focuses on low-level tasks such as smart messaging and face recognition ( Google LLC , 2021 ) . In parallel , more complex and nuanced tasks are deployed in the cloud , which is a term we use to refer to over-provisioned hardware platforms like a GPU server ( Alemi , 2016 ) . The fundamental drawback of cloud ML , however , is the increased latency and energy consumption arising from communication , which can be prohibitive for many applications . Indeed , the meager cost , size , and power requirements of MCUs make them the platform of choice for a large number of applications , so that MCU shipments outnumber GPU shipments by roughly 50 to 1 ( Fedorov et al. , 2019 ; Lin et al. , 2020 ) . While methods like pruning ( Molchanov et al. , 2017 ; Han et al. , 2015 ) , quantization ( Jacob et al. , 2018 ) , knowledge distillation ( Hinton et al. , 2015 ) , and adaptive computation ( Bejnordi et al. , 2019 ) could be leveraged to reduce the size of cloud-based models , these strategies fundamentally limit the resulting achievable accuracy due to the reduced model capacities ( Fedorov et al. , 2019 ; Lin et al. , 2020 ) . For example , SOTA accuracy on ImageNet is 84.3 % ( Tan & Le , 2019 ) with 37B FLOPs ; whereas the best deployable model on an STM32F746 MCU under 5 frames per second constraint achieves 51.1 % accuracy with 12.8M FLOPs ( Lin et al. , 2020 ) . Hybrid Edge-Cloud Inference . Motivated by these emerging trends , we propose a best-of-both hybrid solution , which allows for deploying cloud-based AI tasks on edge devices like MCUs , while lowering the average total latency ( the sum of communication and computational latency ) . Our end-to-end hybrid approach involves architecting and training three distinct networks in concert , individually named ‘ base ’ , ‘ global ’ , and ‘ routing ’ ( Fig . 1 ) . The base model is compact and designed for devices with low-resource hardware constraints like latency and memory usage . In contrast , the global model has a large capacity and is deployed in the resource-rich cloud environment . Finally , the routing model , which is very compact , is used to decide whether a query should be communicated to the global model , or handled entirely by the base model , thus enabling the two to work in tandem to maximise performance while controlling usage of computational or communication resources ( see Fig . 3 ) . Ideally , the global and base models are fine- tuned to be most accurate on the queries routed specifically to them , while the routing model in turn only issues queries to the global model when they are too hard to be processed by the base model . Technical Contributions .. Training Methodology . Learning an efficient hybrid model requires us to solve the challenging problem of discovering regions of ‘ easy ’ queries , where classification can reliably be performed by a simple model . In addition , training is challenging since a choice of base and global model affects the optimal routing , which cyclically affects the former . We propose an alternating-optimisation scheme that can be modularly executed , and design an efficient proxy supervision for the router , which together allow for simplified training . The result is a flexible scheme that can be used either to train all three models or only a subset . The routing model is further designed with a flexible assignment criterion that allows efficiently trading-off between a range of accuracy and resource usages , thus yielding a variety of operating points with a single training round . Neural Architecture Search ( NAS ) . Hybrid design also raises a novel architectural issue - while present DNN architectures are aligned towards a single model that performs end-to-end inference , a hybrid scheme may require a coupled design for the base and global models to best exploit the available resources . To discover efficient joint designs , we propose a NAS method that utilises an efficient proxy score to quickly determine fitness of a pair of base and global architectures , and performs an evolutionary search to optimise the accuracy at any given combined resource usage . Our approach is flexible and can also be used to , e.g. , adapt a base architecture to a given global model . Empirical Validation . Proposed above is a novel end-to-end methodology to design and train resourceefficient hybrid architectures and models that can perform at SOTA accuracy whilst satisfying hardware constraints . Extensive experimentation on the ImageNet dataset shows that the resulting scheme pareto dominates methods that learn a single efficient architecture , demonstrating 2− 3.5 % accuracy gains at any FLOP count when compared to prior efficient architecture designs such as MobileNetV3 and OFA ( Fig . 3 , 4 ) . Further , in settings where the global device has much higher compute capacity ( hence negligible inference cost ) , we show that whilst processing 70 % of queries at the base , our design can match the accuracy of these designs with up to 4.5× improvement in base FLOPs ( Fig . 2 ) . RELATED WORK . Efficient Architectures . Previous works have designed low complexity DNNs for mobile applications ( Iandola et al. , 2016 ; Gholami et al. , 2018 ; Sandler et al. , 2018 ; Howard et al. , 2019 ) using low-rank decomposition , separable convolutions , and hand-crafted feature blocks . Tan & Le ( 2019 ) study the effect of width , depth , channels , and input resolution on DNN memory and FLOP costs . In parallel , neural architectures searches under constraints such as FLOP ( Liu et al. , 2017 ; Zoph & Le , 2016 ; Dong & Yang , 2019 ; Elsken et al. , 2019 ) , latency ( Cai et al. , 2020 ) , memory ( Fedorov et al. , 2019 ) , etc have been carried out . These efforts are complementary to our hybrid scheme since we can leverage these improved architectures as base models and achieve similar gains in performance . Low Compute Transformations . Researchers have explored methods to obtain low-capacity models from SOTA DNNs , including compression and pruning ( Han et al. , 2016 ) , quantization ( Wu et al. , 2016a ) , hashing ( Chen et al. , 2015 ) , and knowledge distillation ( Hinton et al. , 2015 ) . Since these transformations are orthogonal to our proposal and can be leveraged post-hoc , we do not pursue these techniques in order to simplify our exposition . Adaptive Neural Networks . Han et al . ( 2021 ) present a comprehensive survey on designing dynamic neural networks that budget more computational resources for harder examples . These include ( a ) cascade-based early exit networks ( Park et al. , 2015 ; Bolukbasi et al. , 2017 ; Nan & Saligrama , 2017 ; Wang et al. , 2018 ) , where the constituent networks are independently designed and do not share intermediate features ; ( b ) early exit networks ( Teerapittayanon et al. , 2017 ; Dai et al. , 2020 ; Li et al. , 2019 ) where classifiers are introduced at intermediate layers ; and ( c ) multi-scale networks with early exits ( Huang et al. , 2018 ; Yang et al. , 2020 ) , which are allowed to operate at different input resolution , width or depth . Kang et al . ( 2017 ) splits network execution between device and cloud , resulting in higher communication as the features require more storage than input . Further , a high-performing model can neither be stored nor executed on a constrained MCU due to low RAM and Flash . In the context of our problem , much of the focus in these works is on scaling-up capacity without a proportional increase in inference time . Although our proposed hybrid approach bears resemblance to these works , we are focused on the opposite scenario , namely , how to overwhelmingly reduce resource usage ( FLOPs and communication latency ) to allow for deployment on edge devices without degrading accuracy achievable by a large model . As such , our perspective necessitates posing an end-to-end system-wide hybrid objective and requires systematic integration and optimization of all of the degrees of freedom ( architectures , routing & coverage , base and global networks ) . In contrast , prior works optimize these aspects in a decoupled and isolated manner . As a case in point , works focusing on architectures and early exit networks utilize simple entropy thresholding for routing . As we will show in our results , carefully designed routing schemes , which are jointly optimized along with base and global models , can result in substantial gains over entropy thresholding . Li et al . ( 2021 ) model a hybrid system with a similar design but missing crucial details , namely , ( a ) no coverage penalty in the train loss , ( b ) router is entangled with the base and global network , while we decouple it using the routing oracle , and ( c ) no evaluations on Imagenet ( see Appendix §B.7 for details ) . Learning with Abstention . Many researchers ( Liu et al. , 2019 ; Gangrade et al. , 2021 ; Geifman & El-Yaniv , 2019 ) have studied the problem of learning with a reject option , where a model can abstain prediction on some examples with the goal of minimizing the number of errors and abstentions . Although we get an abstaining classifier from the hybrid model by simply ignoring the global model , our main objective is to improve the performance of the hybrid system that includes the global model . 2 METHOD . Let X be a feature space and Y a set of labels . A hybrid design is composed of three models : • A base model b : X → Y , that can be deployed on an edge device . • A global model g : X → Y , that is deployed in the cloud and typically has high accuracy . • A routing model r : X → { 0 , 1 } , that is a very low resource model deployed alongside the base model , and routes hard queries to the global model . We will treat these models as soft classifiers , outputting |Y|-dimensional scores { by } and { gy } , and two scores r0 ( x ) and r1 ( x ) for the routing model . In this paper , r is realized by a 2-layer DNN with input by ( x ) . The default hard output for the base is the top entry b ( x ) = arg maxy by ( x ) , and similarly for g. By default r assigns x to the global model if r1 ( x ) > r0 ( x ) , i.e. , r ( x ) = 1 { r1 ( x ) > r0 ( x ) } , but this can be relaxed to r ( x ; t ) : = 1 { r1 ( x ) > t + r0 ( x ) } , where the hyper-parameter t allows a routing model to trade-off accuracy and resource usage in order to avoid separately training for each desired level . The decision produced by the system for an instance x ∈ X is ŷ ( x ) : = ( 1− r ( x ) ) b ( x ) + r ( x ) g ( x ) . ( 1 ) The coverage of the hybrid system is the fraction of instances that are processed by the base only , i.e . C ( r , b , g ) : = P ( r ( X ) = 0 ) , where P denotes the joint law over ( X , Y ) . The hybrid accuracy is A ( r , b , g ) = P ( ŷ ( X ) = Y ) = P ( r ( X ) = 0 , b ( X ) = Y ) + P ( r ( X ) = 1 , g ( X ) = Y ) . Architectures and Costs . The resources required to evaluate a DNN are mainly a function of its architecture - the number and arrangement of its layers and weights . We use α to denote a generic architecture , and say that a model f ∈ α if it is realizable by this architecture . To quantify the resource consumption , letR ( α ) denote the cost per inference for a model with architecture α . In our design , the base is always executed , with the output fed into the routing model . Therefore , the hybrid FLOP count of a hybrid model ( r , b , g ) , such that b ∈ αb and g ∈ αg , is R ( r , b , g ) : = Rr +R ( αb ) + ( 1− C ( r , b , g ) ) R ( αg ) ( 2 ) whereRr is a fixed , small quantity required to execute r. We can model many resources , including the FLOPs required to execute the model , and the latency on edge devices like MCUs ( Banbury et al. , 2021 ) . Additionally , for settings where the global model is on the cloud with a compute-rich environment , the resource costs are dominated by communication latency , modeled as R ( αg ) = τ , R ( αb ) = 0 ( where τ is the mean communication delay ) . We use coverage as a proxy metric for communication latency as it measures the data split between the base and global . In the following , we will focus on FLOP and coverage metrics , although we investigate inference latency in §3.4 .. Overall Formulation Let Ab and Ag be sets of base and global architectures , which may incorporate implementation restrictions , and % a target resource usage level . Our objective is max αb∈Ab , αg∈Ag max r , b∈αb , g∈αg A ( r , b , g ) s.t . R ( r , b , g ) ≤ % . ( 3 ) The outer maximisation over ( αb , αg ) in ( 3 ) amounts to an architecture search , while the maximisation over ( r , b , g ) in a fixed architecture corresponds to learning a hybrid model . The following sections describe our method for solving ( 3 ) . Briefly , we propose to decouple the inner and outer optimization problems in ( 3 ) for the sake of efficiency - hybrid models are trained by an empirical risk minimisation ( ERM ) strategy , whilst the architecture search is carried out using fast proxies for the accuracy attainable by a given pair of architectures without directly training hybrid models .

Summary: This paper introduces a hybrid Cloud-Edge design for Deep Neural Network (DNN) inference in which only a small portion of samples (more complicated ones) are sent to cloud for processing. Inference for most of the samples happens on the edge devices itself which reduces the task’s latency.  The paper provides an end-to-end training methodology for system’s components, a Neural Architecture Search (NAS) approach for the design of models deployed on the edge and the cloud. It also does an empirical evaluation of its design.