Hybrid Cloud-Edge Networks for Efficient Inference
Authors: Anil Kag
Igor Fedorov
Aditya Gangrade
Paul Whatmough
Venkatesh Saligrama
Venue: ICLR 2022 Submitted (2022)
Keywords: low-capacity model
large-scale prediction
efficient inference
hybrid networks
routing nets
coverage and latency
FLOPs

Abstract: Although deep neural networks (DNNs) achieve state-of-the-art accuracy on large-scale and fine-grained prediction tasks, they are high capacity models and often cannot be deployed on edge devices. As such, two distinct paradigms have emerged in parallel: 1) edge device inference for low-level tasks, 2) cloud-based inference for large-scale tasks. We propose a novel hybrid option, which marries these extremes and seeks to bring the latency and computational cost benefits of edge device inference to tasks currently deployed in the cloud. Our proposed method is an end-to-end approach, and involves architecting and training two networks in tandem. The first network is a low-capacity network that can be deployed on an edge device, whereas the second is a high-capacity network deployed in the cloud. When the edge device encounters challenging inputs, these inputs are transmitted and processed on the cloud. Empirically, on the ImageNet classification dataset, our proposed method leads to substantial decrease in the number of floating point operations (FLOPs) used compared to a well-designed high-capacity network, while suffering no excess classification loss. A novel aspect of our method is that, by allowing abstentions on a small fraction of examples ($<20\%$), we can increase accuracy without increasing the edge device memory and FLOPs substantially (up to $7$\% higher accuracy and $3$X fewer FLOPs on ImageNet with $80$\% coverage), relative to MobileNetV3 architectures.


Under review as a conference paper at ICLR 2022
HYBRID CLOUD-EDGE NETWORKS FOR EFFICIENT INFERENCE
Anonymous authors
Paper under double-blee buibee-blind review
ABSTRACT
Although deep neural networks (DNNs) achieve slate-of-the-art accuracy large- scale and fine-grained prediction tasks, they are high capacity models and often cannot be deployed edge devices. As such, two distinct paradigms have emerged in parallel: 1) edge device inference for low-level tasks, 2) cloud-based inference for large-scale tasks. We propose novel hybrid option, which marries these extremes and seeks to bring the latency and computational cost benefits of edge device inference to tasks currently deployed in the cloud. Our proposed method is an end-to-end approach, and involves architecting and training two networks in tandem. The first network is a low-capacity network that can be deployed on an edge device, whereas the second is a high-capacity network deployed in the cloud. When the edge device encounters challenging inputs, these inputs are transmitted and processed on the cloud. Empirically, on the ImageNet classification dataset, our proposed method leads to substantial decrease in the number of floating point operations (FLOPs) used compared to well-designed high- eighcappcity network, while suffering no excess classification loss. A novel aspect of our method is that, by allowing abstentions accuracy without increasing the edge device memory and FLOPs substantially (up 7% higher accuracy and 3X fewer FLOPs on ImageNet with 80% coverage), relative to MobileNetV3 architectures.
small fraction of examples (< 20%), we can increase
INTRODUCTION
Deep Neural Networks (DNNs) achieve state-of-the-ee art (SOTA) performance on challenging tasks
such as image recognition (Tan & Le| 2019 Howard et al. 2019), language modelling (Devlin et al.,
2018), and machine translation (Wu et al.
20T6b). High accuracy on such tasks often comes at
high memory and compute cost, making DNN deployment on low resource edge hardware like
microcontroller units (MCUs) very challenging (Banbury et al. 2021 Fedorov et al. 2019 Lin et al.
2020 Fedorov et al. 2020 Gural& & Murmann 2019).
Edge-Device and Cloud ML As such, two paradigms currently co-exist. Edge-device inference utilizes lightweight architectures and focuses on low-level tasks such smart messaging and face recognition (Google LLC 2021). In parallel, more complex and nuanced tasks are deployed in the cloud, which term we use to refer ever-provisioned hardware platforms like GPU server (Alemi 2016). The fundamental drawback of cloud ML, however, is the increased latency and energy consumption arising from communication, which can be prohibitive for many applications. Indeed, the meager cost, size, and power requirements of MCUs make them the platform of choice for large number of applications, so that MCU shipments outnumber GPU shipments by roughly 50 to (Fedorov et al. 2019 Lin et al. 2020).
While methods like pruning (Molchaov et al. 2017 Han et al., 2015), quantization (Jacob et al. 2018), knowledge distillation (Hinton et al. 2015). and adaptive computation (Bejnordi et al.) 2019) could be leveraged to reduce the size of cloud-based models, these strategies fundamentally limit the resulting achievable accuracy due to the reduced model capacities (Fedorov et al. 2019 Lin ct al.) 2020). For example, SOTA accuracy on ImageNet is 84.3% (Tan Le| (2019) with 37B FLOPs; whereas the best deployable model on STM32F746 MCU under 5 frames per second constraint achieves 51.1% % accuracy with 12.8M FLOPs (Lin et al. 2020)
Hybrid Edge-Cloud Inference. Motivated by these emerging trends, we propose best-of-both hybrid solution, which allows for deploying cloud-based Al tasks on edge devices like MCUs, while lowering the average total latency (the sum of communication and computational latency).

---

Under review as a conference paper at ICLR 2022
Our end-to-end hybrid approach involves architecting and training three distinct networks in concert, indi- vidually named 'base', 'global', and 'routing' (Fig. [ The base model is compact and designed for devices with low-resource hardware constraints like latency and memory usage. In contrast, the global model has large capacity and is deployed in the resource-rich cloud en- vironment. Finally, the routing model, which is very compact, used decide whether query should be communicated to the global model, or handled entirely by the base model, thus enabling the two to work in tandem to maximise performance while controlling us- age of computational or communication resources (see Fig. 3). Ideally, the global and base models are fine- tuned to be most accurate on the queries routed specifically to them, while the routing model in turn only issues queries to the global model when they too hard to be processed by the base model.
8(2)
Easy
a(x)
Hard
Figure 1: HYBRID MODEL. Cheap basc (b) and routing models (r) run on, e.g.. micro- controller; Expensive global model (g) runs on, e.g. a cloud server. r uses æ and features of b to decide if g evaluated not.
Technical Contributions.
Training Methodology. Learning an efficient hybrid model requires problem of discovering regions of "easy" queries, where classification can reliably be performed by a simple model. In addition, training challenging since a choice of base and global model affects the optimal routing, which cyclically affects the former. We propose an that can be modularly executed, and design an efficient proxy supervision for the router, which together allow for simplified training. The result is a flexible scheme that can be used either to train all three models or only a subset. The routing model further designed with a flexible assignment criterion that allows efficiently trading-off between a range of accuracy and resource usages, thus yielding a variety of operating points with a single training round.
solve the challenging
scheme
Neural Architecture Search (NAS). Hybrid design also raises a novel architectural issuc - while present DNN architectures aligned towards a single model that performs end-to-end inference, a hybrid scheme may
Top1 Imagenet Accuracy (Base @70% Coverage)
80
3x redoction
Imagenet Top1 Accuracy (8)
75
4.5x reduction
require coupled design for the base and global mod- els to best exploit the available resources. To discover
70
4x reduction
efficient joint designs, we propose NAS method that utilises an efficient proxy score to quickly determine
60
fitness of a pair of base and global architectures, and
performs an evolutionary search to optimise the ac- curacy at any given combined resource usage. Our approach is flexible and can also be used to, e.g.,
35
Hybrid
Baseline
50
50
adapt a base architecture to a given global model.
Multiiy.Aad Operations (Millions, log scale)
Figure 2: Base FLOP gains in the hy bydr system at different levels of accuracy w.r.t. stand-alone model, collated from Tables and (sce Appendix $B.5 for details); in 10 examples routed to cloud model, 70% reduction in communication latency w.r.t. the ImageNet dataset purely cloud-based service.
Empirical Validation. Proposed above is novel end-to-end methodology to design and train resource- efficient hybrid architectures and models that can per- form at SOTA accuracy whilst satisfying hardware constraints. Extensive experimentation pareto dominates methods that learn a single efficient architecture, demonstrating 2 - 3.5% accuracy gains at any FLOP count when compared to prior efficient architecture designs such as MobileNetV3 and OFA Fig. 3 D. Further, in settings where the global device has much higher compute capacity (hence negligible inference cost), we show that whilst processing 70% of queries the base, our design can match the accuracy of these designs with up to 4.5x improvement in base FLOPs (Fig. 2
that the resulting scheme
RELATED WORK
Efficient Architectures. Previous works have designed low complexity DNNs for mobile applica-
tions (landola et al.) 2016 Gholami et al. 2018 Sandler al.) 20188 Howard al. 2019) using
low-rank decomposition, separable convolutions, and hand-crafted feature blocks. Tan & Le 2019)
study the effect of width, depth, channels, and input resolution on DNN memory and FLOP costs. In
parallel, neural architectures searches under constraints such as FLOP (Lu et al. 2017 Zoph & Le
2016. Dong& & Yang. 2019 Elsken al. 2019), latency (Cai et al. 2020), memory (Fedorov et al.
2

---

Under review as a conference paper at ICLR 2022
2019), etc have been carried out. These efforts complementary to our hybrid scheme since leverage these improved architectures as base models and achieve similar gains in performance.
Low Compute Transformations. Researchers have explored methods to obtain low-capacity models from SOTA DNNs, including compression and pruning (Han et al., 2016). quantization (Wu et al. al., 2015). Since these
2016ab, hashing (Chen al. 2015), and knowledge distillation(Hintor
transformations are orthogonal to our proposal and can be leveraged post-hoe, we do not pursue these techniques in order to simplify our exposition.
Adaptive Neural Networks. Han et al. 2021) present a comprehensive survey on designing dynamic neural networks that budget more computational resources for harder examples. These include (a) auscade-based early exit networks (Park et al., 2015 Bolukbasi et al. 20177] Nan & Saligrama 2017]]
Wang et al. 2018). where the constituent networks are independently designed and do not share intermediate features; (b) early exit networks (Teerapitaaynon et al. 2017 Dai et al. 2020 Li et al.,
2019) where classifiers are introduced at intermediate layers; and (c) multi-scale networks with early exits (Huang al. 2018 Yang et al. (020), which are allowed to operate at different input resolution, width or depth. Kang et al. (2017) splits network execution between device and cloud, resulting in higher communication as the features require more storage than input. Further, a high-performing model can neither be stored nor executed on a constrained MCU due to low RAM and Flash.
In the context of our problem, much of the focus in these works is on scaling-up up capacity without proportional increase in inference time. Although our proposed hybrid approach bears resemblance to these works, are focused on the opposite scenario, namely, how to overwhelmingly reduce resource usage (FLOPs and communication latency) to allow for deployment on edge devices without degrading accuracy achievable by large model. As such, our perspective necessitates posing an end-to-end system-wide hybrid objective and requires systematic integration and optimization of all of the degrees of freedom (architectures, routing & coverage, base and global networks). In contrast, prior works optimize these aspects in a decoupled and isolated manner. As case in point, works focusing on architectures and early exit networks utilize simple entropy thresholding for routing. As we will show in our results, carefully designed routing schemes, which are jointly optimized along with base and global models, can result in substantial gains over entropy thresholding. [Li et al. (2021) model hybrid system with a similar design but missing crucial details, namely, (a) no coverage penalty in the train loss, (b) router is entangled with the base and global network, while we decouple it using the routing oracle, and (c) evaluations on Imagenet (see Appendix $B.7 for details).
Learning with Abstention. Many researchers [Lu et al., 2019 Gangrade et al. 2021 Geifman & El- 2019) have studied the problem of learning with reject option, where model can abstain prediction on some examples with the goal of minimizing the number of errors and abstentions. Although we get an abstaining classifier from the hybrid model by simply ignoring the global model, our main objective is to improve the performance of the hybrid system that includes the global model.
2
METHOD
Let X be a feature space and y set of labels. A hybrid design composed of three models:
A base model b: X + V/ that can be deployed on an edge device.
global model g X ++ +,, that is deployed in the cloud and typically has high accuracy.
A routing model r : X' + {0, 1}, that is a very low resource model deployed alongside the base model, and routes hard queries to the global model.
We will treat these models as soft classifiers, outputting | -imeenniioo scores (b,,) and {gy}, ). and two scores (x( and r (x) for the routing model. In this paper, is realized by 2-layer DNN with input by (x). The default hard output for the base is the top entry b(x)) = arg max, by (1r), and similarly
for g. By default assigns to the global model if (2r) > ro(x), i.e., r(x) = 1 (ri (a) > ro(x)), but this can be relaxed to r(x; () := I (ri (a)) > L + ro(x)): where the hyper-parameter / allows a routing model to trade-off accuracy and resource usage in order to avoid separately training for each desired level. The decision produced by the system for an instance x E X' is
((x) == (1 ((x)) + r(z)g(z).
The coverage of the hybrid system is the fraction of instances that are processed by the base only, i.c.
C(r, b, g) = P(r(X) 0),
3

---

Under review as a conference paper at ICLR 2022
where IP denotes the joint law over (X, Y). The hybrid accuracy is
A(r; b, (()(X) - Y) = P(r(X) 0, b(x) P(((())
Architectures and Costs. The resources required to evaluate a DNN are mainly a function of its architecture the number and arrangement of its layers and weights. We use a to denote a generic architecture, and say that model f E if it is realizable by this architecture. To quantify the resource consumption, let R((a) denote the cost per inference for a model with architecture a. In our design, the base is always executed, with the output fed into the routing model. Therefore, the hybrid FLOP count of a hybrid model (r, b, g), such that b E a6 and E ag. is
b(r,b,g) := R, + R(as)) + (1 C(r, b, R(a)))
where R, is fixed, small quantity required to execute r. We model many resources, including the FLOPs required to execute the model, and the latency on edge devices like MCUs (Banbury et al. 202Tp. Additionally, for settings where the global model is on the cloud with compute-rich environment, the resource costs are dominated by communication latency, modeled as R(ag) = r, ((a)) 0 (where is the mean communication delay). We use coverage as proxy metric for communication latency as it measures the data split between the base and global. In the following, we will focus on FLOP and coverage metries, although we investigate inference latency in $3.4.
Overall Formulation Let on and Sg be sets of base and global architectures, which may incorporate implementation restrictions, and @ a target resource usage level. Our objective is
max
max
A(r, b, g) s.t. R(r, b, S 0.
(3)
Og O,Ea,,, r.her6.9ECg
The outer maximisation over (as, (gg) in 3 amounts an architecture search, while the maximisation over (r, b, g) in a fixed architecture corresponds to learning a hybrid model. The following sections describe our method for solving (3). Briefly, we propose to decouple the inner and outer optimization problems in (3) for the sake of efficency- hybrid models are trained by an empirical risk minimisation (ERM) strategy, whilst the architecture search is carried out using fast proxies for the accuracy attainable by given pair of architectures without directly training hybrid models.
2.1 LEARNING HYBRID MODELS
This section focuses on training hybrid models for fixed architectures as, agg i.e. the inner problem max A(r,b, s.t. R(r, b. g) S e.
(4)
r,bEas.gEa, Since architectures are fixed in (4), the FLOP constraint amounts to a constraint on the hybrid coverage. As is standard, we will approach (4) via ERM over Lagrangian of relaxed losses.
However, a number of design considerations and issues need to be addressed before such an approach viable, as discussed below. The overall scheme is summarised in Algorithm in A
Alternating optimisation. Problem 44) has a cyclical non-convexity. A given r affects the optimal b and g (since these must adapt to the regions assigned by r). and vice-versa. We approach this issue by alternating optimisation. First, we train global and base models according to standard methods. Then, we learn a routing network r under coverage penalty. The resulting r feeds back into the loss functions of b and g, and these models get retrained. This cycle may be repeated many times.
Modularity of training. Resulting scheme allows training r with a fixed (b. g). as it helps learn a cheap routing model with pre-trained cloud and mobile models. Similarly, by dropping the optimisation over g, we can hybridise too expensive to re-train global models. Additionally, one can initially train the global and freeze it after a few cycles to save compute. Finally, we can learn each component to different degrees, c.g., we may take many more gradicnt steps on g than r or b in any training cycle.
Learning Routers via Proxy Supervision. Given a fixed pair of base and global model (b,g), the problem reduces to the following, where Cp is the coverage needed to ensure R S g. max [1 - r(X)) (b(X) = Y} + r(X) [g(x)= = YJ s.t. [[[(X)] S C.-
(5) While a naive approach is to relax and pursue ERM, we instead reformulate the problem. Observe that (5) demands that r(X)= = 0 if b(X) Y, g( (X) + Y, and that r(X)= = 1 if b(X) + yXg(X)= = Y. Further, while case b(X) = x(X) = Y is not differentiated, the coverage constraint promotes r(X)= = 0 for such points. Thus, the program can be viewed as supervised learning problem of fitting the routing oracle, i.e.
b(x;b,9) (b(x)= # f(x) y)-
4

---

Under review as a conference paper at ICLR 2022
Indeed, o is the ideal routing without the FLOP constraint. It can be evaluated training data- we produce the oracle dataset Das(b.g) ==
for any given (b, g) and training dataset D = o(='; g))}- We use this dataset as supervision for the routing model r. which allows us to utilise the entire gamut of tools of machine learning that are essential for practically learning good binary functions, thus gaining over approaches that directly try to relax the objective of
Note that the oracle does not respect the FLOP constraint. We can satisfy such a constraint by randomly assigning some points from 9 to b while incurring an error. The oracle is indifferent to such an arrangement. From a learnability perspective, we would like the points flipped to the base to promote regularity in the dataset. Although, such goal is ill-specified (and unlikely to be captured well by simple rules such as ordering points by soft loss of g). Instead, we handle this issue indirectly by imposing coverage penalty whilst training the routing model and leave it to the optimisation to discover the appropriate regularity by minimising error w.r.t. o under this penalty.
Focusing competency and loss functions. To improve accuracy whilst controlling coverage, we focus the capacity of each the models on the regions relevant to it- so, is biased towards being more accurate the region r- r ({0)), and similarly 9 on r- ({1]). Similarly, for the routing network r, it is more important to match o(x) on the regions where it is 1, since these regions are not captured accurately by the base and thus need the global capacity. We realise this inductive bias by introducing -eeeedde weights in each loss function to emphasise the appropriate regions. The Routing Loss consists of two terms, traded off by a hyperparameter A the first penalises deviation of coverage from a given target (cov), and the second to promote alignment with o and is biased by the weight W,
ooo(oo(())) "((1)). The symbol & denotes surrogate loss (such as cross entropy), while (-)+ is the ReLU function, i.c., (z)+ = max(2, 0). All sums below are over dataset D
+ 20(x) to preferentially fit o
[(a'; }²}} of size N. Empirically, we find that A = 1 produces effective results.
1 routtnng (r; 0) == Xx cov - P.NA TO-v--I-INAA (((00))
+ Σ (.()) (7)
The Base Loss and the Global Loss are each a weighted variant of the standard classification loss, which are biased by the appropriate weights to emphasise the regions assigned to either model by the routing network- (b(x)= = 2 - r(z) and W (z(x)= = 1 1+rr( r(z).
Druse (b; g) = Σ (6()))(u,b(x))) and Colobaal (b:r,g) =
Σ W (((r))((.o(t)).
2.2 EVOLJUIIONARY ARCHITECTURE SEARCH
This section describes the joint architecture search implicit in We use an evolutionary search(Elsken et al. 2019; Liu et al., 202T) due to its simplicity and effectiveness. It requires a fast way to evaluate the fitness of base-globee global pair (as, ag), i.e. the value of the program (4) for given pair of architectures. While this can be obtained by carrying out hybrid training as previously described it is impractical due the time cost. The following describes cheaper proxy for the same.
To quickly approximate the result of training over r, we use the agreement oracle routing ô. This assigns all inputs for which the base and global agree to the base, and the remainder to the global model. We choose the agreement oracle as opposed to the oracle in (6). as it is more realistic in that it does not assume knowledge of the true classification label, while still being easy to compute.
What remains is the optimisation over b and g. We adopt a different solution for these, rooted in the space of architectures itself. Recently, Cai et al. (2020) showed that possible to design spaces of architectures such that each architecture a is associated with a canonical set of parameters 6,, that are near-optimal in an accuracy objective, in the sense that slight fine-tuning these models yields good solution. This is realised via the use of a 'super-net," the components of which can be individually changed, leading to a combinatorially large set of architectures. Importantly, training super-net with such property has comparable costs to training standard network, and this cost is further amortized over a large number of resulting architectures. We use OFA space Ca et al. (2020) as architectural search space, and incorporate the oracle proxy to avoid training r, yielding the score A(ô, p,,, 8%,) for the fitness of (a6, ag).. The resulting architecture search scheme is summarised in Alg. Jin
5

---

Under review as a conference paper at ICLR 2022
3 EXPERIMENTS
In this section, we will first evaluate hybrid models with off-the-shelf base and global architectures. Next, we will perform evolutionary search to find architectures under various resource constraints and evaluate hybrid models with these newly found architectures. Finally, we infer that accuracy of the hybrid model consistently increases with increasing differences between base and global FLOPs.
For a proof-of proof-of-concept. we limited the cloud model to 600M FLOPS, which has SOTA accuracy of x 80% (Cai et al. 2020). While there other models such EfficientNet achieving higher accuracy (84.3%), these require substantially more FLOPs (37B). Although our hybrid training could leverage such models, our limited computing resources made this infeasible.
Highlights. We list a few salient observations from our empirical results.
Pareto Dominance and Latency Reduction. Hybrid models consistently outperform high-capacity models at lower Hybrid FLOPs (Fig. ( and at a latency reduction of 70% (3 in 10 examples pass to the cloud) our FLOP gains are substantial ( Fig. 2). Similarly, we improve accuracy at the same FLOP level both in terms of base FLOPs and Hybrid FLOPs. For instance, 80% is SOTA accuracy for a stand-alone 600M model. We achieve 79% accuracy in a Hybrid scheme with a base model of 143M (see Table. 1 and 78.5% with 350M hybrid FLOPs.
Rapid Customization. Proposed approach allows for optimizing accuracy level to match any intermediate FL.OP count with little training, saving computation for training intermediate models. SOTA performance on Resource Constrained Base. Hybrid scheme allows the base model to be deployable low-resource hardware. With 12M base FLOPs and only 3 in 10 examples passed to a larger model we gain about 16% improvement in accuracy (see 3
Evolutionary Search yields better Hybrid Models. Given any single model, we can obtain better hybrid model using evolutionary architecture search with similar FLOP count.
Routing oulperforms Entropy Metric. Our routing method exploits base and global models charac- teristics and dominates entropy-thresholdin used in many adaptive neural networks.
Experimental Setup. For simple exposition, we focus on the classification task the Imagenet (Russakovsky et al. 2015) dataset, consisting of 1.28M train and 50K validation images. We follow standard data augmentation (mirroring, resize and crop to shape 224 x 224) for training and single crop for testing. Similar to previous works, we report results on the validation set. We borrow the pre-trained baselines from their public implementations as described in the appendix sec. B. For evolutionary architecture search, we utilize the supemnet from the OFA search space (Cai et al. (020). We describe our hyper-parameter settings in the appendix sec. B. Depending on the computational budget, one can create hybrid models in three ways (a) 'Hybrid-(r) only training routing while using pre-trained base and global. (b) 'Hybrid(1bb'' training routing and base while using pre-trained global, and (c) 'Hybrid-(rbg)' training all three components. We trained the hybrid model to achieve a similar coverage level as the oracle. Post training, we vary the coverage level by adjusting the threshold seper-parameter in the routing to generate the performance at different hybrid FLOPs
3.1 HYBRID MODELS USING OFF-THE- THE-HHE-SHEFF CLASSIFIERS
3.1.1 No RESOURCE CONSTRAINTS
In this setting, we assume no constraints on the model deployment. We pick up an architecture family and create a hybrid model using the smallest and largest architecture. For convenience, we perform this experiment for two known families, namely MobileNet (Howaad et al., 2019) and OFA jet al. 2020). From Mobile we pick the smallest model (48M FLOPs, 67.6 accuracy) as base and largest model (215M FLOPs, 75.7% accuracy) as global to create Alybrid-Mobileeett
model. Similarly, from OFA, we pick the smallest model (67M FLOPs, 70.4% accuracy) as base and largest model (595M FLOPs, 80% accuracy) as global to create Hybrid-OFA model.
Figure (a) (a) and (b) plot the FLOPs vs topl accuracy, and compare the hybrid models with best-known baseline models in the architecture space. We show hybrid FLOPPs for the hybrid models (see Eq. 2 These experiments provide evidence for the following properties of hybrid models:
Hybrid models dominate any standalone model between base and global model. Hybrid models outperform off-the-shelf classifiers at every intermediate FLOP count. For ex., a pre-trained model in Mobile with 155M FL.OP achieves 73.3% accuracy while our hybrid model "Hybrid-(rbg)"

---

Under review as a conference paper at ICLR 2022
76-
80
1.5x reslection
reduction
Imageenet Accuracy(5)
74-
Imagenet Accuracy
78-
A Increass
3% inefease
76-
72-
74-
MobileNeetv
OFA
70
Threshaiirns Hybrid MobileNetv33rr)
Entropy Threshilding
Hybrid OFAIr)
Wyboid--Mobileeeee33rr)
Hybrid-OlMrrb)
68
#yooidd-Mabileeee333rggg
Hybrid-OFMrrbg)
70
50
100
125
150
175
200
225
200
300
400
500
600
Multiply-Add Operations (Millions)
Multiply-Add Operations (Millions)
(a) MobileNetV3
(b) Once-for-All
Figure 3: Plot for FLOPs accuracy under no resource constraints. Each intermediate point for the baseline requires expensive training and ine-tuning. addition, OFA requires an evolutionary search find the model. In contrast, the proposed scheme creates hybrid models using two extreme points and achieves better performance than the best fine-tuned models in this region.
Table 1: Results for hybrid models with base various coverage levels. OFA model achieving 80% Topl accuracy is used as global model. Base model belongs to MobileNetV3 space.
Base
Base Accuracy (%)
Coverage=80% Coverage=90% Accuracy (%)
Coverage=70% Accuracy (%)
MACs
Accuracy (%)
Base
Hybrid
Base
Hybrid
Base
Hybrid
48M
67.61
73.3
71.59
78.56
74.61
83.41
76.77
143M
73.3
79.01
75.94
83.88
77.81
88.39
79.01
215M
75.72
81.33
77.61
86.07
79.01
90.11
79.59
achieves 75.5% accuracy. Similarly, model in OFA with 230M FLOP achieves 76.4% accuracy while the hybrid model achieves 77.6% accuracy.
Hybrid achieves SOTA w.r.f a global model at zs 20% lower FLOP count. Global model in MobileNetV3 achieves 75.7% accuracy at 215M FLOPs, while hybrid model achieves same accuracy at 177M FLOPs. Similarly, global model in OFA achieves 79.9% accuracy 595M FLOPs, while hybrid model achieves same accuracy at 483M FLOPs.
Training a Hybrid model for intermediate FLOPs is inexpensive. To achieve a single model at any FLOPs, we find an architecture with the FLOP constraint and train it to achieve non-trivial performance. Hybrid model with smallest and largest model allows us to trade-off FLOPs for accuracy and save compute for training models for intermediate FLOP constraint.
Outperform the entropy thresholding baseline used in dynamic neural networks.
End-to-end training of all components lead increasing gains. Hybrid models improve in performance as additional components are trained in the alternative minimization (Algorithm i.e. Hybrid-rbg is the best performing model followed by Hybrid-rb and Hybrid-r.
3.1.2 RESOURCE CONSTRAINED BASE
In this setting, our base model operates a fixed computational budget on an edge device and one cannot deploy the global model on this device. For simplicity, we assume the latency between the base and global models be negligible. We create hybrid models using base models from the MobileNetV3 family. Since in this setting, the goal is to save compute / battery on the device and achieve near SOTA performance, we use a high performing OFA model as the global model and operate the base model at fixed coverage level. We report two metrics: (a) base accuracy achieved by the base by predicting only on the coverage portion of the routing, (b) hybrid accuracy accuracy achieved by the hybrid model, where the examples abstained by the base are sent to the global model. Table I shows the base and hybrid accuracy at three coverage levels, 90%, 80% and 70%. Hybrid models operating at fixed coverage level provide the following benefits:
Hybrid models achieve near SoTA accuracy with x 3x less FLOPs. Using base with 48M FLOPs and 67.61% accuracy, the hybrid model achieves 71.59% accuracy 90% coverage, improving to 76.77% accuracy at 70% coverage. To achieve 76.77% accuracy with a single model would require 400M FLOPs, which is too large to be deployed on an MCU. Similarly, using a base
7

---

Under review as a conference paper at ICLR 2022
Table 2: Results for the evolutionary search for hybrid architectures different FL.OP constraints
Hybrid NAS 150M Hybrid NAS
250M
Hybrid NAS 350M
Accuracy (%) MACs Accuracy (%)
MACs
Accuracy (%)
MACs
Base Model
68.65
68M
71.68
145M
74.33
225M
Global Model
75.98
263M
78.4
466M
78.75
501M
OFA Search @ Flops
73.71
150M
74.77
250M
74.93
400M
Entropy Thresholding
73.83
150M
76.05
252M
77.2
350M
Hybrid (r)
74.51
150M
76.63
252M
77.91
350M
Hybrid (rb)
74.72
150M
76.91
252M
78.07
350M
with 215M FLOPs and 75. accuracy, the hybrid model achieves 79.59% accuracy with 70%
coverage Global with = 600M FLOPs has 80% accuracy.
Abstaining base model achieves significantly better performance than the base at full coverage.
Hybrid scheme allows the system to operate without global model. In this case. the result is abstaining classifier operating on the device, i.e. it rejects few input examples and provides predictions on the rest. For ex., a base model with 48M FLOP achieves an accuracy of 83.41 41% when only covers 70% examples. Similarly, base model of 215M achieves 90. 90.11% accuracy with 70% coverage. In both cases, there is gain of at least 15 points in the accuracy.
3.2 HYBRID MODELS WITH EVOLUTIONARY ARCHITECTURE SEARCH
So far we have been generating hybrid models using off-the-shelf classifiers that not tuned to maximize hybrid performance. In this experiment, we search for base and global pairs using evolutionary search in the OFA space. We constrain the search to operate at fixed hybrid FLOPs. After finding base and global pairs from the evolutionary search, we create hybrid models with the newly found architectures. We perform this experiment for three hybrid FLOP constraints: 150M, 250M, and 350M. We draw baseline architecture samples from the OFA space using their optimized architecture search. For fair comparison, we do not fine tune models found by the architecture searches for both OFA and hybrid models. Figure Sacureepploo plots the operating curves for the hybrid models found using different FLOP constraints. Table shows the hybrid model performance at the constraint points used in the search. Evolutionary search based hybrid models provide the following benefits
Hybrid Models with evolutionary search yields higher accuracy at any target FLOP. As illustrated in Table 2 evolutionary search finds hybrid models that outperform the models found from the optimized OFA search. For ex., at target 350M FLOP, OFA finds an architecture with 74.93% accuracy while evolutionary search finds hybrid model that achieves 77.91% accuracy. Hybrid models pareto dominate single models at any target FLOP. In figure (lgure[aa. hybrid model for 150M FLOP outperforms the OFA search baseline beyond the target 150M FLOP. Similar observation can be made for 250M and 350M FLOPs.
Hybrid model achieves near SoTA accuracy with 3X less FLOPs. For ex., using base with 146M FLOPs and 71.68% accuracy, the hybrid model achieves 76.1% accuracy at 80% coverage. To achieve 76.1% % accuracy, single model requires 450M FLOPSs.
Base MACs R/RB
Cov.=90 Acc. (%)
Cov.=80
Cov.=70
78
Acc. (%)
Acc. (%)
Imagenet Tont Accurasy(5)
3.5% ncreade
Base Hybrid Base Hybrid Base Hybrid
76-
68M
74.2
71.1
79.2
73.1
84.1
74.4
reduction
0FA
68M
RB
74.4
71.2
80.3
73.4
84.6
74.7
Wybrid-OiA-15001
72-
1yybrid-OFA-1000 OFA-1500rb)
146M
R
77.1
74.2
82.1
76.1
86.5
77.1
Hybrid-OFA--500r
146M
RB
77.6
74.4
82.9
76.4
87.3
77.4
Hyybrid-OFA-250rrt)
70
Bybrid-OFF-3500
225M
R
79.7
76.1
84.7
77.4
88.9
78.2
Hyfomid-OFA-350mb)
225M
RB
79.9
76.3
84.8
77.7
89.1
78.5
100
200
300
400
500
Multiply Add Operations (Millions)
Figure 4: Evolutionary search based hybrid OFA-models with hybrid FLOP constraints: 150M, 250M, Figure plots intermediate FLOPs achievable by three different hybrid models. Table shows hybrid accuracy and base accuracy at different coverage levels.
350M.
8

---

Under review as a conference paper at ICLR 2022
Abstaining base model achieves significantly better performance than the base at full coverage. For ex., a base model with 2 FLOPs achieves an accuracy of 79.86% when it only covers 90% examples. This performance increases to 88.94% accuracy with 70% coverage.
3.3 HYBRID MODELS UNDER OVERWIEELMING RESOURCE CONSTRAINTS
In this experiment, we explore hybrid models where base is severely resource constrained. Concretely, we consider ImageNet classification on tiny MCU (STM32F746 MCU with 320KB SRAM and 1MB Flash), the same seting as MCUNets Lin et al. (2020). Using MCUNet TFLite model (12.79M FLOPs, 51.51% % accuracy) as base, we create hybrid model by adding various OFA global models. The resulting hybrid model performance is given in Table B] with the energy comparison Appendix. B.4 Below summarize the benefits of deploying hybrid models:
SOTA Accuracy and Pareto Dominance. Hybrid model with deployable base achieves near SoTA accuracy with = 3XX fewer FLOPs. In addition, hybrid models consistently dominate stand-alone designs across different target accuracy levels.
Micro-controller Implementation. Successfully deployed base and routing models on micro- controller with negligible (~~ 2%) slowdown.
Table 3: Hybrid models with MCUNet as base (12.79M FLOP, 51.51% % accuracy, deployable STM32F746
controller with 320KB SRAM& IMB Flash), operating various coverage levels.
Coverage=90
Coverage=80
Coverage=70
Coverage=60
Global
Top1 Accuracy (%)
Top1 Accuracy (%) Top1 Accuracy (%) Top1 Accuracy (%)
Base
Hybrid
Base
Hybrid
Base
Hybrid
Base
Hybrid
0FA-125M55.89
56.11
60.52
60.37 62.01
65.71
64.52
71.01
67.99
0FA-595M55.91
57.01
60.83
65.68
66.69
71.01
70.77
3.4 HYBRID MODELS UNDER LATENCY METRIC
For simplicity, we used the coverage (i.e., data kept on the base) as a proxy for the communication latency since it is the major contributor to inference cost in hybrid system. In this section, explicitly define the communication device and measure the three latency components in this hybrid system: (a) inference latency spent on-device, (b) communication latency for examples sent to cloud, (c) inference latency on the cloud. Table benchmarks the latency ( (a+b+c) ) of the hybrid approach against various baselines. shows that the hybrid model operates at nearly half the latency and power consumption compared to the global only solution and provides 8% improvements over the best available model on the device. We provide details and other configurations in the Appendix $B.6
Table 4: Latency and power consumption: Base device communicates via LoR (Ikhps transmission speed) and operates at 21 215MHz with 2.5V power supply and active-mode consumption of 100mA per second.
Base + Global
Method
Params
Top-1
MACs
Latency
Energy
MCU STM32F746 +
Global only
9.1M
79.93
595M
1200ms
300mJ
On-Device
0.6M
51.5
12.8M
197ms
49mJ
On-Device
0.74M
62.6
82M
1075ms
269mJ
GPU Tesla V100
Hybrid 70Cov
66.69
191M
557ms
139mJ
Hybrid @ 60Cov
70.77
250M
677ms
169mJ
4 CONCLUSION
We proposed a novel hybrid edge-cloud network to handle the majority of the workload of large-scale prediction on edge devices. Currently, large-scale prediction tasks are exclusively handled the cloud with high-capacity DNNs, and although recent works propose methods for compression of high-capacity models, the resulting models when required to achieve SOTA accuracy are still too large. Our proposed solution is based on leveraging a low-capacity network that can be deployed on an edge device, along with a high-capacity network deployed in the cloud. When the edge device encounters challenging inputs, these inputs are transmitted and processed on the cloud. We proposed a novel end-to-end framework for optimizing network architectures, network models, as well as the routing protocol in systematic manner. Our proposed method demonstrates substantial decrease in the number of overall floating point operations (FLOPs) on ImageNet dataset compared to a well-designed high-capactty network, while suffering no excess classification loss. Furthermore, when communication latency to the cloud is the dominant issue, we show that across different target accuracy regimes, realize 4x FLOP gains on the low-capacity model with 70% coverage.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Alex Alemi. Google AI blog: Improving Inception and Image Classification in Tensor- Flow, 2016.
URL httt:e/s/web...rrcceeee.errrreee
gi.goolleblog wwwp//////////nneoececo http:///0066/////mmmooooggggggg/ cessed on 2021-09-12.
and www.i.aaaaaa..... html Ac-
Colby Banbury, Chuteng Zhou, Igor Fedorov, Ramon Matas, Urmish Thakker, Dibakar Gope, Vijay Janapa Reddi, Matthew Mattina, and Paul Whatmough. Micronets: Neural network architectures for deploying tinyml applications commodity microcontrollers. Proceedings of Machine Learning and Systems, 3, 2021.
Babak Ehteshami Bejnordi, Tijmen Blankevoort, and Max Welling. shaping for learning conditional channel gated networks. arXiv preprint 1rXiv:1907.06627, 2019.
Tolga Bolukbasi, Joseph Wang. Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 527-536. JMLR. org, 2017.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once for all: Train one network and specialize it for efficient deployment. In International Conference Learning Representations, 2020. htt.htttss//arxivvcoo
09990 pddf
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine
Learning Research, pp. 2285-2294 Lille, France, 07-09 Jul 2015. PMLR. URL https://
proceedings www.. press wttp://///ccccccccecmmmm
Xin Dai, Xiangnan Kong, and Tian Guo. EPNet: Learning to Exit with Flexible Multi-Branch Network, pp. 235-244. Association for Computing Machinery, New York, NY, USA, 2020. ISBN 9781450368599. URL https:: org/ 10.
3411973,
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint / 2018.
Xuanyi Dong and Yi Yang. Network pruning via transformable architecture search. In Neural Information Processing Systems (NeurIPS), 2019.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 1997):19772011 2019.
Igor Fedorov, Ryan Adams, Matthew Mattina, and Paul N Whatmough. Sparse: Sparse architec- ture search for cnns on resource- constraied microcontrollers. Advances in Neural Information Processing Systems, 32, 2019.
Igor Fedorov, Marko Stamenovic, Carl Jensen, Li-Chia Yang, Ari Mandell, Yiming Gan, Matthew Mattina, and Paul N Whatmough. Tinylstms: Efficient neural speech enhancement for hearing aids. arXiv preprint arXiv: 11.5|11138,, 1188, 2020.
Aditya Gangrade, Anil Kag, and Venkatesh Saligrama. Selective classification via one-sided prediction. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th In- ternational Conference on Artificial Intelligence and Statistics, volume 130 Proceedings
of Machine Learning Research, pp. 2179-2187. PMLR, 13-15 Apr 2021. URL https: //proceedings. mlr www.rppesssae.m wexess/v130// http:////gggggaaeeaa.... html
Yonatan Geifman and Ran El- El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In International Conference on Machine Learning, pp. 2151-2159, 2019.
Amir Gholami, Kiseok Kwon, Bichen Wu, Zizheng Tai, Xiangyu Yue, Peter H. Jin, Sicheng Zhao, and Kurt Keutzer. Squeezenext: Hardware-aware neural network design. CoRR, abs/ Ra.//803..10000 2018. URL.ttte. htt.:/ttt///ariv.rr////aaa/
10

---

Under review as a conference paper at ICLR 2022
Google LLC. On-device machine learning, 2021. URL https::/web.archii/
web/ on-device-ml Accessed on 2021-09-20.
google htt.oooee.ccmmeaanntttt........a
Albert Gural and Boris Murmann. Memory-optimal direct convolutions for maximizing classification accuracy in embedded applications. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.),
Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.
PMLR, 09-15 Jun 2019. URL https:
proceedings mwr. wttppp//////////1111e.e/
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NIPS, 2015.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning. trained quantization and huffman coding. International Conference on Learning Representations (ICLR), 2016.
Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: survey, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.023 02531, 2015.
Andrew Howard, Mark Sandler, Grace Chu, Liang- Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang. Vijay Vasudevan, Quoc V. Le, and Hartwig Adam. Searching for mobilenetv3. CoRR, abs/1905.022444 2019. http:/tt://a/t
abaaaaas/b900 wttp./bb////..0///4/444 02244
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q. Weinberger. Multi-scale dense networks for resource efficient image classification, 2018.
Forrest N. landola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and Imm model size. CoRR, abs/1602.073600, 2016. URL http:t/pp//aiiii http://awxix.....aa.aaa/a//////s /602 602. 07360
Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-nnl, inference. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2704-2713, 2018.
Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, and Lingjia Tang. Neurosurgeon: Collaborative intelligence between the cloud and mobile edge. SIGPLAN Not., 5244)6622)11 apr 2017. ISSN 0362- 1340.. doi: 10. 10.1145/0033336330337 11
URL https
3037698
Hao Li, Hong Zhang, Xiaojuan Qi, Ruigang Yang, and Gao Huang. Improved techniques for training adaptive deep networks, 2019.
Min Li, Yu Li, Ye Tian, Li Jiang, and Qiang Xu. Appealnet: An efficient and highly-accurate edge/cloud collaborative architecture for dnn inference, 2021.
Lin, Wei-Ming Chen, Yujun Lin, John Cohn, Chuang Gan, and Song Han. Mcunet: Tiny deep learning on iot devices. arXiv preprint arXiv:2007.103: 10319, 2020.
Chenxi Liu, Barret Zoph, Jonathon Shiens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan L. Yuille, Jonathan Huang, and Kevin Murphy, Progressive neural architecture search. CORR, abs/1712.00559, 2017. URL /attp://arxian wwt.aawwwww....//////a.a. 00559
org
Yuqiao Liu, Yanan Sun, Bing Xue, Mengjie Zhang, Gary G Yen, and Kay Chen Tan. A survey on cvolutionary neural architecture search. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Ziyin Liu, Zhikang Wang, Paul Pu Liang, Russ R Salakhutdinov, Louis-Philiper Morency, and Masahito Ueda. Deep gamblers: Learning to abstain with portfolio theory. In Advances in Neural Information Processing Systems, pp. 00622-10632. 2019.

---

Under review as a conference paper at ICLR 2022
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural PMLR, 2017.
networks. In International Conference on Machine Learning, pp.
Feng Nan and Venkatesh Saligrama. Adaptive classification for prediction under a budget. In Advances in Neural Information Processing Systems, pp. 4727-4737, 2017.
Eunhyeok Park, Dongyoung Kim, Soobeom Kim, Yong-Deok Kim, Gunhee Kim, Sungroh Yoon, and
Sungjoo Yoo. Big/little deep neural network for ultra low power inference. In 2015 International Conference on Hardware/Soffwarr Codesign and System Synthesis (CODES+ISSS), pp. 124-132,
2015. doi: 10. 10010990ODESSSSS22153333333
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 1-252, 2015, doi: 10.1007/s| 1263-015-0816-y;
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Mingxing Tan and Quoe Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6105-61 14, Long Beach, California, USA, 09-15 Jun 2019. PMLR. http //proceedings ml press /t/ aataan99ahhtn 9tm
Surat Teerapittayanon, Bradley McDanel, and H. T. Kung. Branchynet: Fast inference via early exiting from deep neural networks, 2017.
Xin Wang, Yujia Luo, Daniel Crankshaw, Alexey Tumanov, Fisher Yu, and Joseph E. Gonzalez. Idk cascades: Fast deep learning by learning not to overthink, 2018.
Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices, 2016a.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation sys- tem: Bridging the gap between human and machine translation. arXiv preprint arXiv: 1609.0814,, 2016b.
Le Yang, Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, and Gao Huang. Resolution adaptive networks for efficient inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. CoRR, abs/1611.01577 2016. URL BRLlhttpp//axxxi http:////awwwvwooaaaaooo wttp://wwwwww.p///ll....o0 01578
12

---

Under review as a conference paper at ICLR 2022
APPENDIX
A ALGORITHMS
We summarise the methodological proposals as algorithms. The overall method to begin with training super-net in the sense of Cai et al. (2020), for which the methods of their paper can be utilised. This produces a set of architectures a with associated canonical models for each a A. The overall procedure then is summarised as Algorithm IT This uses the main procedures of architecture search (Algorithm 4 and hybrid training (Algorithm 3 as subroutines, which in turn may be executed in modular way discussed at length in the main text.
In addition, we frequently tune a given router r and base and global models to locally trade-off resource usage levels and accuracy (which saves retraining on each different value of that one may be interested in. This is realised by finding a value adjusted to the constraint, and using the routing function r(z; t) 1l (x) 2 (z) + f}. Such may be found as in Algorithm
Algorithm End-to-end Hybrid Procedure
1: Input: Training data B = [(x,, ,') Validation data V = {(~), 1
1 resource constraint
0.
2: Train supernet using the method of Cai et al. (022))
(Architecture Search)
3: <- resulting set of algorithms.
4: (as, ag) < output of Algorithm with V, 0, st.
5: Train initial models 8 E ab. go
0gg using B
(Hybrid Training)
6: br,,,)) + output of Algorithm instantiated with B, 6", ,g", and with appropriate hyperparame-
ters.
7: Return: (r, g)
Algorithm 2 Tuning Routing Model
Input: Validation data V [(x,, M target resource level p. Hybrid model (r, b, g).
2: T <- (ro(z) - (x) : x E V}.
3: c* <- min e R, + R(as) + (1 R)Rg S
4: 1" < c'th quantile of T.
5: Return:
Algorithm 3 Training Hybrid Models
1: Input: Training data B = {(0", y² ) v
2: Hyper-parameetes: A,. Epochs
3: Initialize: random , -pre-trrined b0, 9
4: for
to E 1
5: Randomly Shuffle B
6: rd arg min, Crouting (r, b--1 1 gg
7: g arg min, Lighrall (re, be-1 .9)
8: be arg min, Prane (r g)
9: Return: : (r )²)
13

---

Under review as a conference paper at ICLR 2022
Algorithm 4 Evolutionary Joint Architecture Search
1: Input: Validation data B = {(x², (((x',"')) yy') resource constraint (p, set of architectures o
2: Hyper-parameters: G, Npop, . Npar
3: Initialize: Supp {(ai o//) R(ó, 0z Oa;)) S ( by random sampling
4: for to G do
5: Opar -- par highest (oracle) accuracy configurations from Spop
6: Rehind 0
7: for n = 1 to Npup do
8:
Randomly pick (aj, a')) from Spar
9:
(ag ag f Mutate (aj, ag))
Compute the agreement oracle for oag, Oag.
11:
RØ, Oag, Oag > y then GOTO 9.
13:
Add (ag ag to Rohild
14: Spop par U Uehhid
15:
Return 2ppp
B IMPLEMENTATION DETAILS
B.
SETTINGS.
We use SGD with momentum as the default optimizer in all our experiments. We initialize our hybrid models from the corresponding pre-trained models and use learning rate of and global models. We use learning rate of 1e for learning the routing network. We decay the learning rate using a cosine learning rate scheduler. As recommended in the earlier works, we use a weight decay of le 5. We set the number of epochs to be 50, We use batch size of 256 in our experiments.
for learning base
B.2 MODEL DETAILS
Entropy Thresholding Baseline. As per recommendation in the literature (Teerapitayanoon et al.
2017 Gangrade al. (02T) we compute the entropy of the base prediction probability distribution (x). This baseline allows access to a tunable threshold f. Predictions with entropy below this threshold are kept with the base model while the predictions with entropy above this threshold are sent to the cloud model. We use similar tuning Algorithm trade-off resource usage.
Routing Model. Our routing model uses predictions from the base model and creates 2-layer neural network from these predictions. We create meta features from these predictions to reduce the complexity of the network, by (a) adding entropy as feature, (b) and adding correlations between top 10 predictions, resulting in 101 dimensional input feature vector. The feed-forward network has 256 neurons in the first and 64 neurons in the second layer. The final layer outputs a two dimensional score leading to binary prediction for the routing Note that the routing network described in this manner contributes to less than 2% compute budget of the base model and hence its compute cost is negligible in comparison the base and global models.
Nebile We have used the small and large configurations as base and global models in our experiments (see Sec. 3.I). We borrowed pre-trained models from publicly available implementation Table 5 lists the performance and compute characteristics of these borrowed models.
Once-for-All We borrowed the pre-trained OFA models from the official public repository Table
lists the accuracy, number of parameters and FLOPs for these models. We note that these models have been specialized by the authors with fine-tuning to achieve the reported performance.
14

---

Under review as a conference paper at ICLR 2022
Table 5:
MobileNetV3 Models in our setup.
Top1 Accuracy
#Params
#MACs
MobileNerV--Small
67.613
2.54M
48.3M
MobileNerV3
73.3
3.99M
143.4M
MobileNetV3
71.7
112M
MobileNetV3
70.4
91M
MobileNenV3-Large
75.721
5.48M
215.3M
Table 6: Once-for-All Pre-trained models in our setup.
Top1 Accuracy
#Params
#MACs
OFA-600 ('flops@ 5950p Mtpp 80.0 ffo...finttueeeeee @ 75')
79.9
9.1M
595M
OFA-482 (ffops@488M Mopplloooltppplp @ 119.6_finetune @ 75')
79.6
9.1M
482M
OFA-389 ('flops@ 389M topl @7.1 1it.nn__fftteeeeee (7')
79.1
8.4M
389M
OFA-230 ("LG-G8 24ms top1 @ 76.4. fiinef_intueeee 25')
76.4
5.8M
230M
OFA-151 ("LG-G8 l 16ms_top 74.7_finetune @
74.6
5.8M
151M
OFA-101 ('note8_late) lallattttmmmopppppmm @ 72.8 f_i.etnne @25')
72.8
4.6M
101M
OFA-67 ("note8 22ps.top @7..4_finttmn @25')
70.4
4.3M
67M
B.3 ONCE-FOR-ALL SEARCH EXPERIMENTS.
In our evolutionary search experiments (see Sec. 3.2 we have used the OFA search to create the baseline models that eliminate the effect of fine-tuning available in the pre-trained models from their official repository. We created the baseline by using their optimized search to find models at different FLOPs, namely {70M, 100M, 150M, 200M 250M 300M 400M, 500M). We report the performance of these models in the Table 7 Note that as per recommendation, we tune the batch norm statistics of these models to get the correct accuracy.
For our evolutionary search experiments, we used the OFA search space. In OFA codebase, there are two search spaces with MobileNetV3 backbone: (a) with width multiplier and (b) with width multipler 1.2. For our joint architecture search with target FLOPs 150M, we used the smaller backbone with width= as this space allows smaller base models in the 100M FLOPs region. While used the backbone with =idth= 1.2 for search for target flops 250M and 350M, as these hybrid FLOPs allow larger base models. OFA space allows searching over expansion factor options [3,4,6], kernel sizes [3,5,7], block depths [2,3,4], and resolutions [144, 160, 176, 192, 208, 224]. To perform a mutation, each optimization variable modified with probability 0.1, where modification entails re-sampling the variable from uniform distribution over all the options. The population size is set to 100, and the parent set size is set to 25.
Table 2 shows the characteristics of the base and global models found using this search.
Table 7:
Once-for-All models found using the optimized OFA search (used as baseline in Sec. 3.2
Top Accuracy
#MACs
OFA-500
78.71
500M
OFA-400
74.93
500M
OFA-300
74.92
300M
OFA-250
74.77
250M
OFA-200
74.42
200M
0FA-150
73.71
150M
OFA- 100
73.19
100M
OFA-70
70.64
70M
B.4 MCUNET EXPERIMENTS
We deploy both MCUNet and our base with routing model on the MCU using the TensorFlow Lite for Microcontrollers (TFLM) runtime. Due to lack of operator support for reductions and sorting in TFLM, replace the relevant operators with supported operations whose compute and memory
15

---

Under review as a conference paper at ICLR 2022
complexity upperbounds the un-supported operations. Table compares the performance energy profile of the hybrid model and the baseline when deployed on the micro-controller (STM32F746) with 320KB SRAM & 1MB Flash. It clearly shows that there is negligible cost of deploying the proposed routing scheme and only results < 2% slowdown. Table 8 shows the performance of the hybrid model against the baseline model at various hybrid flops. It can be that the hybrid model dominates the baseline model at intermediate FLOPs.
Table 8:
Comparing MCUNet models with hybrid models (hybrid accuracy and hybrid flops
shown for hybrid models).
Target MACs MCUNet Model
Hybrid Model
Top1 (%)
MACs
Topl (%)
MACS
13M
51.5
12.79M
51.5
12.79M
38M
57.0
38.3M
56.3
36.32M
68M
60.9
67.3M
62.01
67.41M
80M
62.2
81M
64.52
83M
125M
68.4
126M
71.1
127M
Table 9:
Comparing the energy profile for MCUNet and Hybrid model when deployed on
micro-controller.
Model
Latency
SRAM
Energy
MCUNet
0,25368s
156708 bytes 0.1112 joules
Hybrid-MCUNet 0.25951s
158036 bytes 0.1134 joules
B.5 COMPARISON 70% COVERAGE: HYBRID MODEL VS BASELINES
Fig collates the performance of the hybrid and baseline models from the Experiments section (see Tables and 3 70% coverage column). Baseline corresponds to the best baseline models at various and
MACs. Hybrid numbers correspond to the hybrid model where base operates at 70% coverage level. list the baseline and hybrid performance metrics in Table TO for completeness.
Table 10: Baseline and Hybrid Metrics used in the Figure Hybrid model is operating at 70% coverage and MACs shown are the Base MACs.
Model
MACs
12.8M
48M
143M
215M
595M
Baseline 51.5%
67.6%
73.3%
75.5%
79.93%
Hybrid 66.69%
76.77%
79.01%
79.59%
B.6 LATENCY EXPERIMENTS
When we use MCU as the base device, we use the LoRAWAN communication protocol that enables such low capacity device to operate a transmission rate of 1kbps. Thus, takes nearly 1200ms to transfer image of size 150KB (typical image in the Imagenet dataset). Similarly, for base devices capable of operating with 3G, LTE Wi-Fi communication devices, we borrow the transmission numbers from Kang et al. 2017). Specifically, to transfer image with 152KB size, (a) 3G network takes 870ms, (b) LTE takes 180ms, and (c) WiFi takes 95ms. Note that for a micro-controller even a 3G network would not be available, instead much slower communication device is used. We use the on-device and on-cloud inference latency from the MCUNet and OFA repositories.
For any method, compute the latency the inference time taken for example, i.e., inference time on device + communication time + inference time on cloud. To compute energy usage, we use the active-mode operating characteristics of the base device and voltage supply compute the power
16

---

Under review as a conference paper at ICLR 2022
and multiply it by the amount of time the device spends doing these operations. Table I1 the inference latency ( inference cost on device + communication cost + inference cost on the cloud ) of the hybrid approach against various baselines.
benchmarks
Table 11: Deploying hybrid model vs standalone model on device. Latency comparison.
Base + Global
Method
Params
Top-1
MACs
Latency
Energy
MCU
Global-only
9.1M
79.93
595M
1200ms
300mJ
STM32F746 + LoRAWANN
On-Device
0.6M
51.5
12.8M
197ms
49mJ
On-Device
0.74M
62.6
82M
1075ms
269mJ
GPU Tesla V100
Hybrid 70Cov
66.69
191M
557ms
139mJ
@ @ 60Cov
70.77
250M
677ms
169mJ
Mobile
Global-only
9.1M
79.93
595M
205ms
Samsung Note8
On-Device Mlybrid@70ce @
5.3M
75.7
215M
65ms
+LTE
79.59
393M
119ms
GPU Tesla V100
B.7
DIFFERENCE BETWEEN APPEALNET AND OUR HYBRID DESIGN.
Below we highlight main difference between AppealNet (Li et al. (021) and our proposal.
AppealNet formulation does not explicitly model any coverage constraint that enables the base model to operate at tunable coverage level. In contrast, we explicitly model coverage penalty.
Jointly learning the routing without any supervision is a hard problem. Instead, we relax this formulation by introducing the routing oracle that specializes in routing network for given base and global pair. With this oracle, the task of learning routing reduces to a binary classification problem with the routing labels obtained from the oracle. This also decouples the routing task from the base and global entanglement.
In addition, we propose neural architecture search that finds a pair of base and global architectures that optimise the hybrid accuracy at any given combined resource usage.
Empirically, AppealNet does not have any evaluations for the Imagenet scale dataset. The closest comparison we can find with the Tiny-Imagenet dataset (one-tenth of the size of the Imagenet). While we cannot compare the two directly, since we solve a much harder 70% coverage
problem than Imagenet, we can make the following observations. level, for AppealNet, the minimum performance difference between the hybrid model and the global model is 1.2% (see AppealNet, Fig. 5(d)), while our closest to the global in case of the MobileNet baseline is 0.3% (see our paper Table 1, row 3). Note that NppealNet performance will go down on Imagenet in comparison to Iny-Immgenet due to the hardness of the problem.
17


Reviews:
[{"review_id": "g8_dK8QVb9M", "reviewer": "Reviewer_p5u2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe problem they consider is highly significant for practical application of DNNs on edge devices.\n\nWeaknesses:\n\nFrom the introduction: “The fundamental drawback of cloud ML, however, is the increased latency and energy consumption arising from communication, which can be prohibitive for many applications”, “we propose a best-of-both hybrid solution, which allows for deploying cloud-based AI tasks on edge devices like MCUs, while lowering the average total latency (the sum of communication and computational latency)”, but it is not clear if the proposed method reduces energy consumption. It would be nice to give some comparison of the energy cost of classifying an example on device vs. the energy cost of transmitting a sample to a cloud service and then retrieving the result.\n\n\nFrom the introduction: “the best available model which can be deployed on an STM32H743 MCU achieves 62.2% accuracy with 12.8M FLOPs (Lin et al., 2020)”, but Table 4 in (Lin et al., 2020) states that MCUNets achieve 70.7% accuracy on the exact same device. Even their baselines achieve 68% accuracy which is more than 62.2%.\n\nFrom the introduction: “The base model is compact and designed for devices with low-resource hardware constraints like latency and memory usage”. Latency is not a device constraint, it is an application constraint. I suggest using clock frequency instead of latency.\n\nIt is not clear how the Hybrid model was trained and what the baseline is in Fig2. There should be a clear explanation for this figure in the main text..\n\nIt is not clear to me what ‘base limited setting’ and ‘global limited setting’ mean in the last sentence of the introduction. A clear explanation is needed.\n\nA parameter “t” is introduced which “allows a routing model to trade-off accuracy and resource usage in order to avoid separately training for each desired level”. How much does it degrade the performance compared with end-to-end training of components for each desired level? An ablation study is needed.\n\nIn Section 2, “Architectures and Costs”: “we investigate communication limited settings in §3.3.” Where is this investigated? I cannot find the associated experiment.\n\nThe experiments section starts with analyzing results of the experiments and then goes into the description of the experiments, which is a poor writing choice.\n\n“Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge” tries to tackle the same problem by dividing the inference dynamically among the edge and the cloud. How do you compare your method with it?\n\n“AppealNet: An Efficient and Highly-Accurate Edge/Cloud Collaborative Architecture for DNN Inference” Tries to tackle the same problem with a very similar solution. How is your method different from theirs?\n\nComments:\n\nI think it makes the last sentence of the abstract more clear if “substantially” comes after the word “accuracy” instead of where it is right now.\n\nI think it is good to give some numbers for on-device computation latency, cloud computation latency, and also communication latency so that the reader develops a sense of them.\n\nFig1 should change to Fig2 in Empirical validation (last subsection of intro).\n\nI’m curious about the task of learning a router model. Is it a well-defined task? It is interesting to show some validation results in that specific task to see if the router model learns something generalizable and reasonable.\n\nIn the “overall formulation”, the constraint is on R(r, b, g), which is the overall cost per inference of models deployed on both the edge device and the cloud. But based on the overall development, the edge device is the only compute limited part and the cloud is an over-provisioned server. I think having a constraint on R(r, b, g) instead of R_r + R(\\alpha_b) creates some confusion.\n", "summary_of_the_paper": "This paper introduces a hybrid Cloud-Edge design for Deep Neural Network (DNN) inference in which only a small portion of samples (more complicated ones) are sent to cloud for processing. Inference for most of the samples happens on the edge devices itself which reduces the task’s latency. \nThe paper provides an end-to-end training methodology for system’s components, a Neural Architecture Search (NAS) approach for the design of models deployed on the edge and the cloud. It also does an empirical evaluation of its design.", "summary_of_the_review": "I think the authors are thinking about a very interesting problem, but the work is at an early stage and not ready for an ICLR publication. The main reason is the lack of comparison with similar work. For more details, see the cons above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9eK1cRPZtt", "reviewer": "Reviewer_EcwK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The idea of hybird framework to process queries differently based on their difficulty is not a new idea. The basic framework proposed in the submission (big + small model for hard/easy queries and router model for determination) is straight forward and not a very novel idea. Even though the end-to-end training diagram is easy to come up with.\n2. What contributes to this framework should be: 1) How to train the router, 2) How to determine the big/small model architectures. Authors address the first question in Section 2.1 and the second in Section  2.2. But in my opinion, explanation in Section 2.1 is a little bit sophisticated: the algorithm should be straight forward but the writting seems to complicate it. I am still confused about the details of how to train the router. Could author formulate the whole process in an Algorithm format?\n3. For the determination of model architecture, author proposed to use OFA, which is a seperated work.  But author emphasize in techinical contributions that \"propose a NAS method ...\". I don't think using OFA can be described as \"propose\" in the writting. Instead, NAS and the proposed hybird framework is decoupled: NAS does not take advantage of any properties in the hybird framework, vice versa.\n4. I may miss some important points in the submission. Please remind me if necessary.\n\nQuestions:\n1. As author mentions in the related work that this work is similar to dynamic network for they share a similar idea that \"different queries should be processed by different network\". Though these two kinds of methods differ a lot, especially the proposed method can be deployed much easier, I am still interested in the comparison experiments on dynamic networks. Noted that this is not compulsory.\n2. Since the determination of models is handled by another work (OFA), can I regard the submission as the hybird framework as big/small model (given) + router model training. If so, did author try another router model training methods? Different kinds of router models (if applicable) should be compared.", "summary_of_the_paper": "This paper proposed a hybird framework to process inference efficiently: the framework contains a global network to deal with hard query, a base model for easy query and a router protocol to deliver queries. It designed a proxy supervision algorithm to train the router and proposed to use Neural Architecture Search (NAS) to search for global/base models.", "summary_of_the_review": "The overall contribution is not significant as the hybird framework is straight forward and the model determination is decoupled and directly from published work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8aQR8MDpj7m", "reviewer": "Reviewer_37D1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n**Strengths**\n1. It is interesting to have models with different MACs on cloud and edge to improve the inference accuracy with minimal latency increase.\n2. The idea of introducing a routing model to route more challenging inputs to the cloud is quite novel.\n3. The paper proposes to provide a better tradeoff between the latency and accuracy, by leaving the more challenging inputs to the clouds with higher latency. \n\n**Weakness**\n1. The goal is to improve the inference accuracy by predicting the more challenging inputs on the high latency cloud. It is weird to only focus on the MACs of the models while evaluating the system. How the system affects the latency of edge inference should be shown in the paper.\n2. The router model seems only to be updated with soft constraints, which may make the whole system unreliable in practice, since it does not guarantee the average latency (how many inputs are sent to the cloud). \n3. Table 7 is the only experiment that indicates the latency of the hybrid inference system proposed, but no detailed experiment setup is given, including the network bandwidth and the computation resources on the edge and cloud.\n4. In the technical contribution and experiment section, the author mentioned the 70% communication latency reduction (3 out of 10 examples to the cloud), but it is <20% in the abstraction. \n5. It would be better to have a detailed comparison of the edge, cloud, and hybrid inference in terms of latency taking the computation latency of the edge, cloud, and router model into consideration.\n\n\n**Minor Issues**\n1. The MACs and FLOPs are different terms. FLOPs are typically 2x MACs. It would be better to use the same term in the paper.\n2. There are several missing related works that combine edge and cloud inference together.\n   [1] Huang, Yangsibo, et al. \"Instahide: Instance-hiding schemes for private distributed learning.\" ICML, 2020.\n   [2] Liu, Zhijian, et al. \"DataMix: Efficient Privacy-Preserving Edge-Cloud Inference.\" ECCV, 2020.\n\n", "summary_of_the_paper": "The paper proposes to combine the inference on edge and cloud together, taking advantage of the communication-free inference on edge and the high-accuracy model on cloud. It utilizes the OFA to obtain models with different resource requirement at low cost. It achieves higher accuracy on the ImageNet dataset compared to the edge-only inference.", "summary_of_the_review": "The paper proposes an interesting idea to combine the edge and cloud inference together, but as mentioned above, it is not practical to use MACs as a metric when the main concern of the paper is to maintain the low latency of edge inference while improving the accuracy. The experiments should be redesigned to make the proposed method more convincing.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7AhA0Dzdyi", "reviewer": "Reviewer_Jnva", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As edge devices are severely resource constrained, models (with low capacity) deployed on them don't achieve the same level of accuracy on many prediction tasks as their high capacity cloud counterparts. This work attempts to improve the accuracy of prediction tasks on edge devices by employing a hybrid approach where a low capacity model is deployed on the edge device and a counterpart high capacity model is deployed on the cloud. A query routing model running on the edge device decides which query needs to be handled by the cloud model and thereby achieving a trade-off between classification accuracy and cloud computation/communication costs. All the three models are learnt end-to-end from training data for the desired trade-off. \n\n\nPositives:\nA principled solution to the problem by modelling it as a coupled maximization problem. This involves  a) solving the architecture search problem using evolutionary search and proxies for accuracy of the architecture and b) learning the hybrid model using alternating optimisation.\nPrincipled solution to the routing problem through supervised learning of the routing oracle.\nEmpirical results that demonstrate that the proposed approach can give competitive results compared to cloud models while reducing the computation required substantially. \n\nNegatives:\nOnly one dataset is used in the empirical study.\nFrom the graphs, it appears that rbg and rg are very marginally better than r. This means that end-to-end training of all the three models is not really helping much. \n\nThe effectiveness of the proposed approach is not only contingent on the predictive accuracy of the routing model (it needs to do a good job of identifying challenging queries (on which the low capacity model is likely to fail) but also on most queries being relatively less challenging (so that the low capacity model can make accurate predictions on them). This implies that predictive tasks that are inherently challenging for low capacity models are unlikely to benefit from the hybrid approach as most queries would need to be routed to the cloud model thereby making it cloud-heavy. \n", "summary_of_the_paper": "As edge devices are severely resource constrained, models (with low capacity) deployed on them don't achieve the same level of accuracy on many prediction tasks as their high capacity cloud counterparts. This work attempts to improve the accuracy of prediction tasks on edge devices by employing a hybrid approach where a low capacity model is deployed on the edge device and a counterpart high capacity model is deployed on the cloud. A query routing model running on the edge device decides which query needs to be handled by the cloud model and thereby achieving a trade-off between classification accuracy and cloud computation/communication costs. All the three models are learnt end-to-end from training data for the desired trade-off. \n", "summary_of_the_review": "The work is interesting as it addresses an important practical problem and proposes a principled solution to it. Experimental study can be made stronger by considering additional tasks and datasets. Also, practical usefulness of the proposed approach needs to be carefully reasoned as it appears to be critically dependent on most queries being easy for the edge model whereas in practice this might not be the case. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to improve performance on edge devices by utilizing a large-capacity network in the cloud. To this end, the authors suggest using the routing network that decides whether to use the base model (on the edge device) or the global model (on the cloud). They also propose an overall training scheme for learning not only model parameters, but also network architectures. After the discussion period, 3 reviewers are on the negative side, and 1 reviewer is positive. AC thinks that the authors’ response was not enough to convince the negative reviewers. In particular, AC agrees with the negative comments of reviewers on limited novelty, unclear motivation for the proposed method, and unclear presentations. Overall, AC recommends rejection.
Decision: Reject