Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: As edge devices are severely resource constrained, models (with low capacity) deployed on them don't achieve the same level of accuracy on many prediction tasks as their high capacity cloud counterparts. This work attempts to improve the accuracy of prediction tasks on edge devices by employing a hybrid approach where a low capacity model is deployed on the edge device and a counterpart high capacity model is deployed on the cloud. A query routing model running on the edge device decides which query needs to be handled by the cloud model and thereby achieving a trade-off between classification accuracy and cloud computation/communication costs. All the three models are learnt end-to-end from training data for the desired trade-off. 

main_review: As edge devices are severely resource constrained, models (with low capacity) deployed on them don't achieve the same level of accuracy on many prediction tasks as their high capacity cloud counterparts. This work attempts to improve the accuracy of prediction tasks on edge devices by employing a hybrid approach where a low capacity model is deployed on the edge device and a counterpart high capacity model is deployed on the cloud. A query routing model running on the edge device decides which query needs to be handled by the cloud model and thereby achieving a trade-off between classification accuracy and cloud computation/communication costs. All the three models are learnt end-to-end from training data for the desired trade-off. 


Positives:
A principled solution to the problem by modelling it as a coupled maximization problem. This involves  a) solving the architecture search problem using evolutionary search and proxies for accuracy of the architecture and b) learning the hybrid model using alternating optimisation.
Principled solution to the routing problem through supervised learning of the routing oracle.
Empirical results that demonstrate that the proposed approach can give competitive results compared to cloud models while reducing the computation required substantially. 

Negatives:
Only one dataset is used in the empirical study.
From the graphs, it appears that rbg and rg are very marginally better than r. This means that end-to-end training of all the three models is not really helping much. 

The effectiveness of the proposed approach is not only contingent on the predictive accuracy of the routing model (it needs to do a good job of identifying challenging queries (on which the low capacity model is likely to fail) but also on most queries being relatively less challenging (so that the low capacity model can make accurate predictions on them). This implies that predictive tasks that are inherently challenging for low capacity models are unlikely to benefit from the hybrid approach as most queries would need to be routed to the cloud model thereby making it cloud-heavy. 

summary_of_the_review: The work is interesting as it addresses an important practical problem and proposes a principled solution to it. Experimental study can be made stronger by considering additional tasks and datasets. Also, practical usefulness of the proposed approach needs to be carefully reasoned as it appears to be critically dependent on most queries being easy for the edge model whereas in practice this might not be the case.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a hybrid Cloud-Edge design for Deep Neural Network (DNN) inference in which only a small portion of samples (more complicated ones) are sent to cloud for processing. Inference for most of the samples happens on the edge devices itself which reduces the task’s latency. 
The paper provides an end-to-end training methodology for system’s components, a Neural Architecture Search (NAS) approach for the design of models deployed on the edge and the cloud. It also does an empirical evaluation of its design.
main_review: Strengths:

The problem they consider is highly significant for practical application of DNNs on edge devices.

Weaknesses:

From the introduction: “The fundamental drawback of cloud ML, however, is the increased latency and energy consumption arising from communication, which can be prohibitive for many applications”, “we propose a best-of-both hybrid solution, which allows for deploying cloud-based AI tasks on edge devices like MCUs, while lowering the average total latency (the sum of communication and computational latency)”, but it is not clear if the proposed method reduces energy consumption. It would be nice to give some comparison of the energy cost of classifying an example on device vs. the energy cost of transmitting a sample to a cloud service and then retrieving the result.


From the introduction: “the best available model which can be deployed on an STM32H743 MCU achieves 62.2% accuracy with 12.8M FLOPs (Lin et al., 2020)”, but Table 4 in (Lin et al., 2020) states that MCUNets achieve 70.7% accuracy on the exact same device. Even their baselines achieve 68% accuracy which is more than 62.2%.

From the introduction: “The base model is compact and designed for devices with low-resource hardware constraints like latency and memory usage”. Latency is not a device constraint, it is an application constraint. I suggest using clock frequency instead of latency.

It is not clear how the Hybrid model was trained and what the baseline is in Fig2. There should be a clear explanation for this figure in the main text..

It is not clear to me what ‘base limited setting’ and ‘global limited setting’ mean in the last sentence of the introduction. A clear explanation is needed.

A parameter “t” is introduced which “allows a routing model to trade-off accuracy and resource usage in order to avoid separately training for each desired level”. How much does it degrade the performance compared with end-to-end training of components for each desired level? An ablation study is needed.

In Section 2, “Architectures and Costs”: “we investigate communication limited settings in §3.3.” Where is this investigated? I cannot find the associated experiment.

The experiments section starts with analyzing results of the experiments and then goes into the description of the experiments, which is a poor writing choice.

“Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge” tries to tackle the same problem by dividing the inference dynamically among the edge and the cloud. How do you compare your method with it?

“AppealNet: An Efficient and Highly-Accurate Edge/Cloud Collaborative Architecture for DNN Inference” Tries to tackle the same problem with a very similar solution. How is your method different from theirs?

Comments:

I think it makes the last sentence of the abstract more clear if “substantially” comes after the word “accuracy” instead of where it is right now.

I think it is good to give some numbers for on-device computation latency, cloud computation latency, and also communication latency so that the reader develops a sense of them.

Fig1 should change to Fig2 in Empirical validation (last subsection of intro).

I’m curious about the task of learning a router model. Is it a well-defined task? It is interesting to show some validation results in that specific task to see if the router model learns something generalizable and reasonable.

In the “overall formulation”, the constraint is on R(r, b, g), which is the overall cost per inference of models deployed on both the edge device and the cloud. But based on the overall development, the edge device is the only compute limited part and the cloud is an over-provisioned server. I think having a constraint on R(r, b, g) instead of R_r + R(\alpha_b) creates some confusion.

summary_of_the_review: I think the authors are thinking about a very interesting problem, but the work is at an early stage and not ready for an ICLR publication. The main reason is the lack of comparison with similar work. For more details, see the cons above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to combine the inference on edge and cloud together, taking advantage of the communication-free inference on edge and the high-accuracy model on cloud. It utilizes the OFA to obtain models with different resource requirement at low cost. It achieves higher accuracy on the ImageNet dataset compared to the edge-only inference.
main_review: 
**Strengths**
1. It is interesting to have models with different MACs on cloud and edge to improve the inference accuracy with minimal latency increase.
2. The idea of introducing a routing model to route more challenging inputs to the cloud is quite novel.
3. The paper proposes to provide a better tradeoff between the latency and accuracy, by leaving the more challenging inputs to the clouds with higher latency. 

**Weakness**
1. The goal is to improve the inference accuracy by predicting the more challenging inputs on the high latency cloud. It is weird to only focus on the MACs of the models while evaluating the system. How the system affects the latency of edge inference should be shown in the paper.
2. The router model seems only to be updated with soft constraints, which may make the whole system unreliable in practice, since it does not guarantee the average latency (how many inputs are sent to the cloud). 
3. Table 7 is the only experiment that indicates the latency of the hybrid inference system proposed, but no detailed experiment setup is given, including the network bandwidth and the computation resources on the edge and cloud.
4. In the technical contribution and experiment section, the author mentioned the 70% communication latency reduction (3 out of 10 examples to the cloud), but it is <20% in the abstraction. 
5. It would be better to have a detailed comparison of the edge, cloud, and hybrid inference in terms of latency taking the computation latency of the edge, cloud, and router model into consideration.


**Minor Issues**
1. The MACs and FLOPs are different terms. FLOPs are typically 2x MACs. It would be better to use the same term in the paper.
2. There are several missing related works that combine edge and cloud inference together.
   [1] Huang, Yangsibo, et al. "Instahide: Instance-hiding schemes for private distributed learning." ICML, 2020.
   [2] Liu, Zhijian, et al. "DataMix: Efficient Privacy-Preserving Edge-Cloud Inference." ECCV, 2020.


summary_of_the_review: The paper proposes an interesting idea to combine the edge and cloud inference together, but as mentioned above, it is not practical to use MACs as a metric when the main concern of the paper is to maintain the low latency of edge inference while improving the accuracy. The experiments should be redesigned to make the proposed method more convincing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a hybird framework to process inference efficiently: the framework contains a global network to deal with hard query, a base model for easy query and a router protocol to deliver queries. It designed a proxy supervision algorithm to train the router and proposed to use Neural Architecture Search (NAS) to search for global/base models.
main_review: 1. The idea of hybird framework to process queries differently based on their difficulty is not a new idea. The basic framework proposed in the submission (big + small model for hard/easy queries and router model for determination) is straight forward and not a very novel idea. Even though the end-to-end training diagram is easy to come up with.
2. What contributes to this framework should be: 1) How to train the router, 2) How to determine the big/small model architectures. Authors address the first question in Section 2.1 and the second in Section  2.2. But in my opinion, explanation in Section 2.1 is a little bit sophisticated: the algorithm should be straight forward but the writting seems to complicate it. I am still confused about the details of how to train the router. Could author formulate the whole process in an Algorithm format?
3. For the determination of model architecture, author proposed to use OFA, which is a seperated work.  But author emphasize in techinical contributions that "propose a NAS method ...". I don't think using OFA can be described as "propose" in the writting. Instead, NAS and the proposed hybird framework is decoupled: NAS does not take advantage of any properties in the hybird framework, vice versa.
4. I may miss some important points in the submission. Please remind me if necessary.

Questions:
1. As author mentions in the related work that this work is similar to dynamic network for they share a similar idea that "different queries should be processed by different network". Though these two kinds of methods differ a lot, especially the proposed method can be deployed much easier, I am still interested in the comparison experiments on dynamic networks. Noted that this is not compulsory.
2. Since the determination of models is handled by another work (OFA), can I regard the submission as the hybird framework as big/small model (given) + router model training. If so, did author try another router model training methods? Different kinds of router models (if applicable) should be compared.
summary_of_the_review: The overall contribution is not significant as the hybird framework is straight forward and the model determination is decoupled and directly from published work.
review_title: null

Meta-review: This paper aims to improve performance on edge devices by utilizing a large-capacity network in the cloud. To this end, the authors suggest using the routing network that decides whether to use the base model (on the edge device) or the global model (on the cloud). They also propose an overall training scheme for learning not only model parameters, but also network architectures. After the discussion period, 3 reviewers are on the negative side, and 1 reviewer is positive. AC thinks that the authors’ response was not enough to convince the negative reviewers. In particular, AC agrees with the negative comments of reviewers on limited novelty, unclear motivation for the proposed method, and unclear presentations. Overall, AC recommends rejection.
Decision: Reject