Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes using a shallow network to route a subset of samples from the edge device to a remote server. The router, edge, and server models are trained jointly but iteratively. 
strength_and_weaknesses: Strengths:
- The paper is well-organized.
- Efficient inference on edge devices is a timely topic. 
- The idea of not sending the data to the server if it's not going to bring any improvement sounds interesting. 

Weaknesses:
- Using average latency as an inference system metric here seems a bit unnatural to me. Controlling for the "average" latency is equivalent to maintaining the average sample throughput of the system. In the case of controlling for throughput, the batching mechanisms become quite effective: collect a batch of samples at the edge for T seconds, compress them as much as possible (even jointly if they have temporal correlation), and upload them to the server to be efficiently batch-processed by the server-class GPU. However, given the evaluations, I think the paper is not focusing on this regime of designs. On the other hand, in latency-sensitive applications, we are actually concerned about the "tail" latencies instead of the average. In that case, the proposed approach cannot provide fine-grained control over the metric of interest (tail latency): you are either going over the network with a 2sec delay or processing locally. For example, if we need a sub-second latency guarantee, the system cannot go over the network at all, and if we go for a 2.5sec latency guarantee, the system will process everything on the remote server. 
- How does this work relate to the Mixture of Experts literature? Can we think of r as the gating network and b and g as two experts with different sizes? How does your performance compare against training such an MoE but only running the expert that is hard-selected by the gating network?
- "We are the first to provably reduce router learning to binary classification and exploit it for end-to-end training based on novel proxy supervision of routing models." This contribution might be a little bit overstated. For example, have you seen the routing section in [1]?

[1] Riquelme, C., Puigcerver, J., Mustafa, B., Neumann, M., Jenatton, R., Susano Pinto, A., Keysers, D. and Houlsby, N., 2021. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34, pp.8583-8595.
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly. The contributions seem incremental from an ML perspective. From a Systems perspective, the optimization goal seems a bit unrealistic, as explained above. 
summary_of_the_review: Given that the evaluations only support latency-sensitive applications, but the optimization goal is not aligned with latency-sensitive systems, I think the paper can be more interesting for the systems community if it addresses either of the issues.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a novel hybrid design where an edge-device selectively queries the cloud only on those hard instances that the cloud can classify correctly to optimize accuracy under latency and edge-device constraints. An end-to-end method to train neural architectures, base predictors, and routing models is also proposed which is enabled by a novel proxy supervision for training routing models.
strength_and_weaknesses: Strength:

1. The paper provide a clear illustration of their methods and especially the comparison with the existing approaches.
2. The experiments are setup solidly and the covered settings and the metrics are comprehensive 

Weakness: 
1. According to Figure 2, improvements over dynamic methods may not be good enough to justify the significance
2. Limited novelty: alternating optimization to solve the coupled optimized parameters seem to be mentioned before by some works, such as [1] and [2]. And anyway it seems to be an intuitive approach.

[1] Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators

[2] A3C-S: Automated Agent Accelerator Co-Search towards Efficient Deep Reinforcement Learning
clarity,_quality,_novelty_and_reproducibility: The article is clear and looks reproducible. Novelty might be limited.
summary_of_the_review: A clear article with simple yet relatively effective methods to solve the efficiency and accuracy dilemma of edge and cloud DNN execution. Would be great if the authors could provide more novelty and improvements justification.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an advanced two-stage inference strategy to reduce resource usage and communication cost. The key idea is that 1) use different edge and cloud models, 2) jointly train the router module, and 3) control the cost during training, dependent on the target environment. As a result, the proposed method achieves near-upper-bound performance in diverse resource constraints.
strength_and_weaknesses: Strengths:
- The proposed framework works end-to-end, including the routing module. Especially, the routing module is trained via proxy supervision (using oracle), which is a clever way to incorporate all modules in the training.
- Experiments are carefully designed and performed. More importantly, actual MCU systems are used for the evaluation. “Hybrid accuracy” seems to be a good metric for these two-stage inference systems.
- Previous approaches (split-computation, dynamic network, early exit, etc.) are thoroughly studied and compared. 

Weaknesses:
- Minor: what if the global model fails to predict correctly? It seems that the overall process assumes the global model is always better than the local model.
- Just a question; doesn’t the communication cost dependent on the model size, resolutions, etc.? It seems that such cost is somewhat fixed through the paper.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and very detailed. The problem statement is solid, and the method is unique. There may be some missing details, but I hope the code will be released soon.
summary_of_the_review: Overall, the paper tries to solve an important and practical issue. The proposed method is novel, and the experimental results support the claim sufficiently. I believe this paper is helpful for both research and production areas.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a supervised training framework to improve the classification model inference in edge-cloud cooperating scenarios. The framework utilizes pre-trained models to achieve labelization for the router model under the criteria that the samples that should be transferred to the cloud are cases in which local models fail. Moreover, the authors also utilize the trained router model to re-balancing the importance of samples for classification models on cloud and device. Optimizing the router model and classification are conducted jointly to improve each system component. Extensive experimental results on benchmarks demonstrate the outperformance of SOTA.


strength_and_weaknesses: Strength:
1. The proposed criteria for label construction for router training is reasonable and efficient. The criteria exclude corner cases in which cloud and device models fail.
2. The joint training of models on cloud and local devices is impactful. Besides training a router to deliver a sample, the authors find that training each model (on a cloud or local device) can also benefit from such a router. This resolution leads a jointly trained base model to outperform an individually trained one experimentally, even with some abstaining.
3. The paper is well-organized. The authors highlight the proposed scheme and provide several model structure optimization solutions.
4. The experiments in this work are sufficient. 

Weakness:
1. The metric used for the "routing model" is unavailable for regression tasks. The metric includes a strict equivalence evaluation between labels and predictions to identify the target execution platforms. However, the equivalence is unavailable in regression.
2. The upper bound accuracy calculation is unreasonable. First, the proportion $\frac{1 − c}{1 − a_b}$ in the formulation of upper bound accuracy is weird. Due to loss function in Eq (7), if optimality holds, $P(r(x) = 0)$ should be equal to $c$. Due to Eqn (1) and related statements, the upper bound should be $c a_b + (1-c)a_g$ since execution on device and cloud are complementary. Furthermore, the authors assume that accuracies on cloud and local devices are unchanged. However, this is unsustainable since distributions change in the proposed hybrid training framework. For example, the loss functions for device and cloud under Eqn (7) erase some samples for both.
3. Some conclusions in the method section lack theoretical or experimental demonstrations. For example, there is no convergence demonstration of proposed algorithm 1. In the loss functions for cloud and device, several experiments to demonstrate the effectiveness of these "hard masks" will make it more reasonable.


clarity,_quality,_novelty_and_reproducibility: 1. The main paper is incomplete. The appendix section is lost in the main paper (in supplementary material).
2. Some of the statements lack clarity and coherence:
  2.1 In the introduction section, the authors fail to demonstrate the potential advantages of joint optimizations on the server, edge devices, and router. However, the "split-computation methods" focus on optimizing accuracy on a single model and have no consideration of communication efficiency. Moreover, there is no argument for why a "post-hoc" solution for routing a base model or edge model in a "dynamic neural network" does not work.
  2.2 In the methods section, the authors present a simple supervised solution for the router model in a complicated manner. Most mathematical formulas in this paper are just symbols. The demonstration and corresponding reformulation are still heuristic and imprecise. 
For example, the equivalence between Eqn (4) and Eqn (5) can only build by illustration since in which most of them are not explicit formulations. Moreover, the so-called "coverage constraint promotes $r(X)=0$" above Eqn (6) makes sense for efficiency improvement that reduces communication to the cloud. However, it is unreasonable from the optimization perspective since constraints tell only feasibility.
  2.3 The statement around Eqn (7) is confusing. The "cov" term even seems unrelated. The "converge" is an upper bound of how many samples need to be delivered to the cloud to achieve a communication efficiency requirement. The authors should explicitly state such a relationship in words.
  2.4. The statements on page 6 before the "Focusing competency and loss functions" paragraph are not convincing. A few examples for illustration will be better.


summary_of_the_review: This paper applies a data-based mixture of expert method on edge-cloud computing scenarios. The main contribution is that the mixture is conducted by data, simultaneously solving the model selection and improvement. Although some details require a more delicate design and the current approach is limited to the classification scenario, the proposed alternative training framework may facilitate delicate and in-depth investigations in the future.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: To improve edge efficiency and privacy the authors propose a hybrid inference approach wherein the inference is served either from the device or from the cloud depending on how confident the predictions are. Although the idea is quite natural and intuitive, the authors proposed a formal framework and develop sound global objectives and algorithms to make this a reality. Experiments are convincing and the proposal solves a very important practical problem. 

 
justification_for_why_not_higher_score: Reviewers liked the idea but were not very excited. The method proposed is sound but a very intuitive direction pursued by many prior works. 
justification_for_why_not_lower_score: The papers contribution is worthy of publication.
Decision: Accept: poster