Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the author considers the model collaboration problem, specifically ensemble learning. The author identifies two issues of ensemble learning: significant runtime cost and high memory access cost. To alleviate these issues, the authors propose a model collaboration framework named Collaboration of Experts (CoE) along with a training algorithm designed for the framework. The training algorithm contains three components: weight generation module (WGM), label generation module (LGM), and selection reweighting module (SRM). Experiments are conducted on ImageNet to demonstrate the effectiveness of the proposed method.
main_review: Strength:

1.	The idea of model collaboration is interesting and new.

2.	The model has achieved 80% accuracy within 100M FLOPS, achieving the SoTA performance on ImageNet.

Weakness:

1.	About related work. 
The paper mentions that the delegator needs to select an expert from the candidate experts and compares to other model selection methods. However, the related work of model selection [A, B] is lacking in the paper. The authors are strongly suggested to enrich the related work part and compare with them.

[A] Ranking Neural Checkpoints. CVPR2021.

[B] LogME: Practical Assessment of Pre-trained Models for Transfer Learning. ICML2021.

2.	About experiments.
+ As the proposed method contains several networks (including delegator and 4/16 experts), which probably introduces larger memory cost. Can the authors also elaborate on the additional memory cost and compare it with other baseline models in the experimental part.
+ In Sec. 4.3.1, while the paper mainly targets the model ensemble problem, the authors only compare the method with one simple model ensemble method, which is not very convincing. Therefore, the authors are suggested to conduct a more extensive comparison with other model ensemble methods.


======================

Post-rebuttal:

I acknowledge that it for the first introduced an instance-wise model selection framework, which is novel. Most of my concerns about the insufficient experiments have been addressed by the rebuttal. I agree to upgrade the score to 6 and hope the authors can improve the writing and paper readability in the next version. 
summary_of_the_review: The problem of this paper is interesting, but the experiments are not sufficient to support the claim. Please see the main review for the details.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a model for classification (on ImageNet) that involves having a very small network act as a selector and early predictor, and a set of experts that specialise on a subset of the training data. The selector chooses one of them and thus keeps a constant cost irrespective of the number of experts. This is then combined with a number of existing techniques to raise the overall performance. Namely, the authors use a PWRL non-linearity instead of ReLU, they use logit-matching knowledge distillation, and conditional convolution (CondConv). The outcome is a tiny model that yields excellent performance on ImageNet, with performance envelopes superior to existing methods.
main_review: Let me first say that I like the paper and I think it can be very valuable to the community. It is practical, it performs really well, and achieves surprising results on a very crowded, high-impact hot topic, which is impressive. That said, there are some very important issues with the explanations and even with the experimental comparisons (despite some good range of ablations). My initial score is very likely to change after rebuttal depending on the authors reply.

Here are some of my issues:

- What the WGM, LGM and SRM do is not clear. There are some verbal descriptions, which do not clarify much, and the maths is not very clear either. The WGM (btw weight generation module, the name seems non-informative at best to me, assignment generation module? assignment balancing module?) seems to do something very similar to the LGM. My understanding is that the LGM assigns an expert to each training instance, thus generating the labels that are used to train the delegator. But what is the role of the WGM? is it balancing so that there is no collapse to a single expert? I am not sure I get why the WGM is needed and what it does in practice. 
I think I understand what the SRM module does, but I'm not sure either. Why is it needed? My best guess given the first paragraph of sec 3.3 is that it softens the penalty of delegator mistakes based on how well the "wrong" experts do. But then, why use an indirect metric like statistics instead of using the actual loss of the chosen expert?

- Equations: Eq. 10 has a \mathcal{L}_j,k, but that's not defined before. I assume it is L_j,k as defined in the LGM? What is \mathcal_P in the first paragraph of Sec 3.4? Not defined or used either. Is it the same as \mathcal{L}_{total}?

- Sec. 4.1 shows the FLOPs of the networks involved. However, the delegator is 24MB and the expert 110MB. How do we get to 100MB?

- The early termination (early exit?) is virtually not mentioned in the whole paper. There is no explanation regarding whether/how/when it is used, training details, etc. Reading sec. 4.2.3 I recalled it being mentioned in the intro, but at that point I had forgotten about it.

- There is no mention on the paper on weather you would release the code upon acceptance. Given that some parts of the paper/method are hard to parse, that would be very helpful. Is this expected?

There are other less critical comments: 
- batching: it is mentioned that the proposed method is better at batching than other conditional convolution methods. However, it is still the case that different examples in a batch will go to different experts, thus negatively affecting batching vs. standard networks. Is this correct? 
- I would like to see MobileNetV3-large Table 2 
- what is the performance without KD? Can you ablate it?
- Did you ablate the elements on the optimization? I was very surprised at things like stochastic depth and autoaugment given the very tiny capacity fo the network. Are these elements really necessary? (e.g. see tradeoffs in data augmentation: an empirical study, ICLR'21). 
- There are some phrases that are hard to understand. The clarify of the paper would improve revising its English (this is not a factor for scoring, just advice on how to improve impact).
- Figures are too small, they cannot be read when printed. The only way to see them properly is to zoom (a lot) into the pdf.
- There are a large number of ablations and supplementary experiments. This thoroughness is appreciated. However, it would be interesting to see an ablation on some of the components of the model - for example, how important are the SRM? and the WGM? and the progressive sharpening of the assignments? etc (I guess this one relates to the KD ablation comment above). I understand this might take resources and time to run and might not fit into a rebuttal, but overall I think it is something missing.

Now, there's plenty of positives with the paper. The performance is excellent on a very important problem (how to do tiny ML effective). The awareness of the literature is superb. There are loads of experiments (CPU latency, vit...). There is solid novelty - I think many people have thought of this way of tackling the problem, but somehow this paper explains how to make it work, so extra kudos. I also like the aggregation of other pieces to build a very solid system (CC, KD, PWLR)... our field tends to focus on one single novelty per paper to maximize clarity (and avoid confusing reviewers?), but then it is very unclear how things stack up and leaves a lot of gap that needs to be filled by the practitioner. I like that this paper bridges the gap and offers a "ready to go" model.

So, all in all, I would really like clarity improved because I think the paper really deserves it. If that happens, I'll happily improve my score.

summary_of_the_review: The paper has a lot of pros and cons. The cons look solvable (clarity mostly), and the pros seem solid enough. The practicality of the approach, the thoroughness of experiments and comparisons, and the excellent performance attained are all very positive. As already mentioned, a good rebuttal that shows clearer explanations will mean I raise the score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an efficient mechanism for  image classification. It has a light-weight delegator to yield coarse prediction, where early termination is possible. Once the coarse prediction is not confident enough, it actuates an down-stream expert with higher computational power for refined prediction. Since the expert selection is done across models, the method can take advantages of heavily-tuned efficient neural networks. The so-called CoE framework achieves very competitive results on the ImageNet dataset with low FLOPs and low CPU latency.
main_review: ++ The paper proposes a novel approach to partitioning the entire dataset to individual experts while maintaining diversity.

++ The expert selection is done at the model level, so that the advantage of state-of-the-art efficient networks (OFANet, MobileNetV3) can be taken for small FLOPs and low CPU latency.

++ The proposed CoE is also applied to translation task with BERT and shows generalization ability.

-- Method: The module names are unclear. It makes me hard to understand the role until I read the main text. Furthermore, even if I go through the writing, I am still unsure what is the true difference between selection matrix $ A_{m\times n} $  and the selection label $ S_{m\times n} $. Are they the same thing but optimized separately by two different constraints (Eq.1 and 6)? Also, the authors didn't clearly specify how the inference works.

-- Why does the "Collaboration" means in the framework "Collaboration of Experts"? The selection is one-hot, so there is actually no collaboration between experts?

-- Unclear comparison. 
 (1) Table 1 shows CoE-Small/Large with both Conditional Computation (CC) and Knowledge Distillation (KD). I am curious the result of CoE-Small/Large without KD.
  (2) CoE-Large use OFA-230 as the expert network, so how could CoE-Large achieve smaller FLOPs compared with OFA-230? Is it also including the early termination?

-- Style of writing: A pair of parentheses over all references is preferred so that the the references will not be confused with the main text. I think "\citep" can do this.

summary_of_the_review: The paper combines the recent advances of efficient (low FLOPs and low latency) networks and some other techniques such as knowledge distillation, conditional conv (CondConv), and highly-engineering activation function. What is the true novelty is somewhat unclear to me. Also, the methods are not clearly written and how different modules work is confusing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the Collaboration of Experts (CoE) framework to both eliminate the need for multiple forward passes and keep hardware-friendly. A training method including weight generation module (WGM), label generation module (LGM) and selection reweighting module (SRM) is proposed to improve the expert selection. Results on ImageNet shows its hardware-friendly performance. 
main_review: Strengths: 

1). The propose CoE frames achieves good performance with little computation cost.

2). The design is hardware-friendly with extreme low-latency. 

3). Promoting within dataset diversity in the expert selection sounds novel. 


Weakness:

1). Isn't Eq (1) encourages more randomized selection? What is the motivation of WGM, if it is refined by SRM?

2). 80.0% top-1 accuracy on ImageNet is not hard. Does the design principle can scale well across different (data size, model parameters, FLOPs)?

3). I don't know which of the proposed components contribute the most to the overall performance. What can be concluded from Appendix Figure 7 and Figure 8?

summary_of_the_review: Good results, but lack of ablation study of each proposed components.
review_title: null

Meta-review: This paper trains an expert style DNN that routes input examples to appropriate expert modules resulting in high accuracy on ImageNet with less compute. Reviewers have been positive about the strong empirical results. However the paper itself is not written well and reviewers had hard time figuring out actual architecture and training methodology. For example reviewers couldn't easily figure out the differences between LGM, WGM and SRM. 

The paper itself is sparse on why some of the choices have been made, their relation to existing methods and how do they affect the final performance. For example - In eq2, TCP objective has been normalized for each expert separately with a vague No Superiority Assumption. What motivates this assumption? Why is it reasonable? Eq 4 is quite similar to the load balancing loss in Switch Transformer paper. However there has been no discussion about the similarities and differences.

I think the paper needs to rewritten with clear explanation of the actual architecture, in what aspects it is similar/differs to existing expert models. What key components are the reason for the superior performance?

While I appreciate the authors for the ablations studies they presented during response phase, I think the paper requires major rewriting and cannot recommend acceptance at this stage.
Decision: Reject