Collaboration of Experts: Achieving 80% Top-1 Accuracy on ImageNet with 100M FLOPs
Authors: Yikang Zhang
Zhuo Chen
Zhao Zhong
Venue: ICLR 2022 Submitted (2022)

Abstract: In this paper, we propose a Collaboration of Experts (CoE) framework to pool together the expertise of multiple networks towards a common aim. Each expert is an individual network with expertise on a unique portion of the dataset, which enhances the collective capacity. Given a sample, an expert is selected by the delegator, which simultaneously outputs a rough prediction to support early termination. To make each model in CoE play its role, we propose a novel training algorithm that consists of three components: weight generation module (WGM), label generation module (LGM) and selection reweighting module (SRM). Our method achieves the state-of-the-art performance on ImageNet, 80.7% top-1 accuracy with 194M FLOPs. Combined with PWLU activation function and CondConv, CoE further achieves the accuracy of 80.0% with only 100M FLOPs for the first time. More importantly, CoE is hardware-friendly, achieving a 3~6x speedup compared with some existing conditional computation approaches. Experimental results on translation task also show the strong generalizability of CoE.

Under review as a conference paper at ICLR 2022
COLLABORATION OF EXPERTS: ACHIEVING 80% TOP-1 ACCURACY ON IMAGENET WITH 100M FLOPS
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we propose Collaboration of Experts (CoE) framework to assemble the expertise of multiple networks towards a common goal. Each expert is an individual network with expertise a unique portion of the dataset, contributing to the collective capacity. Given a sample, delegator selects an expert and simul- taneously outputs a rough prediction to trigger potential early termination. For each model in CoE, we propose a novel training algorithm with three components: weight generation module (WGM), label generation module (LGM) and selection reweighting module (SRM). WGM adapts the losses of experts, enabling them to focus on different portions of the dataset. LGM generates the label to constitute the loss of delegator for expert selection. SRM aims to promote delegator to select experts better. CoE achieves the state-of-the-art performance on ImageNet, 80.7% top-1 accuracy with 194M FLOPs. Combined with PWLU activation function and CondConv, CoE further boosts the accuracy to 80.0% with only 100M FLOPs for the first time. Furthermore, experimental results on the translation task also demonstrate the strong generalizability of CoF. CoE is hardware-friedlf yielding 3~-6x acceleration compared with existing conditional computation approaches.
INTRODUCTION
There are many approaches for model collaboration, among which ensemble learning (Hansen & Salamon, 1990; Wen al., 2020; Wenzel et al., 2020) popular one. Ensemble learning uses a consensus scheme to decide the collective result by vote. However, it requires multiple forward passes, leading significant runtime cost. MIMO (Havasi et al., 2021) draws inspiration from model sparsity (Frankle & Carbin, 2019) and tries to ensemble several subnetworks within one regular network. It only needs one single forward pass of the regular network but is incompatible with compact models. Conditional computation methods (Cheng et al., 2020; Yan et al., 2015; Shazeer et al., 2017) alleviates this issue via delegation scheme, i.c. assigning one or several, rather than all models, conditionally to make the prediction. Some recently proposed conditional computation methods (Zhang et al., 2020b; Yang et al., 2019; Zhang et al., 2021) have achieved remarkable performance based on dynamic convolution. Nonetheless, they usually have high memory access cost (MAC) and a low degree of parallelism, which increases the real latency (Ma et al., 2018).
Motivated by this, we propose the Collaboration of Experts (CoE) framework to both eliminate the need for multiple forward passes and keep hardware-friendly CoE consists of one delegator and multiple experts. Firstly, delegator gives a rough prediction and makes the expert selection. If the rough prediction is unreliable, the selected expert will make the refined prediction. Otherwise, the procedure will be early terminated save FLOPs. Moreover, we only need to load the selected expert into memory, thus keep the ratio of MAC to FLOPs as constant. By contrast, dynamic convolution methods (Zhang et al., 2020b; 2021) need to load a large number of parameters. namely basis models experts, to synthesize the input-dependent ones. It enlarges MAC and reduces the degree of parallelism, resulting significant deceleration.
To make each model in CoE play its role, we propose novel training algorithm (as shown in Fig. which consists of three components: weight generation module (WGM), label generation module (LGM) and selection reweighting module (SRM). LGM generates the label (selection label) to constitute loss of delegator for expert selection (selection loss). Selection label is a one-hot vector, indicating the suitable expert for each given input. Due to the random initialization of experts, selection labels are irregular in the early training stage. Nonetheless, delegator tends to learn generalizable patterns first, since networks learn gradually more complex hypotheses during

---

Under review as a conference paper at ICLR 2022
Training
Dhee her vith podker
Fas Loe 1an
(Dea
Inference
Dvieggaoor
Early Termination
Delegater
Experts
Exgrrr
Expert
SdutiesLahei
sigHTi Lighty AIRET
Expertia
Figure I: Framework of CoE with 3 experts. Delegator outputs the probabilities to select each expert and a rough prediction to trigger potential early termination. Each input is processed through all experts during training, but only through at most expert during inference.
training (Arpit et al., 2017). With delegator the bridge, WGM can partition the training data into portions based on generalizable patterns, and reweight expert losses to impel each expert to focus on one portion. This procedure makes selection labels more regular in return. Thus, delegator avoids overfitting the irregular labels. SRM aims promote delegator to select experts better. is achieved by enforcing delegator to focus on samples whose recognition results are sensitive to expert selection.
Experimental results on ImageNet demonstrate the superiority of CoE. It achieves 78.2/80.7%, 70 top-1 accuracy with only 100/194M FLOPs, while the accuracy for ensembled models (Hansen & Salamon, 1990) is 79.6% with 920M FLOPs. Compared with dynamic network approaches, CoE more hardware-friendll. It only outperforms the SOTA dynamic method BasisNet which achieves 80.0% accuracy with 198M FLOPS (Zhang et al., 2021), ), but also accomplishes a 3. xx speedup on hardware. Besides, CoE can be equipped with CondConv and further improve the accuracy to 79.2/81.55 with 102/214M FLOPs. More surprisingly. we further boost the accuracy to 80.0% with only 100M FLOPs for the first time by using PWLU activation function (Zhou al., 2021). Experimental results on the translation task also demonstrate the strong generalizability of CoE.
The contributions of this paper can be summarized as follows:
We propose collaboration framework named Collaboration of Experts (CoE) and demon- strate that can lead to outstanding performance with little computation cost. Moreover, is hardware-frient friendly and able to achieve real speedup.
We present novel optimization strategy for CoE. The core insight is promote diversity within experts by distributing their expertise over different portions of the dataset.
We update the state-of-the-an on ImageNet for mobile setting, achieving 80.0% top-1 accuracy with only 100M FLOPs for the first time.
RELATED WORK
2. ENSEMBLE LEARNING AND MODEL SELECTION
Ensemble learning (Hansen Salamon, 1990) aims at combining the predictions from several models to get more robust one. Some recently proposed literatures (Wen et al., 2020; Wenzel et al., 2020) demonstrate that significant gains can be achieved with negligible additional parameters compared to the original model. However, these methods still require multiple (typically, 4-10) forward passes for prediction, leading to a significant runtime cost. Differently, CoE utilizes a delegator to select only one expert for the refined prediction, thus at most two forward passes are needed. MIMO (Havasi et al., 2021) draws inspiration from model sparsity (Frankle Carbin, 2019) and holds the view that multiple independent subnetworks can be concurrently trained within regular network because of the heavy parameter redundancy. Therefore, those subnetworks can ensembled with a single forward pass of the regular model. However, MIMO cannot be applied to compact models which
2

---

Under review as a conference paper at ICLR 2022
have already been pruned or the ones constructed by AutoML methods (Zhong et al., 2018: Zhang et al., 2020a; Cai et al., 2020). It is because these models are compact enough and have few redundant parameters. By contrast, CoE is free from the compactness of experts and compatible with various models. Recently, some works about model selection are proposed (Li et al., b; You et al., 2021). These methods are concerned with ranking number of pre-trained models and finding the one transfers best to downstream task of interest. Therefore, they select models task-wisely. By contrast, CoE aims improving the task performance via selecting the most suitable expert for each sample instance- instance-wisely. Moreover, these methods conduct model selection based on a set of samples (training set) which cannot be adopted in CoE.
2.2 DYNAMIC NETWORKS
Dynamic networks achieve high performance with low computation cost by conditionally varying the network parameters (Zhang ct al., 2020b; Yang et al., 2019) or network architectures (Yuan et al., 2020). HD-CNN (Yan et al., 2015) and HydraNet (Mullapudi et al., 2018) select branches based on the category, they cluster all categories into n groups, where n is the number of branches. While CoE learns the model selection pattern automatically. it can be based any property, rather than limited to the category. MoE (Shazeer et al., 2017) and Switch Transformer (Fedus et al., 2021) enable the direct training of Router by scaling the output feature of experts with the predicted gate-values of Router. These methods aim at conditionally selecting a specific layer or block. Differently, CoE can take more advantage of conditional computation as the selection of whole network makes every parameter input-dependent. The recently proposed Dynamic Convolution methods (Zhang et al., 2020b; Yang et al., 2019; Chen et al., 2020) share the same idea and achieve remarkable performance with low FLOPs but high latency. It is because these methods need to load many basis models or experts to synthesize the dynamic parameters, causing high MAC and low degree of parallelism (Ma et al., 2018). By contrast, CoE only needs to load the selected expert into memory, avoiding these problems. Finally yet importantly, batch processing is an important method to enhance the degree of parallelism. Because of the input-dependent parameters (Zhang et al., 2021) architectures (Yuan et al., 2020), these methods cannot process samples in batch. Differently, CoE supports batch processing because the number of experts is limited and each one of them corresponds to many test samples.
METHOD
As shown in 1, CoE consists of delegator and experts, total of n + individual neural networks. To make the best of limited ca- pacity, each expert is encouraged to focus on just one unique portion of the dataset. This is achieved by WGM which reweights train- ing losses of experts. Given a sample, delegator selects an expert and simultaneously outputs a rough prediction to trigger potential early termination. generated from LGM that indicate the suitable expert. Moreover, SRM aims to boost delegator by enforcing it to focus on samples whose recognition results are sensitive to expert selection. Since the inference of delegator is conducted all the time, we prefer to make delegator more lightweight than expert. Delegator consists of three modules: feature extractor, task predictor and expert selec- tor shown in 22g.2. Based on the feature derived from feature extractor, task predictor and expert selector output the probabilities for classification and expert selection respectively. We describe the inference procedure and training strategy of CoE comprehensively in the following subsections. The number of samples and experts are denoted as m and n respectively.
Pxpast Selection
Bough
Predktion
SorcMax
Exper Selector
ltnk
SuroMar
Preditto
100
trained with cross-entropy loss and labels
Festure Extrndor
Figure 2: Architecture of dele- gator.
3.1 INFERENCE PROCEDUREF
CoE firstly uses delegator to obtain the rough prediction and determine the selected expert for each sample. Afterward, Maximum Class Probability (MCP, Corbière et al. 2019) of rough prediction is calculated. It is the probability of the predicted class. For samples with MCP larger than given threshold r, the final recognition result is derived from the rough prediction (early termination). Other samples are partitioned into n groups based on which expert is selected. Then batch processing can be conducted within each group to obtain the refined prediction. The averaged FLOPs/nnstance of CoE ranges from Fp to FD + FE by varying T, FD and FE are FLOPs of delegator and experts.
3

---

Under review as a conference paper at ICLR 2022
3.2 LABEL GENERATION MODULE (LGM)
IGM generates the label (selection label) to constitute loss of delegator for expert selection (selection loss). Selection label is one-hot vector, indicating the suitable expert for a given input. Model accuracy can be measured by True Class Probability (TCP, Corbière al. 2019):
JCP P(Y = yy my ;xpert,),
where, By is the j-th sample, Y and yjj are the predicted and true class. But accuracy is not the only factor for suitability. For example, when models are of different sizes, the larger one usually has higher TCP. But it may not be more suitable, due to the large inference cost. Given that our concern is not the optimization of network architecture, Superiority Assumption, NSA). Motivated by NSA, we leverage the standardized TCP as the metric for sutability:
suppose expert is superior to others (No
SJ.² TCP j.b (CCP
(2)
Std(TCP,,))
where, Mcan(TCP.W) and Std(TCP..,) are mean value and standard deviation for TCPs of Experty.
Given samples and experts, selection labels can be denoted by a binary matrix Lmxxs thus each row of which is a selection label. According to NSA, the sum of each colum vector Lmxn should be same, namely Σ, Li.b= for l I,..., n. Therefore, L can be obtatined by maximizing the sum of standardized TCP (i.e. SJ,x in Eq.2):
min
Σ-5,e Sj.k T.j.k
j.k
s.t. L (0.1), Σ4 = m
This problem can be modeled the balanced transportation problem (BTP, Shore 1970), where each sample is a supply source with a supply of one, each expert is a demand source with a demand of m/n. SSA is the per-umit transportation cost from the j-th supply source to the k-th demand source. We solve this problem via Vogel approximation method (VAM, Shore 1970) as introduced in Appendix A. which is short-cut approach to invariably obtain good solution.
3.3 WEIGHT GENERATION MODULE (WGM)
To maximize the collective capacity of CoE, the dataset needs to be partitioned into portions then WGM encourages each expert to focus one portion by reweighting losses of experts. The partition can be indicated by an assignment matrix Amxns with one-hot row vectors. Aÿ, = means the j-th sample zy is assigned to the k-th expert, thus the loss weight for Expert, gets larger than other experts on Zz--
A naive partition can be based expert suitability, namely, partitioning the dataset with selection labels Lmxn generated from LGM. However, it results a poor generalization to delegator. As- suming Expert, is suitable on sample Ty, thus Aj.: I.J.A 1. Due AJ.x 1, the loss weight for Expert, gets larger than other experts on Tj. making Expert, more suitable in return. Therefore selection labels cannot be updated. Moreover, selection labels are irregular in the early training stage because of the random initialization, thus selection labels will remain irregular consistently. With the training going on, delegator will overfit to those irregular labels, yielding poor generalization as shown in Fig.3a. As a result, CoE performs poorly because delegator can hardly select suitable expert during validation. This also verified Appendix B.4.3.
Since networks learn gradually more complex hypotheses during training (Arpit et al., 2017), delegator tends to learn generalizable patterns first. Therefore, the partition can be based on generalizable patterns with delegator as the bridge. In this way, selection labels get more regular in return thanks to the reweighting of expert losses in WGM. As shown in Fig.3b, delegator avoids overfitting to the irregular labels, hence generalizing well to the validation set.
Given m samples and experts, delegator outputs probability matrix PmXX E Rmx whose element PR.* e [0, 1] represents the probability of assigning the j-th sample to the k-th expert. As analyzed above, it is better to partition the training data based on Pmxn- thus Amxn can be obtained by maximizing Σj. PjA+ Aj,x- Moreover, according NSA, the number of samples assigned to each expert should be same, i.e. Amxa needs satisfy E, Aj,A = m/n. Thus, Amxa xn n is optimized by:
4

---

Under review as a conference paper at ICLR 2022
e: Exper, is sutileee Exper,
Portion Exper, Portion lispen,
Mic Hypothesis delegstor
(a) Partition the dataset based expert suitahility.
(b) Partition the dataset based on delegator. Figure 3: A demo to illustrate the training procedure of CoE. The training samples are denoted as green triangles or pink circles based on which expert is suitable. They are partitioned into two portions based expert suitability hypothesis learned by delegator, then WGM enables each expert to focus on one portion. Delegator is trained with selection labels that indicate the suitable expert, thus delegator learns hypothesis select the suitable expert for each sample.
min
Σ -P3. Aj.k
J.k
(4)
s.t.
A E 10,1),
AA 3 ΣAA==
This problem can also be modeled BTP, and solved via VAM as described in section 3.2. In the early training stage, experts are underfitted. Thus we cannot trust Amxn and need to make the gap between loss weights for experts smaller. We achieve this by smoothing Amxn to Amxn with Eq.5, where grows linearly from 0.2 to 0.8 with the training going on,
{
a 1-a n
if A3 =
AsAh =
(5)
if AjA -0
Finally, the output of WGM (i.e. W w,xx) is obtained by normalizing Amxn with the coefficient Z Aj
A₁. Z
W =
(6)
3.4 SELECTION REWEIGHTING MODULE (SRM)
Expert suitability can be measured with standardized TCP (Eq.2). If experts have similar suitabilities for given sample, the expert selection will have little influence the final performance of CoB. As the capacity of delegator is limited, it should pay less attention to those samples. This is achieved by SRM, which reweights losses of delegator based on statistic that reflects suitability similarity.
The output of SRM is denoted as {eyli = 1, I.., m), which are weights for selection losses over m samples. The suitabilities for experts over the j-th sample, i.e. {S = I.... n) are denoted as Sj.- A small value of the standard deviation Std (S;.)) indicates experts have similar suitabilities on the j-th sample, thus vj gets smaller. Therefore SRM determines the value of vy by:
Sta(S,.)
0i Σ Std(S..))
3.5 TRAINING METHOD
The training framework CoE is shown in Fig 1, which consists two stages. During the first stage, only feature extractor and task predictor of delegator trained minimize L.p, cross-entropy loss for the rough prediction of delegator. During the second stage, we fix these two modules, jointly optimize the expert selector and experts with LTotal:
Lrotal = 1) * Ls + Lr.
here, the hyperparameter 17 is set 0.8.
Ls used optimize the expert selector. Based on the selection label Lj., we can get the cross- entropy loss C/g for the j-th sample. Then Cg is determined by the weighted sum of (Co = 1...., m) with weights (uylů 1,...:m) output by SRM:
5

---

Under review as a conference paper at ICLR 2022
Cs Cs= = Es=Cus
Ug* C'c
Lr is used to optimize the experts. Based on the class labels of m samples, we can get m x cross-entropy losses (C4"15 1, ...., mi k-th expert on the j-th sample. Then Lr is obtained by the weighted sum of 24 with weights W output by WGM:
k L,....; ..., where Cif is the cross-entropy loss for the ,,...,
LT = Σ W
C
(10)
J.k
Each of our experiments includes either four sixteen experts in this paper. When using four experts, the training method is same as described. However, we will meet slow-convergence problem if the number of experts is sixteen. To alleviate this problem, we propose strategy which is described in Appendix A.2.
EXPERIMENTS
We conduct the main experiments on ImageNet classification task. After comparing with some popular methods in terms of computation and memory cost, verify the superiority of CoE over some other existing model collaboration methods. Moreover, we try to generalize CoE to the translation task and re-evaluate CoE using Reassessed Labels (Real.) (Beyer al., 2020). Finally, we try to analyze the reasonability of learned expert selection patterns. Elaborated ablation studies are illustrated in Appendix B.4. Statistics on referenced baselines in section 12.2.2 are directly cited from original papers, others are implemented with the following setting unless otherwise stated.
4.1 IMPLEMENTATION DETAILS
We conduct experiments with two settings: CoE-Small and CoE-Large. For CoE-Small, we take TinyNet-E (Han et al., 2020b) with 24M FLOPs as the feature extractor of delegator by removing the last fully connected layer. We use OFA-110 (Cai et al., 2020) with 110M FLOPs as the expert. For the CoE-Large, MobileNerV3-Snm construct the delegator by analogy. We use OFA as the experts. We have also tried to introduce CondConv (Zhang et al., 2020b) and PWLU activation fuction (Zhou et al., 2021) to achieve the extreme performance. To combine with CondConv, we replace the convolutions within each inverted residual block of the experts with CondCony (ezpert_naue num 4). To take advantage of PWLU, replace all activation layers except those that have tiny input feature maps as illustrated in Zhou et al. (2021). Models are trained using SGD optimizer with 0.9 momentum. We use mini-batch size of 4096, and a weight decay of 0.00002. Cosine learning rate decay is adopted and the number of training iterations 313000. We use fixed auto-augment (Cubuk al., 2019) as well. Inspired by BasisNet, we use knowledge distillation with EfficientNer-B2 (Tan & Le, 2019; Xie et al., 2020) as the teacher. The learning rate is 0.8/1.6 for CoE-SmallVLaage and dropout rate is 0.2. The stochastic depth (Huang et al., 2016) is used except for TinyNet-E with a survival probability of 0.8.
(Howard et al., 2019) with 56M FLOPs is adopted to
4.2 RESULTS AND ANALYSIS
4.2.1 ACCURACY AND COMPUTATION COST
Accuracy curves for CoE in Fig. are drawn by varying the threshold of early termination (sec- tion 3.1). We pick out a point from each curve to compare with some efficient networks in Table Our method achieves 78.2% and 80.7% top-1 accuracy with 16 experts and averaged FLSPa/Instance as 100M and 194M respectively. Compared with OFA, CoE reduces the FLOPs from 230M to 100M and from 595M to 194M, with better top-1 accuracy. Compared with EfficientNet-BI with noisy student training, CoE also reduces the FLOPs by 3.6x while improving the accuracy by 0.5%. Though dynamic networks like GFNet, CondCony and BasisNet are more efficient than traditional networks, CoE still has significantly higher accuracy with smaller FLOPs. Compared with these approaches, CoE improves the accuracy by 2.2/2.4/0.7%, respectively. When combined with CondConv, can achieve 79.2% and 81.5% top-1 accuracy with only 102M and 214M FLOPS respectively, which indicates CoE is complementary and BasisNet share similar essence, namely using a group of basis to dynamically synthesize the input-dependent convolution kernel, the combination of them only arouses little collaborative benefit with the top-1 accuracy of only 80.5%. More surprisingly, we achieve the accuracy of 80.0% with only 100M FLOPs for the first time by further making use of PWLU.
dynamic networks like CondConv. On the contrary, as CondCony

---

Under review as a conference paper at ICLR 2022
Table I: CoE performance ImageNet. "CC" and "KD" indicates conditional computation approach and knowledge distillation strategy.
Mcthod
CC KD
FLOPs
TOP-1 Acc
WeightNet (Ma et al., 2020)
141M
72.4%
DS-MBBeetMM* al., 2021a)
319M
72.8%
GhostNet 1.0x (Han al., 2020a)
141M
73.9%
MobileNerV3-Large (Howard et al., 2019)
219M
75.2%
OFA-230 (Howard ct al., 2019)
230M
76.9%
TinyNet-A (Han et al., 2020b)
339M
77.7%
CondConn-EEficiietNee-B0 (Yang al., 2019)
413M
78.3%
GFNet (Wang et al., 2020)
400M
78.5%
CoF-Small
100M
78.2%
CoF: Small CondConv
102M
79.2%
CoF-Small + CondCony + PWLU
100M
80.0%
BasisNet (Zhang et al., 2021)
V
198M
80.0%
OFA-595 (Howard ct al., 2019)
595M
80.0%
Efficien(Net-22 (Tan & Le, 2019)
1.OB
80.1 %
ElliciennNee-BINNoissy Student) (Xie al., 2020)
700M
80.2%
BasisNet (Zhang et al., 2021)
290M
80.3%
FBNetV3-C (Dai et al., 2020)
557M
80,5%
BasisNet + CondConv (Yang et al., 2019)
V
308M
80.5%
CoE-Large
194M
80.7%
CoE Carge CondConv
214M
81.5%
4.2.2 INFERENCE SPEED AND MEMORY COST
Compared with conditional computation methods (Yang et al., 2019; Zhang et al., 2021), CoE is more hardware friendly. To verify the advantage, we also analyze the inference latency on hardware. The experiments conducted CPU platform (Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz) with PyTorch version as 1.8.0. We report the averaged latency on the ImageNet validation set in Table 2. We notice the discrepancy between FLOPs and real speed. For example, OFA-230 has 1.6x FLOPs compared with GhostNet 1.0x, but the speed is 1 1,2x faster. Moreover, this discrepancy can be enlarged by CondConv. EfficientNet-B BOO, but the speed is 1.7x slower. BasisNet synthesizes the dynamic parameters all at once, rather than the "layer by layer" fashion like CondConv, thus more efficient. However, it still needs to load large number of parameters for this synthesis, which brings large MAC. This is why CoE (16 experts) can reduce 14.09% latency than BasisNet when the mini-batch size is one. Finally yet importantly, BasisNet and CondConv do not support batch processing, while CoE (16 experts) can take advantage them. We analyze the memory cost from two perspectives: the number of parameters and MAC. As can be seen from Table 2, the accuracy of CoE-Large (4 experts) is no worse than BasisNet and when using similar parameters. Besides, the averaged MAC/Instance of CoE is much smaller than theirs. Compared with GhostNet 1.3x, the accuracy for Coe-Large (16 experts) 5,0% higher with smaller MAC.
GondConv-EffeiennNe
has similar FLOPs with the original
(seciton 3.1) to further achieve 3.1/6.1x speedup compared with
Table 2: CPU latency and memory cost for different methods.
CPU Latency/Instance (ms)
Models
FLOPs
MAC
Params
Accuracy
Batchsize=1
Batchsize=64
MobileNetV3--Smal GhostNet 1.0x
14.77
4.18
56M
2.5M
2.5M
67.4%
39.91
16.50
141M
5.2M
5.2M
73.9%
TinyNet-B
34.58
19.44
202M
3.7M
3.7M
75.0%
MobileNetV3--arge
31.55
18.43
219M
5.4M
5.4M
75.2%
GhostNet 1.3x
43.94
29.70
226M
7.3M
7.3M
75.7%
OFA-230
33.52
15.21
230M
5.8M
5.8M
76.9%
EfficientNet-B0
49.12
35.21
391M
5.3M
5.3M
77.2%
Net-A
45.76
23.71
339M
5.
5.1M
77.7%
81.81
413M
24.0M
24.0M
78.3%
BasisNet
40.61
198M
24.9M
24.9M
80.0%
CoF-Large (4 experts)
38.67
15.02
220M
6.6M
25.7M
79.9%
CoE-Lan CorgLarge (16 experts)
34.89
13.30
194M
6.0M
95.3M
80.7%

---

Under review as a conference paper at ICLR 2022
12%
OoB Large CC
CoE-Large (4 (epperts)
OFA-230
002 Smak
IMALE
50%
79.7%
mp.
7y-0kkuicco Leng
2593
1000
77%
78.7%
4000 VS.
coaradiaaal
2000
78.2%
500
75%
dPfee
327.18355
78.06
738
77.2%
72%
50
300
150
200
250
300
150
400
4.0
502
100
150
Training Cost days)
(a) Comparison with some popular networks on Im-
(b) Accuracy v.s. training cost for OFA-230 and CoE-
ageNet. "CC" and "PWLU" means CondConv and PWLU activation function.
Large (with 4 OFA-230 experts).
Figure 4: Accuracy v.s. FLOPs and training cost on ImageNet. 4.2.3 ANALYSIS THE TRAINING COST
To achieve superior performance with less inference FLOPs and latency, CoE may consume more training time. For example, the applying of CoE (4 experts) on OFA-230 improves the accuracy from 78.0% 79.9%, but the expense of 2.2x training cost. To verify whether the improvement still exists with similar training cost, we get series of accuracies by varying the number of training epochs as shown in Fig.4b, where "xx ep." means the number of training epochs is "xx" and 32 Tesla- V 1110-PCCe-16600 (from 78.3% to 79.9%) even if the training cost controlled to be similiar (127 vs GPU days).
GPUs are used for training. It is seen that CoE can boost the performance
4.3
COMPARISON WITH OTHER MODEL COLLABORATION APPROACHES
4.3.1
COMPARISON WITH REGULAR MODEL ENSEMBLE
We train four OFA-230 models with different initialization seeds. The results are shown in Table The random seed only causes minor variety in accuracy. However, an improvement of 1.6% is still achieved via ensemble. It is because these models fall into different local minima, yielding the diversity of output. Compared with the ensemble, CoE achieves 0.3/1.3% higher accuracies. indicates CoE can realize more potential of each individual model. Besides, CoE keeps the computation cost constant, while model ensemble increases the computation cost by four times.
4.3.2 COMPARISON WITH MODEL SELECTION METHOD
HD-CNN (Yan et al., 2015) and HydraNets (Mullapudi et al., 2018) select branches based on the category. MoE (Shazeer et al., 2017) and Switch Transformer (Fedus et al., 2021) enable the direct training of Router by scaling the output feature of experts with gate-values predicted from Router. Despite these methods are originally designed to conditionally select specific layer block, we apply them to the expert selection.
To select the expert based on category, the categories should be partitioned into groups, where is the number of experts. We try two schemes: random partition (RP) or clustering- based partition (CBP). Then, an expert can selected according to the rough prediction of delegator. During the training procedure, we also reweight losses of each expert based on the assignment matrix Amxn with Eq.5 and 6. Amxn is obtained directly based on the rough prediction. We can enable the direct training of expert selector well. Directly trained selector (DTS) is obtained by scaling the loss of each expert, rather than the output feature of each branch as the original paper (Shazeer et al., 2017; Fedus al., 2021). While CoE optimizes the expert selector via weighted cross-entropy loss L. (Eq.9). The results with 4 experts are shown in Table 3. It can be seen that CoE outperforms the compared methods, demonstrating better collaboration pattern is learned.
4.4 ANALYSIS OF THE GENERALIZATION
To verify the generalizability, conduct two extra experiments: generalizing CoE to translation task and using Reassessed Labels (ReaL) (Beyer et al., 2020) to re-evaluate CoE. We mainly introduce the first one here, another discussed in Appendix B.I.
8

---

Under review as a conference paper at ICLR 2022
Table 3: Comparison with other model selection methods. "RP", "CBP" and "DIS" means "Ran- dom Partition" "Clustering-Bar Based Partition" and "Directly Trained Selector" respectively.
Table 4: Comparison with model ensembe.
Method
FLOPs Acc.
Seedl
230M
78.1%
Sced2
230M
78.0%
Method
FLOPs
Top-1 Acc.
OFA-230
Sced3
230M
78.
RP
220M
78.3%
Sced4
230M
78.0%
Category- Based
CBP
220M
77.5%
Ensemble
920M
79.6%
Experts
220M
79.9%
DTS
220M
78.7%
CoE-Large
Selector-Bas -Based
CoE
220M
79.9%
16 Experts
220M
80.9%
To generalize CoE to translation task, we build ToF-Traasformer model based on Transformer (base model) ( (Vaswani et al., 2017). Considering the decoding procedure is much more time-consuming than encoding because target tokens are generated one by one, only the decoder of CoE- Transfommer is conditionally selected. Con-Transformer has four decoders, given sentence, one decoder will be selected to decode the features output by encoder. To select the decoder, an extra constant token is added the beginning of each sentence, the feature of this token output by encoder is input to the Expert Selector (Fig.2) for expert (decoder) selection. During training, the TCP of a sentence is obtained by averaging the TCPs of each token.
Following (Vaswani et al., 2017; Ott et al., 2019), CoE-Transformer is trained on the standard WMT 2014 Germma dataset. As mentioned, extra token will be added into this vocabulary. We adopt the same training and evaluating setting as (Ot et al., 2019), more details are shown in Appendix B.2. Although CoE-Transformer has larger number of parameters compared with Transformer (base model), the MAC is nearly identical. We calculate MAC based on the number of parameters loaded given a sentence and report it on Table 5 as well. From Table 5 we can see, Tran-frrofrrmmr outperforms the regular Transformer (base model) by large margin and achieves similar performance with Transformer (big).
Table 5: The BLEU scores newstest2014 Singlish-to-Germee
Model
MAC
Parameters
BLEU
Transformer (base model)
62.4M
62.4M
28.1 (Out et al., 2019)
Transformer (big)
213.0M
213.0M
29.3 (Ott et al., 2019)
CoE-Troantee Transformer
62.5M
138.2M
29.4
4.5 ANALYSIS OF THE LEARNED EXPERT SELECTION PATTERNS
We also conduct experiments to analyze the expert selection patterns of CoE and find them quite reasonable. When experts have different architectures. the delegator learns assign easy samples to smaller experts and complex samples to heavier experts. When experts share the same architecture, delegator learns the expert selection patterns automatically, it can be based on any property (e.g. whether humans are contained), rather than limited to the category. The details are shown in Appendix B.3.
CONCLUSION
We propose a CoE framework to pool together the expertise multiple networks towards a common aim. Experiments in this paper demonstrate the superiority of CoE on both accuracy and real speed. We also analyze the collaboration patterns and find them has interpretability. will be extended to the trillion parameters level. Meanwhile, we will try to implement CoE to more tasks and verify its compatibility with quantification and other technologies. Besides, CoE can be conducted to solve the problems of lifelong learning by updating experts.
the future, CoE

---

Under review as a conference paper at ICLR 2022
REFERENCES
Devansh Arpit, Stanistaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste- Julien. A closer look at memorization in deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 233-242. PMLR, 06-11 Aug 2017. URL html..
https: httpp::/ppoceeeeeeeeee http://pooceeeeigs////e wwriiiiinnsmmmm...mmmmemm www.m.rppp... www.pp..ss............. http:///.wm...m..p.sse///o////a/
Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are done with imagenet? CoRR, abs/2006.077 1599 2020.
Han Cai, Chuang Gan, Tianzhe Wang. Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In International Conference Learning Representations, 2020, URL https: wpwpreweeeeeeeeeeeooepeee net/ forum
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11300-1 1009,, 2020.
An-Chich Cheng, Chich Hubert Lin, Da-Cheng Juan, Wei Wei, and Min Sun. Instanas: Instance- aware neural architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3577-3584, 2020.
Charles Corbière, Nicolas THOME, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. Addressing failure prediction by learning model confidence. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: 2tt.:aaa....aa111-1e..pe.p.. cc/ cc/pppppeeeee hts/paper/2019 (tt11////11e777/73221211 wwwc66906666300 (001
neurips.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, et al. Fbnetv3: Joint architecture-recipe search using neural acquisition function. arXiv preprint WxXv:2006.022449 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. Ieee, 2009.
Muyuan Fang, Qiang Wang, and Zhao Zhong. BETANAS: balanced training and selective drop for neural architecture search. CoRR, abs/1912.11191,, 2019.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2 2021.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https: //openrevi eew net foorumid=rr]---
Raphael Gontijo-Lopes, Sylvia Smullin, Ekin Dogus Cubuk, and Ethan Dyer. Tradeoffs in data augmentation: An empirical study. In International Conference on Learning Representations, 2021. URL https: opwprevieneeeeeeeeee
Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1577-1586. 2020a. doi: 10. (0.11000CCVP446602200..00066
Kai Han, Yunhe Wang, Qiulin Zhang, Wei Zhang, Chunjing Xu, and Tong Zhang. Model rubik's cube: Twisting resolution, depth and width for tinynets. In NeurIPS. 2020b. URL httpps://proceedingg:
www.innggnn.uuurreee weeuuips.ccpppppr... 22tp//222h///////////////// hash/ Abst.rat.... e0 400e -ast html.
10

---

Under review as a conference paper at ICLR 2022
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions pattern analysis and machine intelligence, 12(10):993-100 1001, 1990.
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lak- shminarayanan, Andrew Mingbo Dai, and Dustin Tran. Training independent subnetworks for robust prediction. In International Conference on Learning Representations, 2021. URL
https: oppnreviewww net. wtw.reewwwwweee/tooummlooooooooo ( 9X
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition (CVPR), June 2016.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang. Vijay Vasudevan, al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314-1324, 2019.
Gao Huang and Danlu Chen. Multi-scale dense networks for resource efficient image classification. ICLR 2018, 2018. URL https::pp// /par/.aaa..//// nsf. gov/bibl o 0/00064650
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pp. 646-661. Springer, 2016.
Changlin Li, Guangrun Wang, Bing Wang, Xiaodan Liang Zhihui Li, and Xiaojun Chang. Dynamic slimmable network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8607-8617, June 2021a.
Yandong Li, Xuhui Jia, Ruoxin Sang, Yukun Zhu, Bradley Green, Liqiang Wang, and Boqing Gong. Ranking neural checkpoints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2663-2673, June 2021b.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116-131. 2018.
Ningning Ma, Xiangyu Zhang, Jiawei Huang, and Jian Sun. Weightnet: Revisiting the design space of weight networks. CoRR, ahs/2200.118233 2020.
Ravi Teja Mullapudi, William R Mark, Noam Shazeer, and Kayvon Fatahalian. Hydranets: Spe- cialized dynamic architectures for efficient inference. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pp. 8080-8089, 2018
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross. Nathan Ng. David Grangier. and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gattd mixture-of-expenn layer. arXiv preprint 1701. 06538,, 2017.
Harvey Shore. The transportation problem and the vogel approximation method. Decision Sciences, 1(3-4):441-457, 1970.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-61 14. PMLR, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp. 1000-6010, 2017. URL http://papers n nips wwtppcpppppcceccccccc 81 atteationniittt - 1-you-need

---

Under review as a conference paper at ICLR 2022
Yulin Wang, Kangchen Lv, Rui Huang, Shiji Song, Le Yang. and Gao Huang. Glance and focus: a dynamic approach to reducing spatial redundancy in image classification. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp- 2432-2444. Curran Asso- ciates, Inc., 2020. URL ptttpp::/pproceediinn neurii wweriis.cc/papeee wwcppppppr//////////cc
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations, 2020. URL www.thtpp//// / /tppnreppeeeeeeeeeeeeee ew. net forum? i f1
Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. arXiv preprint arXi::2006. 13570, 2020.
Qizhe Xie, Minh- Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10687-10698, 2020.
Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Zhicheng Yan, Hao Zhang, Robinson Piramuthu, Vignesh Jagadeesh, Dennis DeCoste, Wei Di, and Yizhou Yu. Hd-cnn: hierarchical deep convolutional neural networks for large scale visual recognition. In Proceedings of the IEEE international conference on computer vision, pp. 2740- 2748, 2015.
Brandon Yang, Gabriel Bender, Quoe V Le, and Jiquan Ngiam. Condconv: Conditionally parameter- ized convolutions for efficient inference. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: /ptp//eeeodnnnseeeeeeeeep.e..sse ieeurrps.. ps. cc/ paper/2011 /201 9/ fileff/lf///e/e wateeeff2200000 151 91 a 449ccca f0 4 f409 6 P.p.-....p.aa.. pdf.
Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre-trained models for transfer learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12133-12143. PMLR, 18-24 Jul 2021. URL https: http:///e/oceeeses.e/eeo mlr. mlr.prrss// v13 / .html.
Kun Yuan, Quanquan Li, Dapeng Chen, Aojun Zhou, and Junjie Yan. Dynamic graph: Learning instance-aware connectivity for neural networks. arXiv preprint arXiv:2010.01097, 2020.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi- supervised learning. In Proceedings of the IEFF/CVE International Conference on Computer Vision (ICCV), October 2019.
Mingda Zhang, Chun-Te Chu, Andrey Zhmoginov, Andrew Howard, Brendan Jou, Yukun Zhu, Li Zhang, Rebecca Hwa, and Adriana Kovashka. Basisnet: Two-stage model synthesis for efficient inference. arXiv preprint arXiv:2105.03014 2021.
Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong, Adversarial autoaugment. In International Conference on Learning Representations, 2020a. URL / wttps////e/e/eeeeee/rere.e net forumm id=ByxdUySKvS.
Yikang Zhang, Jian Zhang, Qiang Wang, and Zhao Zhong. Dynet: Dynamic convolution for
accelerating convolutional neural networks. arXiv preprint arXiv:2004, 20X60:2004.10694 2020b.
Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng- Liu. Practical block-wise neural network architecture generation. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
Zhao Zhong, Zichen Yang. Boyang Deng, Junjie Yan, Wei Wu, Jing Shao, and Cheng Lin Liu. Blockqnn Efficient block-wiss neural network architecture generation. IEEE Transactions Pattern Analysis and Machine Intelligence, 43(7):2314-23 2328, 2021. doi: 00.11.99PPAM...000... 2969199 193.
12

---

Under review as a conference paper at ICLR 2022
Yucong Zhou, Zezhou Zhu, and Zhao Zhong. Learning specialized activation functions with the piecewise linear unit. CoRR, abs/2104.03693. 2021.
13

---

Under review as a conference paper at ICLR 2022
A EXTRA DETAILS FOR METHOD
A.1 INTRODUCTION OF VOGEL APPROXIMATION METHOD (VAM)
In weight generation module (WGM) and label generation module (LGM), we need to solve the balanced transportation problem (BTP, (Shore, 1970)) via Vogel approximation method (VAM, (Shore, 1970)). We will introduce it in this section with the number of samples and experts as m and respectively.
The BTP involved in WGM and LGM has m supply sources, each of which is denoted Silo, with a supply of one, as well as demand sources. each of which is denoted as Milly with demand of m. Cg,& is the per-unit transportation cost from Silo, to Milly. Specifically, Cj -P,, j.k in WGM and Cj. = j,x in LGM. To make it clear, we illustrate this algorithm with toy example, where the problem is simplified as Fig.5 (a) with m 4, n In the first step, we calculate the penalty cost Frow, for each row and pCooly for each column of the tableau in Fig.5 (a). Penalty cost is determined by subtracting the lowest unit cost in the row (column) from the next lowest unit cost. The penalty costs of the respective rows and columns have been marked in red color for clarity in Fig.5 (b). Since the third row has the largest penalty cost (pCrows =11) and C3,1 is the lowest unit cost of that row, Siloy is allocated to Milli, i.. As.: [1.0] in WGM or La. [1.0] in LGM. Then the corresponding row should be crossed out and the demand of Mill, should minus one, if this results in zero demand, the first column will be crossed out well. After adjusting penalty cost for each row and column, the tableau becomes Fig.5 (c), where the changed values are marked in orange. The described procedure will be looped until no rows remained.
Considering the calculation of pCool, is much more time-consuming compared with pCrom; be- cause m >> n in WGM and LGM, we modify VAM by only seeking lowest penalty cost among (pCrows..... ..... PCrow. }. We find this modification makes VAM more efficient while keeps the supe- riority of the solution. It is because the mechanics of VAM makes meaningful take peeol, into account only when the demand of Mill, is one, which rarely happens. Thus, we adopt this modification to promote efficiency in this paper.
Mill: demand
Mill: demand
Mill; demand Place
Mill;: Plan
Mill: demand Pliccts
Mill: demand
suppy-uppyy
G =
C2y =
soppy: =1
poayss m2
Cry
10 C m
stpp:mupply phrcu #2
G
Siloy: supply-
C22
Stloyisupply Plrser
Siloyisapply =1 Pleanr
Cas 13
Cea = 18
Siloy:
G = U
G32 =
11
Silogi supply= =1
Plens m11
5 U
G32 = 12
Siloy supply 11
G
Ca2
suppyy Phrare
C
C22
Sileprupply -1 Pl =1
Cus =9
C42 =8
(a) Tableau the transportation problem
(b) Penalty costs of respectivve rows and columns
[c] Tableau after allocating Stln, to Mill,
Figure 5: The Vogel approximation method.
A.2 A STRATEGY TO FACE THE LARGE NUMBER OF EXPERTS
Each of our experiments involves either four or sixteen experts. For the four-experts setting, the training method is the same as described. However, we will meet the slow-convergence problem when the number of experts sixteen. motivates us to decompose the task into four subtasks, each of which involves four experts and can be trained with the proposed training method. With this strategy, the number of samples assigned to each expert increases from T6 to 4. Because these samples contribute most to the optimization of the expert, the rate of convergence becomes nearly four times faster.
m
m.
To fulfill task decomposition, we introduce new module to delegator, named subtask selector as shown in Fig.6. The subtask selector is used to allocate the input samples into different subtasks. The expert selector outputs sixteen probabilities, which are partitioned into four groups well. For
14

---

Under review as a conference paper at ICLR 2022
each subtask, only group of probabilities is visible. The experts within each subtask and the corresponding weights of the expert selector are jointly optimized. As for the feature extractor, task predictor, and subtask selector, their weights directly derive from the delegator trained with the setting of four experts and then fixed. During this procedure, the weights of subtask selector derive from the expert selector.
Expert Selection
Expert
Expert
1x
Subtosk
Subtosk2 Subtosk
Subtoski
Subtask
Subtask 2
Subtask
Rough Prediction
Subtask Selection
t
Task Predictor
Expert Selector
Subtask Selector
Feature Extractor
S
Figure 6: The modified architecture of delegator.
B EXTRA DETAILS FOR EXPERIMENTS
B.1 RE-EVALUATION WITH REASSESSED LABELS
As described in paper (Beyer et al., 2020), the validation set labels have set of deficiencies that makes the recent progress on ImageNet classification benchmark suffer from overfitting to the artifacts. To verify the generalization, we use the Reassessed Labels (ReaL) (Beyer al., 2020) to re-evaluate our method. The results are shown in Table 6. It can be seen that our method still has remarkable performance, achieving higher accuracy than the compared methods with significantly smaller FLOPs.
Table 6: ReaL and original top-1 accuracies. CC means CondConv.
Method
FL.PPs
Real. Acc.
Ori. Acc.
OFA-595 (Cai et al., 2020)
595M
86.0%
80.0%
S4L. MOAM (Zhai et al., 2019)
4B
86.6%
80.3%
ResNeXt-100 101 (Xie et al., 2017)
16B
85.2%
79.2%
ResNet-152 (le al., 2016)
11B
84.8%
78.2%
CoE-Large
194M
86.5%
80.7%
CoE-Large + CC
214M
86.9%
81.5%
15

---

Under review as a conference paper at ICLR 2022
B.2 EXPERIMENT DETAILS FOR THE TRANSLATION TASK
Following (Vaswani et al., 2017; Ott et al., 2019), CoE- Transformof is trained on the standard WMT 2014 English-German German dataset, which has shared source-target vocabulary of about 37000 tokens. As mentioned, an extra token will be added into this vocabulary. The default training setting is identical with the one described (Vaswani al., 2017), except for the batch size and learning rate become larger following (Ott et al., 2019) Moreover, the parameter a in Eq.5 grows linearly from 0.1 to 0.4 with the training going on. We report BLEU on news2014 with a beam width and length penalty of 0.6 based on single model obtained by averaging the last 5 checkpoints following (Vaswani et al., 2017; Ott et al., 2019).
B.3 ANALYSIS OF THE LEARNED EXPERT SELECTION PATTERNS
B.3.1 EXPERT SELECTION PATTERNS WHEN EXPERTS HAVE VARIOUS ARCHITECTURES
Considering TCP can measure the complexity of a given sample if the inference model is fixed, we experiment analyze the relationship between sample complexity and expert selection. If the experts have different computation cost, it is reasonable to assign easy samples (the ones with large TCP) to smaller experts and complex samples heavier experts. To verify this, we take four architectures searched via OFA (Cai et al., 2020) the experts, i.e. OFA- 10, OFA-163, OFA-230 and OFA-595. The delegator is also MobileNett3-smme each sample based on the delegator. We count the selection probability for each expert at different TCP values on the validation set. As shown in Fig.7, it meets our expectation that the selection probability for smaller models increases the input sample getting simpler (with the increase of TCP).
as described in section I. We obtain the TCP value for
1.0
Selection Probability
OFA-163
0.4
OFA-230
OFA-595
0.1
0.5
0.6
True Class Probability (TCP)
Figure 7: Selection probabilities for each expert at different TCP values.
B.3.2 EXPERT SELECTION PATTERNS WHEN EXPERTS SHARE THE SAME ARCHITECTURE
We have analyzed the selection patterns when the experts have different architectures, here we focus on the case that all experts share the same architecture. We adopt the CoE-Large setting with four experts.
Considering many works (Yan et al., 2015; Mullapudi et al., 2018) select branches based on the category, we firstly experiment to observe the relationship between selection patterns and rough prediction of the delegator on ImageNet validation set. Based on the predicted class of rough prediction, the validation set can be partitioned into 1000 subsets. Then we calculate the probabilities to select each expert within each subset and get 1000 probability vectors. After clustering, the probability vectors on Fig.8, each column of which is probability vector. can be seen that samples with the same rough prediction class assigned to different experts. Therefore, we conclude the expert not always selected based on category.
plot
Besides, further make qualitative analysis the ImageNet validation set and find some interesting patterns. For example, we find that images predicted as "meat market" are most likely to be assigned to the fourth expert if humans are contained. We show those images in Fig.9. It can be seen, 27
16

---

Under review as a conference paper at ICLR 2022
images assigned to the fourth expert, among which 22 images contain humans with ratio of 81.5%. By contrast, among the 32 images that are assigned to the other experts, only 7 images contain humans with ratio of 21.9%. This indicates CoE learns the expert selection patterns automatically, can be based on properties other than the category.
10
Selection Probability
Expert
Expert 2
0.4
Expert
Expert
101
201
301
401
501
601
701
801
901
Category
Figure& 8: Selection probabilities for each expert. The horizontal axis indicates the rough prediction class. The 1000 probability vectors are clustered for better visualization.
Samples assigned first expert:
Samples assigned to second expert:
Samples assigned expert:
Samples assigned to the fourth expert:
Figure 9: Images that are predicted 'meat market' by the delegator. They are partitioned into four groups based on which expert is selected. The red border indicates humans are contained, green border indicates humans are contained.
17

---

Under review as a conference paper at ICLR 2022
B.4 ABLATION STUDIES FOR CoE
B.4.1 EFFECT OF EXPERT NUMBER
We analyze the number of experts in this section, including 1, 4, and 16 experts. The results shown in Table 7. Using one expert brings little improvement compared with the original model. When increasing the number of experts, the accuracy becomes 1.9% better with four experts and 2.9% better with sixteen experts. It demonstrates CoE can make full use of multiple experts, leading to large collaborative benefit. What's more, the accuracy also reaches 79.9% by combining CondConv with OFA-230. In this manner, CoE can further enhance the accuracy to 80.8/81.555 with 4/16 experts.
Table 7: Comparison among different number of experts. "CC" indicates CondConv.
Method
Experts
FLOOPs
Acc.
OFA-230
230M
78.0%
220M
78.0%
CoB-Large
4
220M
79.9%
16
220M
80.9%
CC-OFA-230
242M
79.9%
214M
79.9%
CoE-Large CC
4
214M
80.8%
214M
81.5%
B.4.2 EFFECT OF EARLY TERMINATION
The original OFA-230 has 78.0% top-1 accuracy with 230M FLOPs. We can introduce a MobileNetV33 Small to conduct early termination. By varying the threshold, get series of accuracies and FLOPs as shown in Fig. 10. It can seen that the accuracy becomes 78.0% with 220M FLOPs. This indicates the computation cost brought by MobileNetV3--Snn termination strategy. Inspired by this, expect to eliminate the computation cost brought by delega- tor as well. does reduce the computation cost by 60/66M FLOPs, demonstrating the effectiveness of early termination.
is eliminated via early
OFA-230
OFA-230
CoE-Large (With ET)
CoE-Large (W/O
CoE-Large (With ET) CC
CoB CoE-Large (W/O ET) CC
81.6%, 238
298
60M
81.0%
66M
80.9%, 220
80.0%
78.0%,
78.
77.0%
78.0%
MFLOPs
Figure 10: Accuracy FLOPs. "ET" means Early Termination and "CC" indicates CondConv.
18

---

Under review as a conference paper at ICLR 2022
B.4.3 ABLATION STUDY FOR EACH COMPONENT OF COE
CoB consists of major components: WGM, LGM and SRM. Apart from directly removing each one component, we also try to alter some elements inside them. We propose several modified versions of CoE for ablation as below:
CoEWGM; Remove WGM from CoE. Thus, losses of experts have identical weights for each sample.
CoF WGM partitions the training data based expert suitability, i.e. the assignment matrix Amxn in WGM equals to the output matrix Lmxn of L.MM.
CoEWGMe Remove the "E, Ag m/n" constraint in Eq.4, so that Amxn neglects the No Superiority Assumption (NSA). Then, take Amxn the output of WGM without the smoothing (Eq.5) and normalizing(Eq.6),
CoEWGM" Remove the progressive sharpening of assignment in WGM. Specifically, using constant 0.8 for a in Eq.5, instead of linearly increasing it from 0.2 to 0.8.
CoELGM, Remove LGM from CoE. Thus, CoE collapse to single expert with delegator to
trigger the early termination.
CoELGM" Abandon the refining of suitability criterion (Eq.2) and remove the "E, Lj m//nconstraint in Eq.3. So that L neglects the No Superiority Assumption (NSA).
CoE Remove SRM from CoE.
We conduct experiments for those CoE versions with the CoE-Large setting and 4 experts. Results shown in Table 8, and the conclusions are listed below:
1. WGM and LGM are the most important components in CoE. The removal of WGM and LGM reduce the accuracy from 79.9% to 78.1% and 78.0%, respectively.
2. In WGM, the training data should be partitioned based delegator. Otherwise, delegator will overfit to irregular selection labels as illustrated in section 3.3. That is why Coe- LargeWGM* only achieves an accuracy of 78.4%.
3. The No Superiority Assumption (NSA) important for CoF. Without this assumption, LorgL GGM and CoE-Large only reach the accuracy of 79.2% and 79.4%.
4. The progressive sharpening of assignment in WGM can also boost the performance, thus the accuracy for CoE-Large WGM* 0.5% lower than CoE-Large.
5. The component SRM is also useful. It promotes delegator to select experts better, yielding 0.4% improvement for accuracy.
Table 8: Ablations for each component of CoE.
Method
Experts
FLOPs
Acc.
CoE-Large WGM
W
220M
78.1 %
CoE-Large WGM
220M
78.4%
CoB-Large WGM
220M
79.2%
CoF-Large Large" WGM
4
220M
79.4%
CoE- CargLarge-GGr LGM
220M
78.0%
CoE Large LGM
220M
79.4%
CoE- CoE-Large SRM SRM
4
220M
79.5%
CoE-Large
220M
79.9%
B.4.4 ABLATION STUDY FOR ELEMENTS OF THE TRAINING STRATEGY
Knowledge distillation (KD), auto augment (AA) and stochastic depth (SD) are widely-used strategies to overcome the overfitting problem. We think only when the overfitting problem is solved can task accuracy reflect model capacity exactly. Because this paper is concerned with improving model
19

---

Under review as a conference paper at ICLR 2022
capacity with limited computation cost, these strategies. Nonetheless, conduct ablations for them in this section. We adopt the CoE-Large setting and use 4 experts. Results are shown in Table 9. We find KD extremely important for CoE, may indicate CoE is easy to be overfitted. In addition, SD decreases the accuracy CoE. By removing SD, CoE- (4 experts) boosts the accuracy from 79.9% to 80.2%. Perhaps, it is because SD makes the capacity of delegator and each expert too tiny (Gontijo-Lopes et al., 2021).
Table 9; Ablation study for each component of training strategy. "KD", "AA" and "SD" denotes "knowledge distillation". "auto-augment" and "stochastic depth" respectively.
KD
AA
SD
Experts
FLOPs
TOP-1 Acc
V
4
220M
79.9%
220M
80.2%
V
V
220M
79.4%
V
V
220M
76.2%
V
220M
79.7%
V
4
220M
76.3%
V
4
220M
75.2%
220M
75.1%
20


Reviews:
[{"review_id": "xvfako2p17E", "reviewer": "Reviewer_SUb5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Let me first say that I like the paper and I think it can be very valuable to the community. It is practical, it performs really well, and achieves surprising results on a very crowded, high-impact hot topic, which is impressive. That said, there are some very important issues with the explanations and even with the experimental comparisons (despite some good range of ablations). My initial score is very likely to change after rebuttal depending on the authors reply.\n\nHere are some of my issues:\n\n- What the WGM, LGM and SRM do is not clear. There are some verbal descriptions, which do not clarify much, and the maths is not very clear either. The WGM (btw weight generation module, the name seems non-informative at best to me, assignment generation module? assignment balancing module?) seems to do something very similar to the LGM. My understanding is that the LGM assigns an expert to each training instance, thus generating the labels that are used to train the delegator. But what is the role of the WGM? is it balancing so that there is no collapse to a single expert? I am not sure I get why the WGM is needed and what it does in practice. \nI think I understand what the SRM module does, but I'm not sure either. Why is it needed? My best guess given the first paragraph of sec 3.3 is that it softens the penalty of delegator mistakes based on how well the \"wrong\" experts do. But then, why use an indirect metric like statistics instead of using the actual loss of the chosen expert?\n\n- Equations: Eq. 10 has a \\mathcal{L}_j,k, but that's not defined before. I assume it is L_j,k as defined in the LGM? What is \\mathcal_P in the first paragraph of Sec 3.4? Not defined or used either. Is it the same as \\mathcal{L}_{total}?\n\n- Sec. 4.1 shows the FLOPs of the networks involved. However, the delegator is 24MB and the expert 110MB. How do we get to 100MB?\n\n- The early termination (early exit?) is virtually not mentioned in the whole paper. There is no explanation regarding whether/how/when it is used, training details, etc. Reading sec. 4.2.3 I recalled it being mentioned in the intro, but at that point I had forgotten about it.\n\n- There is no mention on the paper on weather you would release the code upon acceptance. Given that some parts of the paper/method are hard to parse, that would be very helpful. Is this expected?\n\nThere are other less critical comments: \n- batching: it is mentioned that the proposed method is better at batching than other conditional convolution methods. However, it is still the case that different examples in a batch will go to different experts, thus negatively affecting batching vs. standard networks. Is this correct? \n- I would like to see MobileNetV3-large Table 2 \n- what is the performance without KD? Can you ablate it?\n- Did you ablate the elements on the optimization? I was very surprised at things like stochastic depth and autoaugment given the very tiny capacity fo the network. Are these elements really necessary? (e.g. see tradeoffs in data augmentation: an empirical study, ICLR'21). \n- There are some phrases that are hard to understand. The clarify of the paper would improve revising its English (this is not a factor for scoring, just advice on how to improve impact).\n- Figures are too small, they cannot be read when printed. The only way to see them properly is to zoom (a lot) into the pdf.\n- There are a large number of ablations and supplementary experiments. This thoroughness is appreciated. However, it would be interesting to see an ablation on some of the components of the model - for example, how important are the SRM? and the WGM? and the progressive sharpening of the assignments? etc (I guess this one relates to the KD ablation comment above). I understand this might take resources and time to run and might not fit into a rebuttal, but overall I think it is something missing.\n\nNow, there's plenty of positives with the paper. The performance is excellent on a very important problem (how to do tiny ML effective). The awareness of the literature is superb. There are loads of experiments (CPU latency, vit...). There is solid novelty - I think many people have thought of this way of tackling the problem, but somehow this paper explains how to make it work, so extra kudos. I also like the aggregation of other pieces to build a very solid system (CC, KD, PWLR)... our field tends to focus on one single novelty per paper to maximize clarity (and avoid confusing reviewers?), but then it is very unclear how things stack up and leaves a lot of gap that needs to be filled by the practitioner. I like that this paper bridges the gap and offers a \"ready to go\" model.\n\nSo, all in all, I would really like clarity improved because I think the paper really deserves it. If that happens, I'll happily improve my score.\n", "summary_of_the_paper": "The paper proposes a model for classification (on ImageNet) that involves having a very small network act as a selector and early predictor, and a set of experts that specialise on a subset of the training data. The selector chooses one of them and thus keeps a constant cost irrespective of the number of experts. This is then combined with a number of existing techniques to raise the overall performance. Namely, the authors use a PWRL non-linearity instead of ReLU, they use logit-matching knowledge distillation, and conditional convolution (CondConv). The outcome is a tiny model that yields excellent performance on ImageNet, with performance envelopes superior to existing methods.", "summary_of_the_review": "The paper has a lot of pros and cons. The cons look solvable (clarity mostly), and the pros seem solid enough. The practicality of the approach, the thoroughness of experiments and comparisons, and the excellent performance attained are all very positive. As already mentioned, a good rebuttal that shows clearer explanations will mean I raise the score.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "X_bnViLfnFv", "reviewer": "Reviewer_u2o7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "++ The paper proposes a novel approach to partitioning the entire dataset to individual experts while maintaining diversity.\n\n++ The expert selection is done at the model level, so that the advantage of state-of-the-art efficient networks (OFANet, MobileNetV3) can be taken for small FLOPs and low CPU latency.\n\n++ The proposed CoE is also applied to translation task with BERT and shows generalization ability.\n\n-- Method: The module names are unclear. It makes me hard to understand the role until I read the main text. Furthermore, even if I go through the writing, I am still unsure what is the true difference between selection matrix $ A_{m\\times n} $  and the selection label $ S_{m\\times n} $. Are they the same thing but optimized separately by two different constraints (Eq.1 and 6)? Also, the authors didn't clearly specify how the inference works.\n\n-- Why does the \"Collaboration\" means in the framework \"Collaboration of Experts\"? The selection is one-hot, so there is actually no collaboration between experts?\n\n-- Unclear comparison. \n (1) Table 1 shows CoE-Small/Large with both Conditional Computation (CC) and Knowledge Distillation (KD). I am curious the result of CoE-Small/Large without KD.\n  (2) CoE-Large use OFA-230 as the expert network, so how could CoE-Large achieve smaller FLOPs compared with OFA-230? Is it also including the early termination?\n\n-- Style of writing: A pair of parentheses over all references is preferred so that the the references will not be confused with the main text. I think \"\\citep\" can do this.\n", "summary_of_the_paper": "The paper proposes an efficient mechanism for  image classification. It has a light-weight delegator to yield coarse prediction, where early termination is possible. Once the coarse prediction is not confident enough, it actuates an down-stream expert with higher computational power for refined prediction. Since the expert selection is done across models, the method can take advantages of heavily-tuned efficient neural networks. The so-called CoE framework achieves very competitive results on the ImageNet dataset with low FLOPs and low CPU latency.", "summary_of_the_review": "The paper combines the recent advances of efficient (low FLOPs and low latency) networks and some other techniques such as knowledge distillation, conditional conv (CondConv), and highly-engineering activation function. What is the true novelty is somewhat unclear to me. Also, the methods are not clearly written and how different modules work is confusing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LvnJX0HRrAj", "reviewer": "Reviewer_u2in", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\n1). The propose CoE frames achieves good performance with little computation cost.\n\n2). The design is hardware-friendly with extreme low-latency. \n\n3). Promoting within dataset diversity in the expert selection sounds novel. \n\n\nWeakness:\n\n1). Isn't Eq (1) encourages more randomized selection? What is the motivation of WGM, if it is refined by SRM?\n\n2). 80.0% top-1 accuracy on ImageNet is not hard. Does the design principle can scale well across different (data size, model parameters, FLOPs)?\n\n3). I don't know which of the proposed components contribute the most to the overall performance. What can be concluded from Appendix Figure 7 and Figure 8?\n", "summary_of_the_paper": "This paper proposes the Collaboration of Experts (CoE) framework to both eliminate the need for multiple forward passes and keep hardware-friendly. A training method including weight generation module (WGM), label generation module (LGM) and selection reweighting module (SRM) is proposed to improve the expert selection. Results on ImageNet shows its hardware-friendly performance. ", "summary_of_the_review": "Good results, but lack of ablation study of each proposed components.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EGIt3Ilmhvu", "reviewer": "Reviewer_dWQ7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1.\tThe idea of model collaboration is interesting and new.\n\n2.\tThe model has achieved 80% accuracy within 100M FLOPS, achieving the SoTA performance on ImageNet.\n\nWeakness:\n\n1.\tAbout related work. \nThe paper mentions that the delegator needs to select an expert from the candidate experts and compares to other model selection methods. However, the related work of model selection [A, B] is lacking in the paper. The authors are strongly suggested to enrich the related work part and compare with them.\n\n[A] Ranking Neural Checkpoints. CVPR2021.\n\n[B] LogME: Practical Assessment of Pre-trained Models for Transfer Learning. ICML2021.\n\n2.\tAbout experiments.\n+ As the proposed method contains several networks (including delegator and 4/16 experts), which probably introduces larger memory cost. Can the authors also elaborate on the additional memory cost and compare it with other baseline models in the experimental part.\n+ In Sec. 4.3.1, while the paper mainly targets the model ensemble problem, the authors only compare the method with one simple model ensemble method, which is not very convincing. Therefore, the authors are suggested to conduct a more extensive comparison with other model ensemble methods.\n\n\n======================\n\nPost-rebuttal:\n\nI acknowledge that it for the first introduced an instance-wise model selection framework, which is novel. Most of my concerns about the insufficient experiments have been addressed by the rebuttal. I agree to upgrade the score to 6 and hope the authors can improve the writing and paper readability in the next version. ", "summary_of_the_paper": "In this paper, the author considers the model collaboration problem, specifically ensemble learning. The author identifies two issues of ensemble learning: significant runtime cost and high memory access cost. To alleviate these issues, the authors propose a model collaboration framework named Collaboration of Experts (CoE) along with a training algorithm designed for the framework. The training algorithm contains three components: weight generation module (WGM), label generation module (LGM), and selection reweighting module (SRM). Experiments are conducted on ImageNet to demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "The problem of this paper is interesting, but the experiments are not sufficient to support the claim. Please see the main review for the details.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper trains an expert style DNN that routes input examples to appropriate expert modules resulting in high accuracy on ImageNet with less compute. Reviewers have been positive about the strong empirical results. However the paper itself is not written well and reviewers had hard time figuring out actual architecture and training methodology. For example reviewers couldn't easily figure out the differences between LGM, WGM and SRM. 

The paper itself is sparse on why some of the choices have been made, their relation to existing methods and how do they affect the final performance. For example - In eq2, TCP objective has been normalized for each expert separately with a vague No Superiority Assumption. What motivates this assumption? Why is it reasonable? Eq 4 is quite similar to the load balancing loss in Switch Transformer paper. However there has been no discussion about the similarities and differences.

I think the paper needs to rewritten with clear explanation of the actual architecture, in what aspects it is similar/differs to existing expert models. What key components are the reason for the superior performance?

While I appreciate the authors for the ablations studies they presented during response phase, I think the paper requires major rewriting and cannot recommend acceptance at this stage.
Decision: Reject