Long-tailed Recognition by Routing Diverse Distribution-Aware Experts
Authors: Xudong Wang
Long Lian
Zhongqi Miao
Ziwei Liu
Stella Yu
Venue: ICLR 2021 Spotlight (2021)
Keywords: Long-tailed Recognition
Bias-variance Decomposition

Abstract: Natural data are often long-tail distributed over semantic classes.  Existing recognition methods tackle this imbalanced classification by placing more emphasis on the tail data, through class re-balancing/re-weighting or ensembling over different data groups, resulting in increased tail accuracies but reduced head accuracies.
We take a dynamic view of the training data and provide a principled model bias and variance analysis as the training data fluctuates: Existing long-tail classifiers invariably increase the model variance and the head-tail model bias gap remains large, due to more and larger confusion with hard negatives for the tail.
We propose a new long-tailed classifier called RoutIng Diverse Experts (RIDE).  It reduces the model variance with multiple experts, reduces the model bias with a distribution-aware diversity loss, reduces the computational cost with a dynamic expert routing module.  RIDE outperforms the state-of-the-art by 5% to 7% on CIFAR100-LT, ImageNet-LT and iNaturalist 2018 benchmarks.  It is also a universal framework that is applicable to various backbone networks, long-tailed algorithms and training mechanisms for consistent performance gains. Our code is available at: https://github.com/frank-xwang/RIDE-LongTailRecognition.

Published as a conference paper at ICLR 2021
LONG-TAILED RECOGNITION BY ROUTING DIVERSE DISTRIBUTION-AWARE EXPERTS
Xudong Wang', Long Lian', ZZongqi Miao', Ziwei , X. Yu!
UC Berkeley / ICSI, *Nanyang Technological University {xdwang, Longlin ian,, zhongqi miac, stellayuJsberkeley.coo siwei. edu.
ABSTRACT
Natural data are often long-tail -ail distributed over semantic classes. Existing recog- nition methods tackle this imbalanced classification by placing more emphasis on the tail data, through class re-balancing/re- weighhtin or ensembling over different data groups, resulting in increased tail accuracies but reduced head accuracies.
We take a dynamic view of the training data and provide a principled model bias and variance analysis as the training data fluctuates: Existing long-tail classifiers invariably increase the model variance and the head-tail model bias gap remains large, due to more and larger confusion with hard negatives for the tail.
We propose a new long-tailed classifier called Routing Diverse Experts (RIDE). It reduces the model variance with multiple experts, reduces the model bias with a distribution-awve diversity loss, reduces the computational cost with dynamic expert routing module. RIDE outperforms the state-of-the-arr by 5% to 7% on CIFARIO0-LT, ImageNet-LT and iNaturalist 2018 benchmarks. It is also uni- versal framework that is applicable to various backbone networks, long-tailed al- gorithms, and training mechanisms for consistent performance gains. Our code is available at: httpps//github.ccom Www.aaannnwwg//DDEEEI.... LongaiDDEnnLnngTlleRcoginoooo Teilgeconnt
INTRODUCTION
Real-world data are often long-tail distributed over semantic classes: A few classes contain many instances, whereas most classes contain only few instances. Long-tailed recognition is challenging, as it needs to handle not only a multitude of small-data learning problems on the tail classes, but also extreme imbalanced classification over all the classes.
There are two ways to prevent the many head instances from overwhelming the few tail instances in the classifier training objective: 1) class re-balancing/re-weiehhe
which gives more importance to tail instances (Cao et al. 2019 Kang et al.) 2020 Liu et al. 2019). 2) ensembling over different data distributions which re-organizes long-taile data into groups, trains model per group, and then combines individual models in multi- multi-expert framework (Zhou et al. 2020: Xiang et al., 2020).
We compare three state-of-the-art (SOTA) long-tail classifiers against the standard cross-entropy (CE) classifier: cRT and 7-norm (Kang et al. (2220) which adopt two-stage optimization, first rep- resentation learning and then classification learning, and LDAM (Cao et al., 2019), which trained end-to-end with marginal loss. In terms of the classification accuracy, common metric for model selection on a fixed training set, Fig. T shows that, all these existing long-tail -til methods increase the overall, medium- and few-shot accuracies over CE. but decrease the many-shot accuracy.
These intuitive solutions and their experimental results seem to suggest that there is a head-tail per- formance trade-off in long-tailed recognition. We need principled performance analysis approach that could shed light on such a limitation if it exists and provide guidance on how to overcome
Our insight comes from a dynamic view of the training set: It is merely sample set of some underly- ing data distribution. Instead of evaluating how a long-tailed classifier performs on the fixed training set, evaluate how it performs as the training set fluctuates according to the data distribution.

---

Published as a conference paper at ICLR 2021
All
Many-shot bias
Med-shot
Few-shot
acc
bias
acc
bias
bias
CE
31.6
0.60
0,47
57.3
0.28
0.35
28.2
0.61
0.51
6.3
0.94
0.57
T-norm
35.8
0.52
0.49
55.9
0.28
0.37
33.2
0.53
0.52
16.1
0.78
0.60
CRT
36.4
0.50
0.50
51.3
0.32
0.41
38.6
0.44
0.50
17.0
0.76
0.61
LDAM
34.4
0.53
0.51
55.1
0.28
0.38
31.9
0.53
0.54
13.9
0.81
0.63
RIDE LDAM
40.5
0.50
0.42
60.5
0.28
0,30
38.7
0.50
0.44
20.1
0.74
0.52
(a) Comparisons of the mean accuracy, per-class bias and variance of baselines and our RIDE method. Better (worse) metrics than the cntrrrbutionunnwaan cross entropy (CE) reference are marked in green (red).
Many-shot: 0.09
Medium--hot:: =012
Few-shot: 0.18
trequency
1500
1000
LDAM, 0-2.29
RIDE, 00
frequency
600
800
LDAM, μ 4
RIDE, 0=0.30
frequency
400
300
LDAM, 0.59
RIDE, 4=0.41
750
200
500
250
200
0.0
0.2
1.0
score of the hardest negative
score of the hardest negative
score of the hardest negative
(b) Histograms of the largest softmax score of the other classes (the hardest negative) per instance.
Figure Our method RIDE outperforms SOTA by reducing both model bias and variance. a) These metrics are evaluated over 20 independently trained models, each on a random sampled set of CIFAR100 with an imbalance ratio of 100 and 300 samples for class O. Compared the standard CE classifier, existing SOTA methods almost always increase the variance and some reduce the tail bias at the cost of increasing the head bias. b) The metrics are evaluated over CIFARI 10-LT Liu et al. (2019). LDAM is more likely to confuse the tail (rather than head) classes with the hardest negative class, with an average score of 0.59. RIDE with LDAM can greatly reduce the confusion with the nearest negative class, especially for samples from the few-shot categories.
Consider the training data D as random variable. The prediction error of model h instance with output varies with the realization of D. The expected variance with respect to variable D has well-known birs-variance decomposition:
(1) For the above L2 loss on regression h(x) YY the model bias measures the accuracy of the pre- diction with respect to the true value, the variance measures the stability of the prediction, and the irreducible error measures the precision of the prediction and is irrelevant to the model h.
Error(x: h) = [((h(x; D) - y) Bias(a; + Variance(1;; h) + irreducible error(z).
Empirically, for random sample sets of data, D(J)..., D(x), the k-th model trained on D predicts y()) on instance z, and collectively they have mean prediction ym- For the L2 regression loss, the model bias is simply the I.2 loss between yo and ground-truth t= E(Y,, whereas the model variance is the variance of (4) with respect to their mean ym:
L2 regression loss:
L(yi z) = (3) x)²
(2)
mean prediction:
Um Σg (b)
= aarg min Ep (N(x);
(3)
x=1
model bias:
(ias(x;:h)) h) (ym 1)²
=L (ymit)
(4)
model variance:
Variance h)
(k)
Vm
=Ep|C (h(z))
(5)
k=1
As shown on the above right, these concepts can be expressed entirely in terms of L2 loss L. We can thus extended them to classification (Domingos) 2000) by replacing with Za.1 for classification:
0-1 classification loss:
Lo1(1)) 2) if y = 2, and 1 otherwise.
The mean prediction Fm. minimizes Ee. 1= Co.1 (k); ym) and becomes the most often or main pre- diction. The bias and variance terms become Lo (ymi t) and J Σい La. (M) ym) respectively.
We apply such bias and variance analysis to the CE and long-tail classifiers. We sample CIFAR100 (Krizhevsky 2009) according to long-tail distribution multiple times. For each method, we train
2

---

Published as a conference paper at ICLR 2021
model per long-tai tail sampled dataset and then estimate the per-class bias and variance over these multiple models on the balanced test set of CIFARIO0-LT et al. (2019). Fig. Th shows that:
On the model bias: The head bias is significantly smaller than the tail bias, at 0.3 vs. 0.9 for CE. All the existing long-tail methods reduce the overall bias by primarily reducing the tail bias. However, the head-tail bias gap remains large at 0.3 vs. 0.8.
2. On the model variance: All the existing long-tail methods increase the model variance across all class splits, with a slight reduction in the medium-shot variance for cRT.
That is, existing long-tail methods reduce the model bias for the tail at the cost of increased model variance for all the classes, and the head-tail model bias gap remains large.
We conduct further statistical analysis to understand the head-tail model bias gap. We examine the largest softmax score in the other classes of {c: o t}. where & is the ground-truth class of an instance. The smaller this hardest negative score is, the less the confusion, and the lower the model bias. Fig. D shows that there is increasingly more and larger confusion from the head to the tail.
Guided by our model bias/variance and confusion pattern analysis, we propose a new long-tai tail clas- sifier with four distinctive features: 1) It reduces the model variance for all the classes with multiple experts. 2) It reduces the model bias for the tail with an additional distribution-awar diversity loss. 3) It reduces the computational complexity that comes with multiple experts with a dynamic expert routing module which deploys another trained distinctive expert for a second (or third, ...) opinion only when it is called for. 4) The routing module and shared architecture for experts of reduced complexity effectively cut down the computational cost of our multi-expert model, to a level that could be even lower than the commonly adopted baseline with the same backbone.
Our so-called Routing Diverse Experts (RIDE) not only reduces the model variance for all the classes, but also significantly reduces the model bias for the tail classes and increases the mean accuracies for all class splits, all of which existing long-tail methods fail to accomplish.
RIDE delivers 5%-7% higher accuracies than the current SOTA methods on CIFARIOO-LT, ImageNet-LT (Liu et al., 2019) and iNaturalist (Van Horn et al. 2018). RIDE is also a universal framework that can be applied to different backbone networks for improving existing long-tal tail algo- rithms such as focal loss (Lin et al. 2017). LDAM (Cao et al. 2019), 7-norm (Kang et al. 2020).
2 RELATED WORKS
Few-shot learning. To generalize from small training data, meta-learning (Bertinetto et al. 2016 Ravi & Larochelle| 2017 Santoro et al. 2016 Finn et al. 2017 Yang et al. (018)) and data aug- are two most studied approaches (Chen et al. 2019 Schwartz et al. 2018 Zhang et al. 2019 Liu et al. 2018). Matching Network (Vinyals et al. 2016) and Prototypical Net- work (Snell et al. 2017) learn discriminative features that can be transferred to new classes through meta-learners without big training data. Hariharan & Girshick (017)) Wang et al. (2018) and Liu al. (2018) utilize samples from generative model to augment the training data. However, few- shot learning relies on balanced training data, whereas long-tail recognition has to deal with highly imbalanced training data, e.g., from hundreds in the head to a few instances in the tail.
mmetattiongeeneaatiin
seapuiiinn/r-weighting A direct approach to achieve sample balance is to under- or over- sample training instances according to their class sizes (He & Garcia 2009). Another option is data augmentation, where additional samples are generated to supplement tail classes, sometimes directly in the feature space (Liu et al., 2020 Chu al.) 2020 Kim et al. 2020). Re-weighting
modifies the loss function and puts larger weights on tail classes (Lin et al. [017]] et al. 2019
Cao et al. 2019; Wu et al., 2020) or randomly ignoring gradients from head classes (Tan et al., 2020). However, both sample- wise and loss-wise balancing focus on tail classes, resulting in more sensitivity to fluctuations in the small tail classes and thus much increased model variances (Fig. IT).
Knowledge transfer. OLTR (Liu et al. 2019) and inflated memory (Zhu & Yang 2020) use memory banks to store and transfer mid- and high-leve features from head to tail classes, enhancing feature generalization for the tail. However, this line of work (Liu et al., 2019 Zhu & Yang 2020; Kang ctt al., 2020; Jamal et al. 20200 Wang et al. 2017) usually does not have effective control over the knowledge transfer process, often resulting in head performance loss.
3

---

Published as a conference paper at ICLR 2021
Ensembling and grouping. One way to counter imbalance is separate training instances into different groups based their class sizes. Models trained on individual groups are ensembled together in
framework. BBN (Zhou et al.. 2020) adaptively fuses two-branches that each focus on the head and the tail respectively. LFMH (iiang et al. (020)) distills multiple teacher models into unified model, each teacher focusing on relatively balanced group such many-shot classes, medium-shot classes, and few-shot classes. BBN and LFME still lose head performance and overall generalizability, as no expert has a balanced access to the entire dataset.
Our RIDE is a non-traditional ensemble method. 1) Its experts have shared earlier layers and reduced later channels, less prone to small tail overfitting. 2) Its experts are jointly optimized. 3) It deploys experts on an as as-needed basis for individual instances with dynamic expert assignment module. 4) reaches higher accuracies with a smaller model complexity and computational cost.
3 RIDE: ROUTING DIVERSE DISTRIBUTION-AWAT EXPERTS
We propose a novel multi-expert model (Fig. 2 with shared earlier layers fo and independent channel-reduced later layers V/ fer, .... Vo They are jointly optimized at Stage and dynam- ically deployed with learned expert assignment module at Stage 2. At the inference time, all the m active experts are averaged together in their logits for final ensemble softmax classification:
m
p = softmax
tho (o((o))
Softmax of the average logits is equivalent to the product of individual classification probabilities, which approximates their joint probability if individual experts makes independent decisions.
Experts with shared early backbone and reduced later channels. Consider independent experts of the same convolutional neural network (CNN) architecture. Since early layers of a CNN tend to encode generic low-level features, we adopt the common practice in transfer learning and have all the n experts share the same backbone fe. Each expert retains independent later layers the i = 1.....77 .n. To reduce overfitting to small training data in the tail classes, we reduce the number of filter channels in via e.g., by 1/4, All these n experts are trained together on long-tailed data distribution-awanl diversity loss DD-Diversity and classification loss CCassitys such as CE and LDAM.
Individual expert classification loss. One way to combine multiple experts to apply the classifi- cation loss to the aggregated logits of individual experts. While this idea works for several recently proposed multi-expert models (Zhou et al., 2020) Xiang et al. 2020), it does not work for our shared experts: Its performance is on-par with equal-sizze sized expeett model. Let L denote the classi-
Stage 1: Jointly Optimize Diverse Distribution-wwrre Experts
b) Stage 2: Routing Diverse Experts
c)) RIDE vs. SOTA
Expert
improved ace. reletive CECK)
SoTó
RICE
Ditrributionnaare diversity
dlaza Indox
10.3
Expert22
Lp Deverall
Vily
Routing
Classification Eciosity
onlott
loss Epuatina
Expert
trainable
Vee.
Mettrratie
frozen
omibff
Figure 2: RIDE learns experts and their router in two stages. a) We first jointly optimize multiple experts with individual classification losses and mutual distribution- aware diversity losses. b) We then train a router that dynamically assigns ambiguous samples to additional experts on an as-nedded basis. The distribution of instances seen by each expert shows that head instances need fewer experts and the imbalance berween classes gets reduced for later experts. At the test time, we collect the logits of assigned experts to make final decision. e) RIDE outperforms SOTA methods (i.e. LFME (Xiang et al. 2020) for CIFAR 100-LT, LWS (Kang et al. 2020) for ImageNet-LT and BBN Zhou et al. 2020 for iNaturalist) on all the benchmarks.
4

---

Published as a conference paper at ICLR 2021
fication loss (e.g. CB) over instance and its label y. We call such aggregation loss collaborative:
Coollaborative (x, v) L ba (fo(z)) .2
as it leads to correlated instead of complementary experts. To discourage correlation, require each expert to do the job well by itself. Such an aggregation loss is essentially an individual loss, and it contributes large portion of our performance gain in most of our experiments:
Cindiividual (x.1) =
Σ c (we. (fe(z))
Distribution-aware diversity loss. The individual classification loss and random initialization lead to diversified experts with shared backbone. For long-tailed data, we add a regularization term to encourage complementary decisions from multiple experts. That is, we maximize the KL-divergence between different experts' classification probabilities on instance x class y over total of classes:
diversity loss:
DD-Diversat (z, 0,)
-1
DKL (p') (lll))lp (10)
n
デ=1
KL divergence:
(pl(p)
Σ P log
Pk
(11)
9A
classification by 0,
p ( (x, = softmax
[Va Ti
ste (fa(m))
(12)
We vary the temperature T (or concentration) (Hadsell et al. 2006 Wu et al. 2018) applied to class k's logit the (fo(z))= For class k with n, instances, the smaller the nuk, the lower the temperature Tk. the more sensitive the classification probability p is to change in the feature zb. Specifically,
class-wise temperature:
Tg = BK - 1 max (3)
(13)
5
normalized class size:
8x = I E n2
+ (11
(14)
T scales linearly with the class size, ensuring 3a -, 1, = a for balanced set. This simple adaptation allows us find classifiers of enough complexity for the head and enough robustness for the tail: On one hand, we need strong classifiers to handle large sample variations within head classes; on the other hand, such classifiers are prone to overfit small training data within tail classes. We adapt the temperature only after the CNN network is trained for several epochs and the feature is stabilized, similar to the training scheme for deferred reweighting (Cao et al. 2019).
Joint expert optimization. For experts 01, .... 6n with shared backbone 8, optimize their individual classification losses (Cclassify = C. any classification loss such as CE, LDAM, and focal loss) and their mutual distribution-aware divergency losses, weighted by hyperparameter X:
Lroeal (x, 8) Ccusstty (x, y; 0:) + X LD Liveersistty (z (z)):0)))
(15)
Since these loss terms are completely symmetrical with respect to each other, the n experts learned Stage are equally good and distinctive from each other.
Routing diversified experts. To cut down the rest-time computational cost that comes with multiple experts, we train router at Stage 2 to deploy these (arbitrarily ordered) experts sequentially on an as-needed basis. Assume that the first i experts have been deployed for instance ar The router takes in the image feature and the mean logits from the first to the k-th expert, and makes a binary decision Mon on whether to deploy the k + 1-th expert. If the k-th expert wrongly classifies æ, but one of the rest n experts correctly classifies æ,, ideally the router should switch on, i.e., output you = 1, and otherwise fou = O. We construct simple binary classifier with two fully connected layers to learn each router. Each of the n p-1 routers for n experts has a shared component to reduce the feature dimensions and an individual component make decisions.
5

---

Published as a conference paper at ICLR 2021
Specifically, we normalize the image feature (z) (for training stability), reduce the feature dimen- sion (to e.g. 16 in our experiments) by a fully connected layer W1 which is shared with all routers, followed by ReLU and flattening, concatenate with the top-s ranked mean logits from the first to k- th expert & Σ vhe (fo(z)), project to a scalar by w! which independent between routers, and finally apply Sigmoid function S(x) = I4e- to get a continuous activation value in [0,1]:
W flatten ReLU (Wi
So(m) [[fo(())
router activation: r(x)= S
(16)
E Vo, (fo(z))
Hop lop soppsocoomonntts
The router has a negligible size and compute, where 8 ranges from 30 for CIFARIO0 to 50 for iNaturalist (8,142 classes). It is optimized with weighted variant of binary CE loss:
Louutnng (r(2), yon) = -ion Yon log (r(=:)
- Yon) log (1 11--(on r(z))
(17)
where wan controls the easiness to switch on the router. We find won = 100 to be a good trade-off between classification accuracy and computational cost for all our experiments. At the test time, we simply threshold the activation with 0.5: If r(x) < 0.5, the classifier makes the final decision with the current collective logits. otherwise it proceeds the next expert.
Optional delf-distillation. While existing long-tail classifiers such as BBN Zhou et al. 22200) and LFME Xiang al.) (2020) have a fixed number of experts, our method could have an arbitrary number of experts to balance classification accuracy and computation cost. We can optionally apply self-distillation from a model with more (6 in our setting) experts to the same model with fewer experts for further performance gain (0.4%~~0.8% for most experiments). We choose knowledge distillation (Hinton et al., 2015) by default. Implementation details and comparisons with various distillation algorithms such CRD Tian et al. 2019) are investigated in Appendix Section A.2
4 EXPERIMENTS
We experiment major long-tailed recognition benchmarks and various backbone networks.
1. CIFARI00-LT (Cao et al.
2019): CIFARIO0 100 is sampled by class per exponential decay across
classes. We choose imbalance factor 100 and ResNet-32 (He et al. 2016) backbone.
ImageNet-LT (Liu et al. 2019): Multiple backbone networks are experimented on ImageNet- LT, including ResNet- 10, Re and ResNeXt-50 (Xie al. 2017). All backbone networks are trained with a batch size of 256 on 8 RTX 2080Ti GPUs for 100 epochs using SGD with an initial learning rate of 0.1 decayed by 0.1 60 epochs and 80 epochs. See more details and results on other backbones in Appendix.
3. iNaturalist 2018 (Van Horn et al.) 20188:: is naturally imbalanced fine-grained dataset with 8, 142 categories. We use ResNet-50 as the backbone and apply the same training recipe as for ImageNet-LT except batch size 512, as in (Kang et al. 2020
CIFARI00-LT Results. Table | shows that RIDE outperforms SOTA by large margin on CIFAR L.. The average computational cost even about 10% less than baseline models with two experts as in BBN. RIDE surpasses multi-expert methods, LFME (Xiang et al.] 2020) and BBN (Zhou et al. 2020), by more than 5.3% and 6.5 respectively.
50.0
(5OTA)
48,7
48.7
48.6
49.2
Figure 3: RIDE is universal framework that can be extended to various long-tail recognition methods and obtain consistent top-1 accuracy increase. RIDE is experimented on CIFAR 100- LT and applied to various training mechanisms. By using RIDE, cross-entropy loss (without any re-balancing strategies) can even outperforms previous SOTA method on CIFAR100-LT. Al- though higher accuracy can be obtained using distillation, we did not apply it here.
baseline
RIDE baselne
Accuracy (96)
43.2
43.1
43.8
42.5
30.4
39.1
35.0
CE
IDAM
AR
Cocine
Focal
Mechanisms

---

Published as a conference paper at ICLR 2021
Table 1: RIDE achieves the state-of-the-ait results on CIFARI00-LT without sacrificing the per- formance of many-shot classes like all previous methods. Compared with BBN (Zhou et al. 2020) and LFME (Xiang et al. 2020), which also contain multiple experts (or branches), RIDE (2 experts) outperforms them by large margin with fewer GFlops. The relative computation cost (averaged on testing set) with respect to the baseline model and absolute improvements against SOTA (colored in green) are reported. j denotes our reproduced results with released code. { denotes results copied
from Cao et al. 2019) and the imbalance ratio is 100.
Methods
MFlops
All
Many
Med
Few
Cross Entropy (CE)
69.5 (I.0x)
38.3
Cross Entropy (CE)
69.5 (1.0x)
39.1
66.1
37.3
10.6
Focal Loss Lin et al. 2017)
69.5 (1.0x)
38.4
OLTR i (Liu et al.
2019)
41.2
61.8
41,4
17.6
LDAM + DRW (Cao el al.
2019
69.5 (1.0x)
42.0
LDAM DRW (Cao al. 2019)
69.5 (1.0x)
42.0
61.5
41.7
20.2
BBN (Zhou et al.
2020)
74.3 (1.1x)
42.6
T-norm (Kang et al.
2020
69.5 (1.0x)
43.2
65.7
43.6
17.3
CRT i (Kang et al. (2020)
69.5 (1.0x)
43.3
64.0
44.8
18.1
M2m Kum al.
2020)
43.5
LFME (Xiang et al. (0200)
43.8
RIDE (2 experts)
64.8 (0.9x)
47.0 (+3.2)
67.9
48.4
21.8
RIDE (3 experts)
77.8 (1.1x)
48.0 (+4.2)
68.1
49.2
23.9
RIDE (4 experts)
91.9 (1.3x)
49.1 (+5.3)
69.3
49.3
26.0
Table 2: RIDE achieves state-of-the-art results on ImageNet-IT (Liu al. 2019) and obtains
consistent performance improvements various backbones. The top- accuracy and computational cost are compared with the state-of-the-arr methods on ImageNet-LT, with ResNet-50 and ResNeXt- 50 as the backbone networks. Results marked with i copied from (Kang et al. 2020 Detailed results on each split are listed appendix materials.
ResNet-50
ResNeXt-50
Methods
GFlops
Acc. (%)
GFlops
Acc. (%)
Cross Entropy (CE)
4.11 (1.0x)
41.6
4.26 (1.0x)
44.4
OLTR + (Liu et al. 2019
46.3
NCM Kang al.
2020
4.11 (1.0x)
44.3
4.26 (1.0x)
47.3
T-norm (Kang et a.. 2020)
4.11 (1.0x)
46.7
4.26 (1.0x)
49.4
cRT Kang et al.
2020)
4.11 (1.0x)
47.3
4.26 (1.0x)
49.6
LWS KKng et al.
2020)
4.11 (1.0x)
47.7
4.26 (1.0x)
49.9
RIDE (2 experts)
3.71 (0.9x)
54.4 (+6.7)
3,92 (0.9x)
55.9 (+6.0)
RIDE (3 experts)
4.36 (1.1x)
54.9 (+7.2)
4.69 (1.1x)
56.4 (+6.5)
RIDE (4 experts)
5.15 (1.3x)
55.4 (47.7)
5.19 (1.2x)
56.8 (+6.9)
RIDE as universal framework. Fig. 3 shows that RIDE consistently benefits from better loss functions and training processes. Whether the model is trained end-to-end (focal loss, CE, LDAM) or in two stages (cRT, T-norm, cosine), RIDE delivers consistent accuracy gains. In particular, RIDE with a simple cosine classifier, which we constructed by normalizing the classifier weights and retraining them with long-tail tail re-sampling strategy (similar to cRT), achieves on- on-par performance with the current SOTA methods. Fig. Fig-3aals, alaso shows that two-stage methods are generally better than single-stage ones. Nevertheless, since they require additional training stage, for simplicity, use the single-stage LDAM as the default Lcassity in RIDE throughout our remaining experiments.
ImageNet-LT Results. Table 2 shows that RIDE outperforms SOTA, LWS and cRT, by more than 7.7% with 5esNet-50. ResNeXt-50 is based on group convolution (Xie et al. 2017), which di- vides all filters into several groups and aggregates information from multiple groups Rt-5e generally performs better than ResNet-50 on multiple tasks. It provides 6.9% gain on ImageNet-LT.
iNaturalist 2018 Results. Table shows that RIDE outperforms current SOTA 6.3%. Surpris- ingly, RIDE obtains very similar results on many-shots, medium-shots and few-shots, ideal for long tailed recognition. Current SOTA method BBN also uses multiple experts; however, it significantly
7

---

Published as a conference paper at ICLR 2021
Table 3: RIDE outperforms previous state-of-the-art methods on challenging iNaturalist 2018 Van Horn et al. 2018) dataset, which contains 8,142 classes, by large margin. Relative im- provements SOTA result of each split (colored with gray) are also listed, with the largest boost from few-shot classes. Compared with previous SOTA method BBN, which also contains multi- ple "experts", RIDE achieves more than 20% higher top-1 accuracy on many-shot classes. Results marked with are from BBN (Zhou et al. 2020) and Decouple (Kang et al. 2020). BBN's results are from the released checkpoint.
Methods
GFlops
Many
Medium
Few
CE
4.14 (1.0x)
61.7
72.2
63.0
57.2
CB-Focal
4.14 (1.0x)
61.1
OLTR
4.14 (1.0x)
63.9
59.0
64.1
64.9
LDAM DRW f
(1.0x)
64.6
cRT
41.40(.0x)
65.2
69.0
66.0
63.2
(1.0x)
65.6
65.6
65.3
65.9
LWS
4.36 (1.0x
65.9
65.0
66.3
65.5
BBN
66.3
49.4
70.8
65.3
RIDE (2 experts)
3.67 (0.9x)
71.4 (+5.1)
70.2 (+1.2)
71.3 (+0.5)
71.7 (+5.8)
RIDE (3 experts)
4.17 (1.0x)
72.2 (+5.9)
70.2 (+1.2)
72.2 (+1.4)
72.7 (+6.8)
RIDE (4 experts)
4.51 (1.1x)
72.6 (+6.3)
70.9 (+1.9)
72.4 (+1.6)
73.1 (+7.2)
Accurazy Diferenoi (80
RIDE vs.CE
Instances per cluss
Acciracy Diffrrnnce 196
50
1250
30
BBN
800
30-
RIDE vs. cRT
1000
20
600
400
10-
500
200
250
1000
2000
3000
4000
5000
6000
7000
8000
200
400
600
800
Class to fall)
(Head tail)
Figure 4: Compared to SOTAs, RIDE improves top-1 accuracy on all three splits (many-/med-/feww shot). The absolute accuracy differences of RIDE (blue) over iNaturalist's current state-of-the- art method BBN (Zhou et al. 2020) (left) and ImageNet-LTTs current state-of-the-at method cRT (Kang al., (2020) (right) are shown. RIDE improves the performance of few- and medium-shee shotts categories without sacrificing the accuracy many-shots, and outperforms BBN on many-shots by a large margin.
decreases the performance on many-shots about 23%. RIDE remarkable increasing the few-shot accuracy without reducing the many-shot accuracy.
Comparing with SOTAs iNaturalist and ImageNet-LT. As illustrated in Fig. 4 approach provides a comprehensive treatment to all the many-shot, mediumm-hoe and few-shot classes, achiev- ing substantial improvements to current state-of-the-an on all aspects. Compared with cRT which reduces the performance on the many-shot classes, RIDE can achieves significantly better perfor- mance on the few-shot classes without impairing the many-shot classes. Similar observations can be obtained in the comparison with the state-of-the oftthe--rttttt method BBN (Zhou et al. 2020) on iNaturalist.
Contribution of each component of RIDE. RIDE is jointly trained with LD-Divrrsity and Ccissiiyy we use LDAM for LClasitty by default. Table shows that the architectural change from the original ResNet-32 the RIDE variant with 2 experts contributes 2.7% 4.3% gain. Applying the individual classification loss instead of the collaborative loss brings 1.5% gain. Adding the diversity loss further improves about 0.9%. The computational cost is greatly reduced by adding the dynamic expert router. Knowledge distillation from RIDE with 6 experts obtains another 0.6% gain. All these components deliver 7. % gain over baseline LDAM.
Impact of the number of experts. Fig. 5 shows that whether in terms of relative absolute gains, few-shots benefit more with more experts. For example, the relative gain is 16% vs. 3.8% for few-shots and mhny-shots respectively. No distillation is applied in this comparison.
8

---

Published as a conference paper at ICLR 2021
Table 4: Ablation studies on the effectiveness of each component on CIFARTOO-LT. LDAM used as our classification loss. The first 3 RIDE models only have architectural change without changes in training method. The performance without Lindividull checked indicates directly applying classification loss onto the final model output, which is the mean expert logits. This is referred to as collaborative loss above. In contrast, Lindividua if checked, we apply individual loss to each individual expert. The difference between collaborative loss and individual loss is described above. By adding the router module, the computational cost of RIDE can be significantly reduced, while the accuracy degradation is negligible. Knowledge distillation step is optional if further improvements are desired. Various knowledge distillation techniques are compared the appendix.
Methods
#expert
Individual
D--Dverssly
Router
distill
GFlops
Acc. (%)
LDAM + DRW
42.0
2
1.1x
44.7 (+2.7)
3
1.5x
46.1 (+4.1)
1.8x
46.3 (+4.3)
RIDE
4
1.8x
47.8 (+5.8)
1.8x
48.7 (+6.7)
4
V
1.8x
49.3 (+7.3)
1.3x
49.1 (47.1)
6,7%, absollte: 3.1
relative:
2.5
Many
Medium
Few
top-1 sccuracy (%)
(60 Cecannn T-do
# #xppert = 3
23.0%
34.7%
41.5%
RIDE (NI
RIDE (Nany shot)
67.3%
number of experts
number of experts
top-2 accuracy (96)
relative: 5,4%, absolute: 2.6
(96)
relative:
# expert =
21.0%
F-do Souun
28.7%
52.8%
9.2%
3.9%
3.9%
65. 55.9%
9.5%
RIDE (Medium-shot)
RIDE (Few-shot)
number
number experts
eesperts
experts
experts
Figure 5: # experts vs. top-1 accuracy for each split (All, Many/Mediun/few) of CIFAR100- LT. Compared with the many-shot split, which is 3,8% relatively improved by adding more perts, the few-shot split can get more benefits, that is, relative improvement of 16.1%.
Figure 6: The proportion of the number of ex- perts allocated to each split of CIFAR 100-LT. For RIDE with 3 or 4 experts, more than half of expert. On the contrary, more than 76% samples of few-shot classes require opinions from additional experts.
many-shoe instances only require
The number of experts allocated each split. Fig. @Jshows that instances in few-shots need more experts whereas most instances in many-shots just need the first expert. That is, low confidence in tail instances often requires the model seek a second (or third, ...) opinion.
SUMMARY
We take a dynamic view of training data and study long-tailed recognition with model bias and variance analysis. Existing long-tail classifiers do not reduce head-tail model bias gap enough while increasing model variance across all the classes. We propose a novel multi-expert model called RIDE to reduce model biases and variances throughout. It trains partially shared diverse distribution- aware experts and routes an instance to additional experts when necessary, with computational costs comparable to single expert. RIDE outperforms SOTA by large margin. It is also universal framework that works with various backbones and training schemes for consistent gains.
Acknowledgments. This work was supported, in part, by Berkeley Deep Drive, US Goverment fund through Etegent Technologies on Low-Shot Detection and Semi-supervised Detection, and NTU NAP and S via Industry Alignment Fund: Industry Collaboration Projects Grant.

---

Published as a conference paper at ICLR 2021
REFERENCES
Luca Bertinetto, João F Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learning feed- forward one-shot learners. In NeurlPS, 2016.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with labcl-distriuution--weee margin loss. In Advances in Neural Information Processing Systems, pp. 1567-1578, 2019.
Zitian Chen, Yanwei Fu, Yu-Xiong Wang, Lin Ma, Wei Liu, and Martial Hebert. Image deformation meta- neeta-networks for one-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8680-8689, 2019.
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling, Feature space augmentation for long-tailed data. arXiv preprint arXiv:2008. 03673, 2020.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song. and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 01677,977, 2019.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2009.
Pedro Domingos. A unified viri-nari decomposition. In Proceedings of 17th International Conference on Machine Learning, pp. 231-238, 2000.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint
2017.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 1735-1742, 2006.
Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3018- 3027, 2017.
Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions knowledge and data engineering, 21(9):1263-1284 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503 0253312 2015.
Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsuan Yang, Liqiang Wang, and Boqing Gong. Rethinking class-balanced methods for long-tailed visual recognition from domain adaptation perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 7610-7619, 2020.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Learning imbalanced datasets with bibblldistribution-waarr margin loss. Interna- tional Conference on Learning Representations, pp. 1567-1578, 2020.
Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major-to- minor translation. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13896-13900 13905, 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
10

---

Published as a conference paper at ICLR 2021
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017.
Liu, Xudong Wang, Mandar Dixit, Roland Kwitt, and Nuno Vasconcelos. Feature space transfer for data augmentation. In Proceedings of the IEEE conference computer vision and patlern recognition, pp. 9090-9098, 2018.
Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. Deep representation learning on long-tailed data: learnable embedding augmentation perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2970-2979, 2020.
Ziwei Liu, Zhongqi Miao. Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE/CVF Conference Computer Viston and Pattern Recognition, pp.
2016.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. Proceedings of the European Conference on Computer Vision, pp. 268-284, 2018.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICLR, 2017.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmenter neural networks. In ICML, 2016.
Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Rogerio Feris, Ab- hishek Kumar, Raja Giryes, and Alex M Bronstein. Delta-encoder: an effective sample synthesis method for few-shot object recognition. arXiv preprint arXiv:1806.04734 2018.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurlPS, 2017.
Jingru Tan, Changbao Wang. Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11662-11677 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Inter- national Conference Learning Representations, 2019.
Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1365-1374, 2019.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun. Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The iNaturalist species classification and detection dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 8769-8778, 2018
Oriol Vinyals, Charles Blundell, Tim Lillicrap, and Daan Wierstra. Matching networks for one shot learning. NeurIPS, 2016.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In Advances in Neural Information Processing Systems, pp. 7029-7039, 2017.
Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pp. 7278-7286, 2018
Tong Wu, Qingqiu Huang, Ziwei Liu, Yu Wang, and Dahua Lin. Intributionnbalancee loss for multi-label classification in long-tailed datasets. In Proceedings of the European Conference on Computer Vision pp. 162-178. Springer, 2020.
11

---

Published as a conference paper at ICLR 2021
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3733-3333 37742, 2018.
Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self-paced knowl- edge distillation for long-tailed classification. In Proceedings of the European Conference on Computer Vision, pp. 247-263. Springer, 2020.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1492-1500, 2017.
Flood Sung Yongxin Yang. Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning compare: Relation network for few-shot learning. In CVPR, 2018.
Hongguang Zhang, Jing Zhang, and Piotr Koniusz. Few-shot learning via saliency-guided halluci- nation of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2770-2777 2019.
Xiao Zhang, Zhiyuan Fang, Yandong Wen, Zhifeng Li, and Yu Qiao. Range loss for deep face recognition with long-tailed training data. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5409-5418, 2017.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. BBN: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEF/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 9719-9728, 2020.
Linchao Zhu and Yi Yang. Inflated episodic memory with region self-attention for long-tailed vi- sual recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4344 4353, 2020.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 DATASETS AND IMPLEMENTATIONS
We conduct experiments on three major long-tailed recognition benchmarks and different backbone networks to prove the effectiveness and universality of RIDE:
I.IEAAR10-LLT (Krizhevsky 2009 Cao et al. 2019): The original version of CIFAR- 100 con- tains 50,000 images on training set and 10,000 images on validation set with 100 categories. The long-tailed version of CIFAR- 100 follows exponential decay in sample sizes across different cat- egories. We conduct experiment on CIFAR100-LL with an imbalance factor of 100, i.e. the ratio between the most frequent class and the least frequent class.
To make fair comparison with previous works, we follow the training recipe of (Cao al. 2019) on
CIFAR 100-LT. We train the ResNet-32 (He et al. 2016) backbone network by SGD optimizer with momentum of 0.9. CIFAR 100-LT is trained for 200 epochs with standard data augmentations (He) et al. 2016) and a batch size of 128 on one RTX 2080Ti GPU. The learning rate is initialized as 0.1 and decayed by 0.01 at epoch 120 and 160 respectively.
ImaageNet-LT (Deng al. 2009 Liu et al. 2019): ImageNet-LT is constructed by sampling subset of ImageNet-2012 following the Pareto distribution with the power value = 6 (Liu et al.) 2019). ImageNet-LT consists of 115.8k images from 1, 1,000 categories, with the largest and smallest categories containing 1,280 and 5 images, respectively.
Multiple backbone networks are experimented on ImageNet-LT, including ResNet-10, ResNet-50 and ResNeXt-50 (Xie et al. 2017). All backbone networks are trained with a batch size of 256 on 8 RTX 2080Ti GPUs for 100 epochs using SGD with an initial learning rate of 0.1 decayed by 0.1 at 60 epochs and 80 epochs. We utilize standard data augmentations as in (He et al. 2016).
3. S.iauurallst (Van Horn et al. 2018): The iNaturalist22111 dataset is an imbalanced datasets with 437,513 training images from 8,142 classes with balanced test set of 24,426 images. We use ResNet-50 as the backbone network and apply the same training recipe as ImageNet-LT, except that we use a batch size of 512.
A.2 ADDITIONAL EXPERIMENTS
Ablation study on distillation methods. Self-distillation step is optional but recommended if fur- ther improvements (0.4%~-0.8% for most experiments) are desired. We apply distillation from a more powerful model with more experts into a model with fewer experts. A simple way to transfer knowledge is knowledge distillation (KD) (Hinton et al. 2015). which applies KD loss (LRD 2 DKL (huecherr TT, Texpen, /T) to match the distribution of logits of teacher and student. We found that for teacher model with more experts using smaller distillation loss factor gives better performance. We hypothesize that since distill from the same teachers, giving large distillation factor prevents the branches from becoming as diversified as it is able to. We also explored other distillation methods, such CRD (Tian al. 2019), PKT (Passalis & Tefas 2018), and SP (Tung & Mori, 2019), and compared the differences in Table 5 Although adding other methods along with KD may boost performance, the difference small. Therefore, we opt for simplicity and use KD only unless otherwise noticed.
Detailed results for ImageNet-L.T experiments. We list details of our ResNet-50 experiments in ImageNet-LT on Table [ With 2 experts, we are able to achieve about 7% gain in accuracy with computational cost about 10% less than baseline. contrast to previous methods that sacrifice many-shot accuracy to get few-shot accuracy, we improve on all three splits on ImageNet-LT. From 3 experts to 4 experts, we keep the same many-shot accuracy while increasing the sew-shot accuracy, indicating that we are using the additional computational power to improve the hardest part of the data rather than uniformly applying to all samples.
We also list our ResNet- 10 and RtsNeXt-50 experiments on Table and respectively, compare against other works evaluated on these backbones. Our method also achieves lower computational cost and higher performance when compared other methods.
Comparison with ensemble method. Since our method requires the joint decision from several experts, which raw ensembles also do, we also compare against ensembles of LDAM in Fig 7 on
13

---

Published as a conference paper at ICLR 2021
Table 5: Comparison of different distillation methods. We transfer from a model based on ResNet-32 with 6 experts to model of the same type, except with fewer experts. We use CIFAR LT for the following comparison. No expert assignment module is used in the following experi- ments. Following the procedure for CRD (Tian et al. 2019), we also apply KD when we transfer from a teacher to students with other distillation methods.
Model Type
#expert
Distillation Method
Accuracy (%)
Teacher
49.7
No Distillation
46.6
2
KD (Hinton al.
2015
47.3
2
CRD
(Tian et al.
2019
47.5
2
PKT
Passalis & Tetas
2018
47.2
2
SP
Tung & Mon
2019)
47.2
Distillaiinn
47.9
3
KD
(Hinton et al. 2015
48.4
Student
3
CRD
(Tian et al.
2019
48.5
3
PKT
Paassalis & Telas
2018
48.3
3
SP
Tung & Mori
2019)
48.7
4
Distillatoo
48.7
KD
(Hinton et al.
2015
49.3
4
CRD)
(T'an et 2019)
49.0
4
PKT (Passalıs & Tetas
2018
48.9
4
SP (Tung Mon 2019)
49.0
RIDE and Ensembles of LDAM
49
Figure 7: Comparison between our method and multiple LDAM models ensembled together. In the figure, ensembles of LDAM start from en- semble (original LDAM) to 7 ensembles, and RIDE starts from 2 experts to 4 experts. Our method achieves higher accuracy with substan- tially less computational cost compared to semble method.
48
8.0
47
Accuracy
46
$5.0
43
RIDE
42
42.0
Ersembles LDAM
200
300
MFlops
10. In the figure, even our method with 4 experts has less computational cost than the minimum computational cost for the ensemble of 2 LDAM models. This indicates that our model much more efficient and powerful in terms of computational cost and accuracy than ensemble on long-tailed datasets.
Ours
head
apple
tail sunflower
streatcar
lamp
tail closs: sunfower
bicycle
sunflower
trout
Isl clsse man head class: apple
tail class: man
Figure 8: t-SNE visualization of LDAM's s and our model's 's embedding space of CIFAR100-LT. The feature embedding of RIDE more compact for both head and tail classes and better separated. This behavior greatly reduces the difficulty for the classifier to distinguish the tail category.
t-SNE visualization. We also provide the t-SNE visualization of embedding space on CIFAR100- LT as in Fig. 8 Compared with the baseline method LDAM, the feature embedding of RIDE is more compact for both the head and tail classes and better separated from the neighboring classes. This greatly reduces the difficulty for the classifier to distinguish the tail category.
14

---

Published as a conference paper at ICLR 2021
What if apply RIDE to balanced datasets? We also conducted experiments CIFARIO0 to check our method can achieve similar performance gains on balanced datasets. However, we only obtained improvement of about 1,, which is much smaller than the improvements observed on the CIFAR 100-LT. Compared with balanced datasets, long-tailed datasets can get more benefits from RIDE.
Table 6: Top-1 accuracy comparison with state-of-the-art methods on ImageNet-LT (Liu et al, 2019) with ResNet-10. Performance Many-shot (> 100), Medum-shot ($ 100 & >00) and Few-
shot (=20) are also provided. Results marked with are copied from (Liu et al. 2019) Results with $ from (Xiang al. 2020
Methods
GFlops
Many
Medium
Few
Overall
Cross Entropy (CE)
0.89 (1.0x)
40.9
10.7
0.4
20.9
Focal Loss f Lin et 2017)
0.89 (1.0x)
36.4
29.9
16.0
30.5
Range Loss Zhang et al. 2017
0.89 (1.0x)
35.8
30.3
17.6
30.7
Lifted Loss f Oh Song cf al. 2016
0.89 (1.0x)
35.8
30.4
17.9
30.8
OLTR (Liu et al.
2019
0.89 (1.0x)
43.2
35.1
18.5
35,6
LFME Xiang et (020)
47.0
37.9
19.2
38.8
Many-shot only
59.3
Medium-shot only $
35.9
Few-shot only
14.3
RIDE (2 experts)
0.85 (1.0x)
57.5
40.8
26.9
45.3 (+6.5)
RIDE (3 experts)
0.97 (1.1x)
57.6
41.7
28.0
45,9 (+7.1)
RIDE (4 experts)
1.07 (1.2x)
58.5
42.4
27.7
46.6 (+7.8)
Table 7: Top-1 accuracy comparison with state-of-the-art methods on ImageNet-LT (Liu et al. 2019) with ResNet-50. Performance on Many-shot (> 100), Medum-shot ($ (500 & 200 and Few-
shot (520) are also provided. Results marked with f are copied from (Kang et 2020
Methods
GFlops
Many
Medium
Few
Overall
Cross Entropy (CE)
4.11 (1.0x)
64.0
33.8
5.8
41.6
NCM Kang ct al. 2020)
53.1
42.3
26.5
44.3
cRT Kang al. 2020)
4.11 (1.0x)
58.8
44.0
26.1
47.3
7-norm (Kang et al. 2020)
4.11 (1.0x)
56.6
44.2
27.4
46.7
LWS (Kang et 2020)
4.11 (1.0x)
57.1
45.2
29.3
47.7
RIDE (2 experts)
3.71 (0.9x)
65.8
51.0
34.6
54.4 (+6.7)
RIDE (3 experts)
4.36 (1.1x)
66.2
51.7
34.9
54.9 (+7.2)
RIDE (4 experts)
5.15 (1.3x)
66.2
52.3
36.5
55.4 (+7.7)
Table 8: Top-1 accuracy comparison with state-of-the-art methods ImageNet-LT (Liu et al.
2019) with ResNeXt-50. Performance on Many-shot (> 100), Medum-shot (< 15000 & >20) and
Few-shot (520) are also provided. Results marked with are copied from (Kang al. 2020
Methods
GFlops
Many
Medium
Few
Overall
Cross Entropy (CE)
4.26 (1.0x)
65.9
37.5
7.7
44.4
NCM Kang al. 2020)
56.6
45.3
28.1
47.3
cRT Kang et al. 2020)
4.26 (1.0x)
61.8
46.2
27.4
49.6
T-norm Kang et al. 2020)
4.26 (1.0x)
59.1
46.9
30.7
49.4
LWS (Kang et al. 2020)
4.26 (1.0x)
60.2
47.2
30.3
49.9
RIDE (2 experts)
3.92 (0.9x)
67.6
52.5
35.0
55.9 (+6.0)
RIDE (3 experts)
4.69 (1.1x)
67.6
53.5
35.9
56.4 (+6.5)
RIDE (4 experts)
5.19 (1.2x)
68.2
53.8
36.0
56.8 (46.9)
15


Reviews:
[{"review_id": "xQ3exjjeMkG", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method termed RoutIng Diverse Experts (RIDE) for reducing both the bias and the variance of a long-tailed classifier. Specifically, RIDE consists of three crucial components: 1) a shared architecture for multiple experts; 2) a distribution-aware diversity loss that encourages more diverse decisions for classes with fewer training instances; 3) an expert routing module that dynamically assigns more ambiguous instances to additional experts. Experiments are conducted on three long-tailed benchmark datasets, i.e., CIFAR100-LT, ImageNet-LT, and iNaturalist. Satisfactory classification results of long-tailed visual recognition are observed.\n\nPaper strengths:\n- The problem, i.e., long-tailed visual recognition, is practical, important and challenging in computer vision and deserves further studies.\n- The proposed method has good motivations and sounds reasonable.\n- The experimental results of the proposed RIDE are significantly better than the results of previous work, which shows the effectiveness of RIDE.\n- The paper is well written and easy to follow.\n- Analyses and ablation studies are sufficient and could bring new insights of long-tailed recognition.\n\nPaper weaknesses:\n- There are typos and grammar mistakes in this paper. For example, in the caption of Fig. 2, \"ImaeNet-LT and BBN\" should be \"ImageNet-LT and BBN\". The authors should carefully proofread the final version.\n- Some references have inconsistent reference formats, e.g, \"In Proceedings of the IEEE conference on computer vision\nand pattern recognition\" of [Xie et al., 2017] vs. \"In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition\" of [Zhou et al., 2020] vs. \"CVPR09\" of [Deng et al., 2009]. In addition, some other references lack detailed information, e.g., [He and Garcia, 2009] lacks page, volume information. Some references also have capital issues, e.g., \"The inaturalist species classification\" -> \"The iNaturalist species classification\" of [Van Horn et al., 2018], and \"Bbn\" -> \"BBN\" of [Zhou et al., 2020].", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z1aWTXn-_ew", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis paper proposes a Routing Diverse Experts (RIDE) framework to solve the long-tailed classification problem. It has 1) a shared low-level feature extractor and multiple expert classifiers, 2) a distribution-aware diversity loss to encourage experts learning different classification strategies, 3) an expert routing module that dynamically selects a subset of experts for each test instance to make a joint decision. This paper firstly increases the performances on all three splits (many-/med-/few-shot), while most of the existing methods have to sacrifice the head for tail improvements.\n\n##########################################################################\n\nPros:\n\n+ The proposed Bias & Variance metrics provide a better understanding of why previous work almost always improves the tail accuracy but decreases head performance(trade-off between bias and var).\n+ To the best of my knowledge, this paper firstly increases the performances on both head and tail.\n+ The proposed RIDE framework can be applied to a variety of long-tailed recognition methods.\n+ Although RIDE includes multiple experts, each expert only has 1/4 channels, which allows overall computation overhead even decreases when using 2 experts.\n\n##########################################################################\n\nConcerns:\n\nI'm overall satisfied with this paper, but I do have a few concerns:\n- The proposed RIDE does look like an ensemble method to me. Although you provide a comparison with the ensemble method in the appendix, I noticed that 2 ensembles will achieve 45.0 accuracies on CIFAR100-LT, which is exactly the same as 2 experts + L_diversity in Table 4. Although RIDE can achieve better performance when the number of ensemble models/experts is increased to 3 or 4, does it means that diversity loss only works on a larger number of experts?\n\n- The expert routing/assignment module dynamically routes cascaded experts, and all the passed experts before the stop will jointly make a decision. How to make sure the early experts are the more robust ones? Since experts have already been frozen when the routing module is trained, and the diversity loss won't force the early experts to be more robust. There is no guarantee that the involved early experts are more useful than the skipped ones.\n\n##########################################################################\n\nQuestions during the rebuttal period: \n\nPlease address and clarify the concerns above. Besides, I think the experts in RIDE may learn to be biased toward different categories due to the diversity loss, and the routing module works as an outlier detector which only stops when it finds a proper expert biased toward the underline ground-truth labels. Is it another way to understand RIDE? or do I misunderstand some parts of our paper?\n\n##########################################################################\n\nReasons for score: \n\nOverall, I vote for accepting. I think this paper shows a new direction for long-tailed classification that we don't need to sacrifice the head performance for a balanced classifier anymore. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "YVyW-r8ie9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a simple but effective method which significantly outperforms the state-of-the-art method by 5% to 7%. The experiments are adequate and rigorous. Besides these , the writing is clear and easy to understand. Totally, this paper is a good work\nPros:\n1. The writing is clear.\n2. The view of bias and variance is novel and interesting.\n3. The experiments are adequate and rigorous.\n4. The performance of proposed method is very strong.\n5. The proposed method is simple, effective and general.\nCons:\n1. The procedure of test is not clear enough. For example, how to turn on/off an expert? By a threshold?\n2. The description of the part about shared, indepent, and recuded extractor is not clear enough.", "extra_scores": {}, "extra_text": {}}, {"review_id": "T42TXqleET", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The majority of feature extraction backbone is shared among different agents and the classifiers of experts are trained with both classification loss and proposed distribution-aware diversity loss. For the second stage, an expert assignment module is trained to re-weight the expert decisions. The whole paper is generally well-organized. However, there are some technical issues authors should further address:\n1) In the introduction, the authors measure the mean accuracy, model bias and model variance by randomly sampling the Cifar100 a few times, and reported different methods in Fig1. The details of how such bias and variance is computed are not given. For example, it is stated the bias also measures the accuracy.\n2) For the motivation of distribution-aware diversity loss, if the same input batch is observed by all the experts, why is it 'distribution aware'? Also, why requires the diversity among experts if setting the lower temperature to tail classes is to encourage the divergent prediction, how both diversity and divergent exists simultaneously?\n3) Is the input batch randomly sampled from the entire training set?\n4) For experiments on ImageNet-LT and iNaturalist, is the RIDE combined with the regular CE loss or other methods?\n5) Can authors demonstrate some details regarding the performance of expert assignment module during the test, as this is the core module in the proposed framework? For example, for each split (many, low) how the different expert behaves?\n\n[Post Rebuttal Comments] Authors have done a good job for addressing my concerns, especially the additional ablation studies regarding the performance of the expert assignment module. I'm updating my score accordingly for recommending acceptance.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the important problem of classification with unbalanced semantic classes. The key idea is a two-stage process that first learns a representation under various distributions (experts), then assign a cascade of experts to hard samples, whose predictions are combined. The approach can be added on top of various backbone networks. Experiments are systematic and extensive, showing improvement on three standard benchmarks for this task.

All four reviewers recommended accept.  The paper extends a recent research direction of learning "balanced" representations, followed by distribution-aware experts. This general approach could have wider impact on architectures designed for out-of-distribution and low-shot learning.  The authors should update the final paper based on their answers and on reviewer feedback. We also encourage authors to make their code available. 
Decision: Accept (Spotlight)