Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognition
Authors: Yifan Zhang
Bryan Hooi
Lanqing HONG
Jiashi Feng
Venue: NeurIPS 2022 Accept (2022)
Keywords: Long-tail Recognition
Class Distribution Shifts

Abstract: Existing long-tailed recognition methods, aiming to train class-balanced models from long-tailed data, generally assume the models would be evaluated on the uniform test class distribution. However, practical test class distributions often violate this assumption (e.g., being either long-tailed or even inversely long-tailed), which may lead existing methods to fail in real applications. In this paper, we study a more practical yet challenging task, called test-agnostic long-tailed recognition, where the training class distribution is long-tailed while the test class distribution is agnostic and not necessarily uniform. In addition to the issue of class imbalance, this task poses another challenge: the class distribution shift between the training and test data is unknown. To tackle this task, we propose a novel approach, called Self-supervised Aggregation of Diverse Experts, which consists of two strategies: (i) a new skill-diverse expert learning strategy that trains multiple experts from a single and stationary long-tailed dataset to separately handle different class distributions; (ii) a novel test-time expert aggregation strategy that leverages self-supervision to aggregate the learned multiple experts for handling unknown test class distributions. We theoretically show that our self-supervised strategy has a provable ability to simulate test-agnostic class distributions. Promising empirical results demonstrate the effectiveness of our method on both vanilla and test-agnostic long-tailed recognition. The source code is available at https://github.com/Vanint/SADE-AgnosticLT. 

Self-Supervised Aggregation of Diverse Experts for gest-Agnostic Long-Tailes Recognition
Yifan Zhang'
Bryan Hooi Lanqing Hong? Jiashi Feng?
National University of Singapore yifan. zhanguu. nus edu, jshfengegmail.co com.
Huuawii Noah's Ark Lab *ByteDance
Abstract
Existing long-tailed recognition methods, aiming to train class-balanced models from long-tailed data, generally assume the models would be evaluated on the uniform test class distribution. However, practical test class distributions often violate this assumption (e.g,, being either long-tailed even inversely long-tailed), which may lead existing methods to fail in real applications. In this paper, we study more practical yet challenging task, called test-agnostic long-tailed recognition, where the training class distribution is long-tailed while the test class distribution is agnostic and not necessarily uniform. In addition the issue class imbalance, this task poses another challenge: the class distribution shift between the training and test data unknown. To tackle this task, we propose a novel approach, called Self- superviede Aggregation of Diverse Experts, which consists of two strategies: (i) new skill-diverse expert learning strategy that trains multiple experts from a single and stationary long-tailed dataset to separately handle different class distributions; (ii) a novel test-time expert aggregation strategy that leverages self-supervision to aggregate the learned multiple experts for handling unknown test class distributions. We theoretically show that our self-supervised strategy has provable ability to simulate test-agnostic class distributions. Promising empirical results demonstrate the effectiveness of our method on both vanilla and test-agnostic long-tailed recognition. The source code is available at https: wwt.l/ccc////aiiiaeeee /wt//t/nnnnnttttteeeeeeeaee waw.aint/SSDD-AAnntt.tt.
//githuss com
Introduction
Real-world visual recognition datasets typically exhibit long-tailed distribution, where a few classes contain numerous samples (called head classes), but the others are associated with only a few instances (called tail classes) [24, 33]. Due to class imbalance, the trained model is easily biased towards head classes and perform poorly on tail classes [2, 58]. To tackle this issue, numerous studies have explored long-tailed recognition for learning well-performing models from imbalanced data [20, 56].
Most existing long-tain loig-tailed studies [3, 9, 10, 48, 52] assume the test class distribution is uniform, i.e., each class has an equal amount of test data. Therefore, they develop various techniques, egg. class re- sampling [13, 18, 25, 55], cost-sensitive learning [1 36, 41, 47] or ensemble learning [2, 13, 27, 53], re-balance the model performance on different classes for fitting the uniform class distribution. However, this assumption does not always hold in real applications, where actual test data may follow any kind of class distribution, being either uniform, long-tailed, or even inversely long-tailed to the training data (cf. Figure ((a).. For example, one may train recognition model for autonomous cars based on the training data collected from city areas, where pedestrians are majority classes and stone obstacles are minority classes. However, when the model is deployed to mountain areas, the pedestrians become the minority while the stones become the majority. this case, the test class distribution is inverse to the training one, and existing methods may perform poorly.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Training Class
Trained Model
Training Distribations
Distributions
dime
Expert
Aggregation
Test Class
Multi-epeert
Test Class
Dicribations
Model
Distributions
(a) Existing long-tailed recognition methods
(b) The idea of our proposed method
Figure I: Illustration of test-agnostic long-tailed recognition. (a) Existing long-tailed recognition methods aim to train models that perform well test data with the uniform class distribution. However, the resulting models may fail to handle practical test class distributions that skew arbitrarily. (b) Our method seeks learn a multi-expert model from a single long-tailed training set, where different experts are skilled in handling different class distributions, respectively. By reasonably aggregating these experts at test time, our method is able handle unknown test class distributions.
To address the issue of varying class distributions, as the first research attempt, LADE [17] assumes the test class distribution to be known and uses the knowledge to post-adjust model predictions. However, the actual test class distribution is usually unknown priori, making LADE not applicable in practice. Therefore, we study a more realistic yet challenging problem. namely test-agnostic long-tailed recognition, where the training class distribution long-tailed while the test distribution is agnostic. To tackle this problem, motivated by the idea of "divide and conquer", we propose to learn multiple experts with diverse skills that excel at handling different class distributions (cf. Figure ((b).. As long as these skill-diverse experts can be aggregated suitably at test time, the multi-expert model would manage to handle the unknown test class distribution. Following this idea, we develop a novel approach, namely Self Aggregation of Diverse Experts (SADE).
The first challenge for SADE is how to learn multiple diverse experts from single and stationary long-tailes tailed training dataset. To handle this challenge, we empirically evaluate existing long-taile oang-ailedd methods in this task, and find that the models trained by existing methods have a simulation correlation between the learned class distribution and the training loss function. That is, the models learned by various losses are skilled in handling class distributions with different skewness. For example, the model trained with the conventional softmax loss simulates the long-tailed training class distribution, while the models obtained from existing long-tailed methods are good at the uniform class distribution. Inspired by this finding, SADE presents a simple but effective skill-diverss expert learning strategy to generate experts with different distribution preferences from single long-tailed training distribution. Here, various experts are trained with different expertise-guided objective functions to deal with different class distributions, respectively. As a result, the learned experts are more diverse than previous multi-expert long-tailed methods [49, 63], leading to better ensemble performance, and in aggregate simulate a wide spectrum of possible class distributions.
The other challenge is how to aggregate these skill-diverse experts for handling test-agnostic class distributions based on only unlabeled test data. To tackle this challenge, we empirically investigate the property of different experts, and observe that there is a positive correlation between expertise and prediction stability, i.e., stronger experts have higher prediction consistency between different perturbed views of samples from their favorable classes. Motivated by this finding, we develop a novel self-supervised strategy, namely prediction stability maximization, to adaptively aggregate experts based on only unlabeled test data. We theoretically show that maximizing the prediction stability enables SADE to Iearn an aggregation weight that maximizes the mutual information between the predicted label distribution and the true class distribution. In this way, the resulting model is able to simulate unknown test class distributions.
We empirically verify the superiority of SADE on both vanilla and test-agnostic long-tailed recogni- tion. Specifically, SADE achieves promising performance on vanilla long- recognition under all benchmark datasets. For instance, SADE achieves 58.8% accuracy on ImageNet-LT with more than 2% accuracy gain over previous state-of-the-art ensemble long-tailed methods, i.e. RIDE [49] and ACE [2]. More importantly, SADE is the first long-tailed approach that is able to handle various test-agnostic class distributions without knowing the true class distribution of test data in advance. Note that SADE even outperforms LADE [17] that uses knowledge of the test class distribution.

---

Compared to previous long- tailet methods (e.g., LADE [17] and RIDE [49]), our method offers the following advantages: (i) SADE does not assume the test class distribution be known, and provides the first practical approach to handling test-agnostic long-tailed recognition; (ii) SADE develops a simple
strategy to learn skill-diverse experts from a single and stationary long-tailed dataset; (iii) SADE presents a novel self-supervised strategy to aggregate skill-diverse experts at test time, by maximizing prediction consistency between unlabeled test samples' perturbed views; (iv) the presented self-supervised strategy has provable ability to simulate test-agnostic class distributions, which opens the opportunity for tackling unknown class distribution shifts at test time.
2 Related Work
Long-tailed recognition Existing long-tailed recognition methods, related to our study, can be categorized into three types: class re- balancing. logit adjustment and ensemble learning. Specifically, class re-balancing resorts to re-sampling [4, 13, 18, 25] or cost-sensitive learning [3, 10, 16, 61] to balance different classes during model training. Logit adjustment [17, 33, 37, 43] adjusts models' output logits via the label frequencies of training data at inference time, for obtaining large relative margin between head and tail classes. Ensemble-based methods [2, 13, 53, 63], e.g., RIDE [49] are based on multiple experts, which seek to capture heterogeneous knowledge, followed by ensemble aggregation. More discussions on the difference between our method and RIDE [49] are provided in Appendix D.3. Regarding test-agnostic long-tailed recognition, LADE [17] assumee the test class distribution is available and uses it to post-adjust model predictions. However, the true test class distribution is usually unknown priori, making LADE inapplicable. In contrast, our method does not rely on the true test distribution for handling this problem, but presents a novel self-supervised strategy to aggregate skill-diverse experts at test time for test-agnostic class distributions. Moreover, some ensemble-base long-tailed methods [39] aggregate experts based on a labeled uniform validation set. However, as the test class distribution could be different from the validation one, simply aggregating experts on the validation set is unable to handle test-agnostic long-tailed recognition.
Test-time training Test-time training [23, 26, 30, 40, 46] is transductive learning paradigm for handling distribution shifts [28, 32, 34, 38, 45, 59] between training and test data, and has been applied with success to out-of-domain generalization [19, 35] and dynamic scene deblurring [6]. In this study, we explore this paradigm to handle test-agnostic long-tailed recognition, where the issue of class distribution shifts is the main challenge. However, most existing test-time training methods seek to handle covariate distribution shifts instead of class distribution shifts, so simply leveraging them cannot resolve test-agnostic long-tailed tailed recognition, shown in experiment (cf. Table 9).
3 Problem Formulation
Long-tailed recognition aims to learn a well-performing classification model from a training dataset with aangttilled class distribution. Let D.= yı 12=1 denote the long-tailed training set, where y; the class label of the sample xy. The total number of training data over C' classes n, n.=EE Ei= ni. where n4 denotes the number of samples in class k. Without loss of generality, we follow a common assumption [17, 25] that the classes are sorted by cardinality in decreasing order (i.e., if 01 < i2, then n,, 2 n;,), and n1 >> ne. The imbalance ratio is defined as max(n;)) (/mi///nnm/n)) = ny/nc. The test data D. {zy. y3 }3= is defined in a similar way.
Most existing long-tailed recognition methods assume the test class distribution is uniform (i.e.. Pr(y)= 1/C), and seek train models from the long-tailed training distribution (v( to perform well on the uniform test distribution. However, such an assumption does not always hold in practice. The actual test class distribution in real-world applications may also be long-tailed (i.e., Pe(y) p. (y)), or even inversely long-tailes tailed to the training data (i.e., Pe()) = inv inv(p.y))) Here, inv(-) indicates that the order of the long tail on classes is flipped. As result, the models learned by existing methods may fail when the actual test class distribution is different from the assumed one. To address this, we propose to study a more practical yet challenging long-tailed problem, i.e., Test-agnostic Long-tailed Recognition. This task aims to learn a recognition model from long-tailed training data, where the resulting model would be evaluated on multiple test sets that follow different class distributions. This task is challenging due to the integration of two challenges: (1) the severe class imbalance the training data makes it difficult to train models; (2) unknown class distribution shifts between training and test data (i.e., Po()) # (2)) makes models hard to generalize.
3

---

Table I: Accuracy of existing long-tailed (LT) methods on ImageNet-LT with various test class distributions, including uniform, forward and backward LT distributions with imbalance ratios of 10 and 50, respectively. The results show that each method strives to simulate specific class distribution terms of many-shot medium-shon and few-shot classes, which does not change when the test class distribution varies. The corresponding visualization results are reported in Figure 5 in Appendix D.4.
Saftmax
Balanced Softmax [21]
LADE wio prior [17)
Text class distrihution
Many
Medium
Fow
Many
Medium
Fow
Many
Medium
Few
Forward-LT-50
67.5
41.7
14.0
63.5
47.8
37.5
63.5
46.4
35.1
Forward-IT-10 IT-10
68.2
40.9
14.0
64.1
48.2
31.2
64.7
47.1
32.2
Uniform
Backward-LT-D
67.4
41.9
13.9
63.4
49.1
33.6
64.4
48.2
34,2
Backward-LT-50
70.9
13.8
66.5
48.4
33.2
66.3
47.8
Method
To tackle the above problem, inspired by the idea of "divide and conquer", we propose to learn multiple skill-diverse experts that excel at handling different class distributions. By reasonably fusing these experts at test time, the multi-expert model would manage to handle unknown class distribution shifts and resolve test-agnostic long-tailed recognition. Following this idea, we develop novel Self-supervised Aggregation of Diverse Experts (SADE) approach. Specifically, SADE consists of two innovative strategies: (1) learning skill-diverse experts from a single long- aaigttiled training dataset; (2) time aggregating experts with supervviinv to handle test-agnostic class distributions.
4.1 Skill-diverse Expert Learning
As shown in Figure 2, SADE builds three-expert model that comprises two components: (1) an expert-shared backbone fe: (2) independent expert networks E1. Eg and Ez. When training the model, the key challenge is how to learn kiile-dieerse experts from a single and stationary long-tailed training dataset. Existing ensemble- uniform test class distribution, and hence the trained experts are not differentiated sufficiently for handling various class distributions (refer to Table 6 for an example). To tackle this challenge, we first empirically investigate existing long-tailed methods this task. From Table we find that there simulation correlation between the learned class distribution and the training loss function, That is, the models learned by different losses are good at dealing with class distributions with different skewness. For instance, the model trained with the softmax loss is good at the long-tailed distribution, while the models obtained from long-tailed methods are skilled in the uniform distribution.
long-tailed methods [13, 49] seek to train experts for the
Motivated by this finding, we develop a simple skill-diverse expert learning strategy generate experts with different distribution preferences. To be specific, the forward expert Fj seeks to be good at the long-tailed class distribution and per- forms well on many-shot classes. The uniform expert E2 strives to be skilled in the uniform dis- tribution. The backward expert E3 aims at the inversely long-tailed distribution and performs well on few-shot classes. Here, the forward and backward experts necessary since they span a wide spectrum of possible class distribu- tions, while the uniform expert ensures retaining high accuracy on the uniform distribution. To three different sxpertise-guided
Softmax Loss fre
Es
Ralanced Softmax Loss Lpal
Shared Backbone
Softmax
Loss Aya
E
Experts
Logits
Figure The scheme of SADE with three experts, where different experts are trained with different expertise-guiie guided losses.
this end, losses to train the three experts, respectively.
The forward expert E, We use the softmax cross-entropy loss to train this expert, so that it directly simulates the original long-tailed training class distribution:
Lee
Σ
- (v1g)(()
(1)
Th
J.ED.
where (-) () is the output logits of the forward expert ET, and o()) (-) is the softmax function.

---

The uniform expert Eg We aim to train this expert to simulate the uniform class distribution. Inspired by the effectiveness of logit adjusted losses for long- recognition [33], we resort to
the balanced softmax loss [21]. Specifically, let ef Ens: exp(u=) be the prediction probability.
exp(u")
The balanced softmax adjusts the prediction probability by compensating for the long-tailed class expfe* xogg=)))))
distribution with the prior of training label frequencies: i/ > T exp("e) >e exp(ue exp(o=+logmr)
exp(e*)
where x² as denotes the training label frequency of class k. Then, given (-))) (-) as the output logits of the expert Eg, the balanced softmax loss for the expert Eg is defined as:
Lbal
Σ The mED,
-y log (1) + log
Intuitively, by adjusting logits to compensate for the long-tailed distribution with the prior и, this loss enables Eg to output glass-balanced predictions that simulate the uniform distribution.
The backward expert Eg We seek to train this expert to simulate the inversely long-tailed class distribution. To this end, we propose new inverse softmax loss, based the same ra- tionale of logit adjusted losses [21, 33]. Specifically, we adjust the prediction probability by: exp(e"+oog +log llg
gd Σوك exp(v=+log -er--lor log mey. where the inverse training prior obtained by inverting the order of training label frequencies Then, the new inverse softmax loss for the expert E3 is defined as:
Dine 1 The R.ED.
Σ
-b log (Uy((,) - Aog
where u-(-) denotes the output logits of Es and Å is hyper-parametee Intuitively, this loss adjusts logits to compensate for the long-tailed distribution with æ, and further applics reverse adjustment with म. This enables E3 to simulate the inversely long-tailed distribution (cf. Table 6 for verification).
4.2 Test-time Self-supervised Aggregation
Based on the skill-diverse learning strategy, the three experts in SADE are skilled in different class distributions. The remaining challenge is how to fuse them to deal with unknown test class distributions. A basic principle for expert aggregation is that the experts should play bigger role in situations where they have expertise. Nevertheless, how to detect strong experts for unknown test class distribution remains unknown. Our key insight is that strong experts should be more stable in predicting the samples from their skilled classes, even though these samples are perturbed.
Empirical observation To verify this hypothesis, estimate the prediction stability of experts by comparing the cosine similarity between their predictions for a sample's 's two augmented views. Here, the data views are generated by the data augmentation techniques in MoCo v2 [5]. From Table we find that there is a positive correlation between expertise and prediction stability, ie., stronger experts have higher prediction similarity between different views of samples from their favorable classes. Following this finding, we propose to explore the relative prediction stability to detect strong experts and weight experts for the unknown test class distribution. Consequently, we develop a novel self-supervised strategy, namely prediction stability maximization.
Prediction stability maximization This strategy learns aggregation weights for experts (with frozen parameters) by maximizing model prediction stability for unlabeled test samples. As shown in Figure 3, the method comprises three major components as follows.
Data view generation For a given sample x, we conduct two stochastic data augmentations to generate the sample's 's two views, i.e., x² and x² Here, we use the same augmentation techniques as the advanced contrastive learning method, i.e., MoCo v2 [5], which has been shown effective in self-supervised learning.
Learnable aggregation weight Given the output logits of three experts (2), U2, U2, E RxC we aggregate experts with learnable aggregation weight [wi, w2, wa] E RR3 and obtain the final softmax prediction by ý = a (wn Un + w2-02 + W3 Wz where w is normalized before aggregation, i.e., W1 w2 + w3=1.
5

---

Table 2: Prediction stability of experts in terms of the cosine similarity between their predictions of a sample's 's two views. Note that expert E1 good many-shot classes and expert E3 skilled in few-shot classes. The experts tend to have better prediction consistency for the samples from their skilled classes. Here, the imbalance ratio CIFARI 100-LT 100.
Maximize prediction stability
Logits
1
10
[-
Cosine similarity between view prodictions
Model
ImageNet-LT
CIFARIOD-LT
Figure 3: The scheme of test-time self-supervised aggregation. Two data augmentations sampled from the same family augmentations (t~T and t'~T) are applied to obtain two data views.
Many
Med.
Few
Mony
Med.
Few
Expert
0.48
0.28
Export Ez
0.56
0.50
0.45
0.25
0.21
0.19
Expert 0.52
0.53
0.58
0,22
0.23
0.25
Objective function Given the view predictions of unlabeled test data, we maximize the prediction stability based on the cosine similarity between the view predictions:
max S, where S 74
1
Σ
to
øED;
Here, g' and g2 are normalized by the softmax function. In test-time training. only the aggregation weight w is updated. Since stronger experts have higher prediction similarity for their skilled classes, maximizing the prediction stability would learn higher weights for stronger experts regarding the unknown test class distribution. Moreover, the supersuue aggregation strategy be conducted in an online manner for streaming test data. The pseudo-code of SADE is provided in Appendix B.
Theoretical Analysis We then theoretically analyze the prediction stability maximization strategy to conceptually understand why it works. To this end, we first define the random variables of predictions and labels as Y ~p(()) and Y~pr(u)- We have the following result:
Theorem 1. The prediction stability S is positive proportional the mutual information between the predicted label distribution and the test class distribution I(Y; Y), and negative proportional to the prediction entropy HÝ)):
S I(V.V) - H(().
Please refer Appendix A for proofs. According to Theorem 1, maximizing the prediction stability S enables SADE to learn aggregation weight that maximizes the mutual information between predicted label distribution p(ù) and the test class distribution (r)),, as well as minimizing the prediction entropy. Since minimizing entropy helps to improve the confidence of the classifier output [12], the aggregation weight is learned to simulate the test class distribution Pa (y) and increase the prediction confidence. This property intuitively explains why method has the potential to tackle the challenging task of test-agnostic long-tailed recognition at test time.
Experiments
In this section, first evaluate the superiority of SADE both vanilla and test-agnostic long-tailed recognition. We then verify the effectiveness of SADE terms of its two strategies, i.e., skill-diverse expert learning and test-time self-supervised aggregation. More ablation studies are reported in appendices. Here, we begin with the experimental settings.
5.1 Experimental Setups
Datasets We use four benchmark datasets (i.e., ImageNet-LT [31], CIFARIO0-LT [3], Places- LT [31], and iNaturalist 2018 [44]) to simulate real-world long-tailed class distributions. Their data statistics and imbalance ratios are summarized in Appendix C.I. The imbalance ratio is defined as max ny/min ny. where n, denotes the data number of class j. Note that CIFAR 100-I.T has three variants with different imbalance ratios.

---

Table 3: Top-1 accuracy CIFAR 100-LT, Places-LT and iNaturalist 2018, where the test class distribution is uniform. More results on three class sub-group groups are reported in Appendix D.I.
(a) CIFARIOO-LT
(b) Places-LT
(c) iNaturalist 2018
Imholance Ratio
10
50
100
Method
Top I accuracy
Method
Top accuracy
Softmax
59.1
45.6
4144
Soltmax
31.4
Sollmax
64.7
[63]
59.8
49.3
Causal [42]
32.2
Causal [42]
64.4
Causal [42]
59.4
48.8
45.0
Balanced Softmax [21]
39.4
Balanced Softmax [21]
70.6
LADE [17]
Balanced Softmax (21)
MISLAS (62]
61.0
62.5
61.6
50.9
50.1
46.8
46.1
45.6
LADE [17]
MISLAS [62]
38.3
39.2
LADE [17]
MiSLAS [62]
70.7
69.3
RIDE [19]
61.8
51.7
48.0
RIDE [49]
40.3
RIDE
71.8
SADE (ours)
63.6
53,9
49.8
SADE (ours)
40.9
SADE (ours)
72.9
We compare SADE with state-of-the-ee long-tailed methods, including two-stage methods (Decouple [25], MiSLAS [62]), logit-adjuster training (Balanced Softmax [21], LADE [17]), ensemble learning (BBN [63]. ACE [2], RIDE [49]), classifier design (Causal [42]), and representation learning (PaCo I. Note that LADE uses the prior of test class distribution for post- adjttaddustment (although it is unavailable in practice), while all other methods do not use this prior.
Baselines
Evaluation protocols In test-agnostic long-tailed recognition, following LADE [17], the models are evaluated on multiple sets of test data that follow different class distributions, in terms of micro accuracy. Same as LADE [17], we construct three kinds of test class distributions, i.e., the uniform distribution, forward taiget distributions training data, and backward long-tailed distributions. In the backward ones, the order of the long tail on classes is flipped. More details of test data construction are provided in Appendix C.2. Besides, we also evaluate methods on vanilla long-taiie tailed recognition [25, 31]. where the models are evaluated on the uniform test class distribution. Here, the accuracy three class sub-groups is also reported, i.e., many-shot classes (more than 100 training images), medium-shot classes (20~100 images) and few-shot classes (less than 20 images).
Implementation details We use the same setup for all the baselines and our method. Specifically, following [17, 49], we use ResNeXt-50 for ImageNet-LTD ResNet-32 for CIFAR 100-LT, ResNet- 152 for Places-LT and ResNet-50 for iNaturalist 2018 as backbones, respectively. Moreover, we adopt the cosine classifier for prediction on all datasets. If not specified, we use the SGD optimizer with the momentum of 0.9 for training 200 epochs and set the initial learning rate 0.1 with linear decay. We set 2=2 for ImageNet-LT and CIFAR test-time training, we train the aggregation weights for 5 epochs with the batch size 128, where use the same optimizer and learning rate as the training phase. More implementation details and the hyper-parameter statistics reported in Appendix C.3.
and x=1 for the remaining datasets. During
5.2 Superiority on Vanilla Long-tailed Recognition
This subsection compares SADE with state-of-the-arr long-tailed methods vanilla sailet recognition. Specifically, as shown in Tables 3-4, Softmax trains the model with only cross-entropy, it simulates the long- tailed training distribution and performs well on many- shot classes. However, it performs poorly medium- shot and few-shot classes, leading to worse overall per- formance. In contrast, existing long-tailed methods (e.g., Decouple, Causal) seek to simulate the uniform class dis- tribution, so their performance is more class-balanced, leading to better overall performance. However, as these methods mainly seek balanced performance, they in- evitably sacrifice the performance on many-shot classes. To address this. RIDE and ACE explore ensemble learning for long-tailed recognition and achieve better performance on tail classes without sacrificing the head-class performance. In comparison, based on the increasing expert diversity derived from skill-diverse expert learning, our method performs the best on all datasets, e-g., with more than 2% accuracy gain on ImageNet-LT compared to RIDE and ACE. These results demonstrate the superiority of SADE over the compared methods that are particularly designed for the uniform test class distribution. Note that SADE also outperforms baselines in experiments with stronger data augmentation (i.e., RandAugment [7]) and other architectures, as reported in Appendix D.I.
Table 4: Top-1 accuracy on ImageNet-LT.
Method
Many
Med.
Few
Soltma
68.1
41.5
14.0
48.0
Decouple-LWW LWS [25]
61.8
47.6
30,9
Causal [42]
64.1
45.8
27.2
50.3
Balanced Saftmax [2]]
64.1
48.2
33.4
52.3
MiSLAS [62]
62.0
49.1
32.8
$1.4
LADE [17]
64.4
47.7
34,3
52,3
PaCa [8]
63.2
39,2
ACE 12]
71.7
54.6
23,5
$6.6
RIDE [49]
68.0
52.9
35.1
56.3
SADE (ours)
66.5
57.0
43.5
58.8
7

---

Table 5: Top-1 accuracy on long-taiil datasets with various unknown test class distributions. "Prior" indicates that the test class distribution is used as prior knowledge. "Uni." denotes the uniform distribution. "IR" indicates the imbalance ratio. "BS" denotes the balanced softmax [21].
IrageNNe-111
CIBARIOOLLLYY (IL100)
Methol
Polo
Forwent-L
Un..
BackwanALL
Ulii
HascooniALL
10
25
25
10
10
Seftaa
05.1
03.8
08.5
56.6
$2.0
48.0
43.9
38.6
34.9
305
27.6
66.3
62.0
56.2
52.5
464
41.4
36.5
33.5
25.8
21.7
17.5
57.3
46.1
MiSLAS
58.
492
41.
35.5
573
55.
31.7
32.1
LADE
03.4
02.1
50.
57.4
14.6
52.3
48.9
46.3
46.9
423
407
56.0
55.5
53.6
51.9
45.0
45.6
43.3
400
383
353
540
LADE
053
638
08.
57.5
54.5
53.3
50.4
48.6
48.6
496
29.2
02.6
103
15.6
52.7
46.3
456
43.8
41.1
41.5
417
416
640
56.3
54.0
51.0
46.
599
57.9
48.0
35.1
35.4
SABI
694
07.4
68.4
63B
58.3
$7.1
54.5
532
SM.1
68.9
62.
3K.3
56.8
$1.3
44.7
419
42.5
42.4
Placs LT
(Neturaiii 2015
Methol
Pilor
ForwentLLE
Una.
Backwond-LY
Usi.
Hascsani-Ll
25
Seftma
05.6
62.2
48.2
36.0
16.1
31.4
28.4
25.4
25.
23.8
1944
6S.4
65.3
64.1
04.0
63.4
42.7
41.3
41.0
39.4
38.5
36.2
10.3
-0.8
7036
10.3
LADE
42.8
41.5
41.2
40.
198
99.2
17.5
16.9
360
35.3
68.4
00.3
696
69.5
LADE
66.3
64.2
42.2
41.2
16.7
99.4
39.2
36.9
40.9
414
43.0
x
68.1
09.3
7003
41.
41.6
42.0
41.0
40.3
39.6
16.1
31.0
35.9
71.5
71,9
71.
71.9
21.8
SAIR
46.4
44.
43.5
42.6
41.3
40,9
60.6
41.1
41.4
420
416
123
72.9
738
73.3
Table 6: Performance of each expert on the uniform test distribution, where the imbalance ratio of CIFARIO0-LT is 100. The results show that our proposed method learns multiple experts with higher skill diversity, which leads better ensemble performance.
SADE (nars)
Model
ImareNet
CIPARIOO LT
ImageNer LT
CIFARIOD
Many
Med.
Few
Many
Med.
Few
All
Many
Med.
Few
All
Many
Med.
Few
All
Expert A)
64.3
49.0
31.9
52.6
63.5
44.5
20.3
44,0
68.8
43.7
17.2
49.8
67.6
36.3
6.8
38.4
Expert A/
64.7
49.4
31.2
52.8
63.1
44.7
2112
43.8
65.5
50.5
33.3
53.9
61.2
44.7
23.5
44.3
Expert A/s
64.3
48.9
31.
52.5
63.9
45.1
20.5
44.3
43.4
48.5
53.9
47.3
14.0
27.6
41.2
25.8
Ensemble
68.0
52.9
35,1
863
67.4
49.5
23.7
48.0
67.0
56.7
42.6
58.8
61.6
50.5
33.9
49.4
5.3 Superiority on Test-agnostic Long-tailed Recognition
In this subsection, we evaluate SADE test-agnostic long-tailed recognition. The results on various test class distributions are reported in Table 5. Specifically, since Softmax seeks to simulate the long- tailed training distribution, it performs well on forward long-tailed test distributions. However, its performance on the uniform and backward long-tailed distributions is poor. In contrast, existing long- tailed methods show more balanced performance among classes, leading to better overall accuracy. However, the resulting models by these methods suffer from a simulation bias, i.e., performing similarly among classes on various class distributions (c.f. Table 1). As result, they cannot adapt to diverse test class distributions well. To handle this task, LADE assumes the test class distribution to be known and uses this information adjust its predictions, leading better performance on various test class distributions. However, since obtaining the actual test class distribution is difficult in real applications, the methods requiring such knowledge may be not applicable in practice. Moreover, in some specific cases like Forward-LT-3 and Backward-LT--3 distributions of iNaturalist 2018, the number of test samples on some classes becomes zero. In such cases, the test prior cannot be used in LADE, since adjusting logits with log 0 results in biased predictions. In contrast, without relying on the knowledge of test class distributions, our SADE presents an innovative self supervveed strategy to deal with unknown class distributions, and obtains even better performance than LADE that uses the test class prior (c.f. Table 5). The promising results demonstrate the effectiveness and practicality of our method on test-agnostic long-tailes toiglt recognition. Note that the performance advantages of SADE become larger as the test data get more imbalanced. Due to the page limitation, the results on more datasets are reported in Appendix D.2.
5.4 Effectiveness of Skill-diverse Expert Learning
We next examine our skill-diverse expert learning strategy. The results are reported in Table 6, where RIDE [49] is state-of-the-an ensemble-based method. RIDE trains each expert with cross-entropy independently and uses KL-Divergence to improve expert diversity. However, simply maximizing the divergence of expert predictions cannot learn visibly diverse experts (cf. Table 6). In contrast, the three experts learned by our strategy have significantly diverse expertise, excelling at many-shol classes, the uniform distribution (with higher overall performance), and few-shot classes, respectively. As result, the increasing expert diversity leads to a non-trivial gain for the ensemble performance of SADE compared to RIDE. Moreover, consistent results on more datasets are reported in Appendix D.3, while the ablation studies of the expert learning strategy are provided in Appendix E.
8

---

Table 7: The expert weights learned by our self- supervised strategy on ImageNet-LT with various test class distributions. Our method learns suit- able weights for various unknown distributions.
Table 8: The performance improvement by our test-time self seff-supervised strategy on ImageNet- LT with various test class distributions.
Inog:Nos1
Tost Dist.
Expert E (Mig)
Expert Fy (egg)
Expert Fg (mn)
Dist
stareyy
strrnge
Forward LT-50
0.52
0.35
0,13
Many
Mail
Fow
All
Mary
Meal
Foww
Forwadd-L1-10
0.46
0.18
Forwand-LT-50
65.6
55.3
44.1
65.5
700
53.2
33.1
094
Uniform
0.33
Forwend-LL--
42.6
093
(00.0
0.21
0.79
0.50
Rackoond-11--00
650
57.6
45.1
51.1
608
17.5
501
54.5
Backward- LT:50
0.17
0.27
0.56
09.1
57.0
42.9
48.8
69.
56.2
50.7
(-13)
Table 9: Comparison among different test-time training strategies for handling class distribution
shifts on ImageNet-LT with various unknown test class distributions.
Backbone
Test time strategy
Furward
Unifomm
Backward
test-time adapiation
65.5
64.4
63.6
62.0
60.0
58.8
56.8
51.7
53.1
51.5
49,8
Testtime pscudo- pacudo-labccing
67.1
66.1
64.7
63.0
66..1
57.7
54.7
51.1
48.1
45.0
42.4
SADR
Test class distritation estimation [29]
69.1
66.6
63.7
60.5
56.5
53.3
49.9
45.6
42.7
39.5
36.8
Entropy minimization with [166
60.5
58.6
18.1
Sell-upervisd expert aggregation (ours)
69.4
67.4
63.4
63.0
60.6
5888
57.1
55.5
54.5
53.7
53.1
5.5 Effectiveness of Test-time Self-supervised Aggregation
This subsection evaluates our test-time self- self-supervised aggregation strategy.
Effectiveness in expert aggregation. As shown in Table 7, our self-supervised strategy learns suitable expert weights for various unknown test class distributions. For forward long-tailed dis- tributions, the weight of the forward expert E₁ is higher; while for backward long-tailed ones, the weight of the backward expert E3 is relatively high. This enables our multi-expert model to boost the performance on dominant classes for unknown test distributions, leading to better ensemble performance (cf. Table 8), particularly as test data get more skewed. The results on more datasets are reported in Appendix D.4, while more ablation studies of our strategy are shown in Appendix F.
Superiority over test-time training methods. We then verify the superiority of our self-supervised strategy over existing test-time training approaches on various test class distributions. Specifically, we adopt three non-trivial baselines: (i) Test-time pseudo-labeling uses the multi-expert model to iteratively generate pseudo labels for unlabeled test data and uses them fine-tune the model; (ii) Test class distribution estimation leverages BBSE [29] to estimate the test class distribution and uses it to pose-adjust model predictions; (iii) Tent [46] fine-tunes the batch normalization layers of models through entropy minimization on unlabeled test data. The results in Table 9 show that directly applying existing test-time training methods cannot handle well the class distribution shifts, particularly on the inversely long-tailed class distribution. In comparison, our self-supervised strategy able aggregate multiple experts appropriately for the unknown test class distribution (cf. Table 7), leading to promising performance gains on various test class distributions (cf. Table 9).
Effectiveness on partial class distributions. Real-world test data may follow any type of class distribution, including partial class distri- butions (i.e.. not all of the classes appear in the test data). Motivated by this, we further evaluate SADE on three partial class distributions: only many-shot classes, only medium-shot classes, and only few-shot classes. The results in Ta- ble 10 demonstrate the effectiveness of SADE in tackling more complex test class distributions.
Table 10: The effectiveness of our self-supervised aggregation strategy in dealing with (unknown) partial test class distributions ImageNet-LT.
ImageNes-LI
Method
Only many
Ouly medium
Ouly fow
SADE w/o test-time strategy
67.4
56.9
42,5
test-time strategy
71.0
Accuracy gain
(+3.6)
(+0.3)
(+11.))
6 Conclusion
In this paper, we have explored a practical yet challenging task test-agnostic long-tailed recognition, where the test class distribution is unknown and not necessarily uniform. To tackle this task, we present novel approach, namely Self-supervised Aggregation of Diverse Experts (SADE), which consists of two innovative strategies, i.e., skill-diverse expert learning and test-time self- seff-supervised aggregation. We theoretically analyze our proposed method and also empirically show that SADE achieves new state-of-the- performance on both vanilla and test-agnostic long-tailed recognition.

---

Acknowledgments
This work was partially supported by NUS ODPRT Grant R252-000-A81-133 and NUS Advanced Research and Technology Innovation Centre (ARTIC) Project Reference (ECT-RP2). We also gratefully appreciate the support of MindSpore, CANN (Compute Architecture for Neural Networks) and Ascend Al Processor used for this research.
References
[1] Malik Boudiaf, Jéröme Rony, al. unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In European Conference Computer Vision, 2020.
[2] Jiarui Cai, Yizhou Wang, and Jenq-Neng Hwang. Ace: Ally complementary experts for solving long-tailed recognition in one-shot. In International Conference on Computer Vision, 2021.
[3] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbal- anced datasets with bbbel-distributio--aaar margin loss. In Advances in Neural Information Processing Systems, 2019.
[4] Nitesh V Chawla, Kevin W Bowyer, et al. Smote: synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 2002.
[5] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprini arXiv:2003.0422 2020.
[6] Zhixiang Chi, Yang Wang, et al. Test-time fast adaptation for dynamic scene deblurring via meta-auxiliary learning. In Computer Vision and Puttern Recognition, 2021.
[7] Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In Advances in Neural Information Processing Systems, volume 33, 2020.
[8] Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In International Conference on Computer Vision, 2021.
[9] Yin Cui, Menglin Jia, al. Class-balanced loss based on effective number of samples. In Computer Vision and Pattern Recognition, 2019.
[10] Zongyong Deng, Hao Liu, Yaoxing Wang, Chenyang Wang, Zekuan Yu, and Xuehong Sun. Pml: Progressive margin loss for long-tailed age classification. In Computer Vision and Pattern Recognition, pages 10503-10512, 2021.
[11] Chengjian Feng, Yujie Zhong, and Weilin Huang. Exploring classification equilibrium in long-tailed object detection. In International Conference on Computer Vision, 2021.
[12] Yves Grandvalet, Yoshua Bengio, al. Semi-supervised learning by entropy minimization. In CAP, 2005.
[13] Hao Guo and Song Wang. Long-tailed multi-label visual recognition by collaborative training on uniform and re-balanced samplings. In Computer Vision and Pattern Recognition, 2021.
[14] Marton Havasi, Rodolphe Jenatton, et al. Training independent subnetworks for robust predic- tion. In International Conference on Learning Representations, 2021.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition, pages 770-778, 2016.
[16] Yin He, Peizhen Zhang, Xiu-Shen Wei, Xiangyu Zhang, and Jian Sun. Relieving long-tailed instance segmentation via pairwise class balance. arXiv preprint arXiv:2201.0277
2022. [17] Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disentangling label distribution for long-tailed visual recognition. In Computer Vision and Pattern Recognition, 2021.
[18] Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In Computer Vision and Pattern Recognition, 2016.
[19] Yusuke wwssawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. Advances Neural Information Processing Systems, volume 34, 2021.
10

---

[20] Muhammad Abdullah Jamal, Matthew Brown, Ming- Haug- Yang, Liqiang Wang, and Boqing Gong. Rethinking class-balanced methods for long-taile visual recognition from a domain adaptation perspective. In Computer Vision and Pattern Recognition, pages 7610-7619, 2020.
[21] Ren Jiawei, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long-tailed visual recognition. In Advances in Neural Information Processing Systems, 2020.
[22] Justin M Johnson and Taghi M Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Data, 6(1):1-54, 2019.
[23] Mohammad Mahdi Kamani, Sadegh Farhang, Mehrdad Mahdavi, and James Z Wang. Targeted data-driven regularization for out-of-distribution generalization. In ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 882-891, 2020.
[24] Bingyi Kang. Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference on Learning Representations, 2021.
[25] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Reprexentations, 2020.
[26] Ildoo Kim, Younghoon Kim, and Sungwoong Kim. Learning loss for test-time augmentation. Advances in Neural Information Processing Systems, 33:4163-4174 2020.
[27] Yu Li, Tao Wang, Bingyi Kang, Sheng Tang, Chunfeng Wang, Jintao Li, and Jiashi Feng. Overcoming classifier imbalance for long-tail object detection with balanced group softmax. In Computer Vision and Puttern Recognition, 2020.
[28] Hongbin Lin, Yifan Zhang, Zhen Qiu, Shuaicheng Niu, Chuang Gan, Yanxia Liu, and Mingkui Tan. Prototype-guided continual adaptation for caassinncrementl unsupervised domain adapta- tion. In European Conference Computer Vision. 2022.
[29] Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International Conference on Machine Learning. pages 3122-3130.
PMLR, 2018.
[30] Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. Ttt++: When does self supervied test-time training fail or thrive? In Advances in Neural Information Processing Systems, volume 34, 2021.
[31] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Computer Vision and Pattern Recognition, pages 2537-2546, 2019.
[32] Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S Yu. Transfer joint matching for unsupervised domain adaptation. In Computer Vision and Pattern Recognition, pages 1410-1417, 2014.
[33] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. International Conference on Learning Representationss, 2021.
[34] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International Confer- ence on Machine Learning, 2022.
[35] Prashant Pandey, Mrigank Raman, Sumanth Varambally, and Prathosh AP. Generalization on unseen domains via inference-time label- aabel-preserving target projections. In Computer Vision and Pattern Recognition, pages 12933, 2021.
[36] Seulki Park, Jongin Lim, Younghan Jeon, and Jin Young Choi. Inffuence-balanced loss for imbalanced visual classification. In International Conference on Computer Vision, 2021.
[37] Hanyu Peng, Mingming Sun, and Ping Li. Optimal transport for long-tailed recognition with learnable cost matrix. In International Conference on Learning Representations, 2022.
[38] Zhen Qiu, Yifan Zhang, Hongbin Lin, Shuaicheng Niu, Yanxia Liu, Qing Du, and Mingkui Tan. frurce-free domain adaptation via avatar prototype generation and adaptation. In International Joint Conference on Artificial Intelligence, 2021.
[39] Saurabh Sharma, Ning Yu, Mario Fritz, and Bernt Schiele. Long-tailed recognition using class-balance experts. In German Conference on Pattern Recognition, pages 86-100, 2020.
11

---

[40] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with sulf-supervision for generalization under distribution shifts. In Interaatoonal Conference Machine Learning. 2020.
[41] Jingru Tan, Changhao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-tailed object recognition. In Computer Vision and Pattern Recognition, pages 11662-11671, 2020.
[42] Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. In Advances in Neural Information Processing Systems, volume 33, 2020.
[43] Junjiao Tian, Yen-Cheng Liu, et al. Posterior re-calibration for imbalanced datasets. In Advances in Neural Information Processing Systems, 2020.
[44] Grant Van Horn, Oisinand Mac Aodha, et al. The inaturalist species classification and detection dataset. In Computer Vision and Puttern Recognition, 2018.
[45] Thomas Varsavsky, Mauricio Orbes-Arteaga et al. Test-time unsupervised domain adaptation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 428-4344 2020.
[46] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
[47] Jiaqi Wang. Wenwei Zhang. Yuhang Zang, Yuhang Cao, Jiangmiao Pang, Tao Gong, Kai Chen, Ziwei Liu, Chen Change Loy, and Dahua Lin. Seesaw loss for long-tailed instance segmentation. In Computer Vision and Pattern Recognition, pages 9695-9704, 2021.
[48] Peng Wang, Kai Han, al. Contrastive learning based hybrid networks for long-tailed image classification. In Computer Vision and Pattern Recognition, 2021.
[49] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu. Long-tailed recogni- tion by routing diverse distribution-awen experts. In International Conference on Learning Representations, 2021.
[50] Yandong Wen, Kaipeng Zhang, et al. A discriminative feature learning approach for deep face recognition. In European Conference on Computer Vision, 2016.
[51] Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations, 2020.
[52] Zhenzhen Weng, Mehmet Giray Ogut, Shai Limonchik, and Serena Yeung. Unsupervised discovery of the long-tail in instance segmentation using hierarchical self-supervision. In Computer Vision and Pattern Recognition, 2021.
[53] Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification. In European Conference on Computer Vision, 2020.
[54] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems, volume 27, pages 3320-3328, 2014.
[55] Yuhang Zang, Chen Huang, and Chen Change Loy. Fasa: Feature augmentation and sampling adaptation for long-tailed instance segmentation. In International Conference on Computer Vision, 2021.
[56] Songyang Zhang, Zeming Li. Shipeng Yan, Xuming He, and Jian Sun. Distribution alignment: unified framework for long-tail tail visual recognition. In Computer Vision and Pattern Recognition, pages 2361-2370, 2021.
[57] Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Unleashing the power of contrastive self-supervised visual models via contrast-regularizee fine-tuning. In Advances in Neural Information Processing Systems, 2021.
[58] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey, arXiv preprint 210.vv2110.04596 2021.
12

---

[59] Yifan Zhang. Ying Wei, et al. Collaborative unsupervised domain adaptation for medical image diagnosis. IEEE Transactions on Image Processing, 2020.
[60] Yifan Zhang, Peilin Zhao, Jiezhang Cao, Wenye Ma. Junzhou Huang, Qingyao Wu, and Mingkui Tan. Online adaptive asymmetric active learning for budgeted imbalanced data. In ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 2768 2777, 2018.
[61] Peilin Zhao, Yifan Zhang, Min Wu, Steven CH Hoi, Mingkui Tan, and Junzhou Huang. Adaptive cost-sensitive online classification, IEEE Transactions on Knowledge and Data Engineering, 31(2):214-228 2018
[62] Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recognition. In Computer Vision and Pattern Recognition, 2021.
[63] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral- network with cumulative learning for long-tailed tailed visual recognition. In Computer Vision and Pattern Recognition, pages 9719-9728, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please refer to Appendix H.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] This is fundamental research that does not have particular negative social impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Please refer to Appendix A.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] Please refer to the supplemental material.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] Please refer to Section 5.1 and Appendix C.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A] The common practice in long-tailed recognition does not report error bars, we follow the previous papers and do not report them.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please refer to Appendix C.3 for details different datasets.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did mention the license of the assets? [N/A] All the used benchmark datasets are publicly available.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] We submitted the source codes of method an anonymized zip file.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] These datasets are open-source benchmark datasets.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] These datasets are open- benchmark datasets.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "hmtLGfyuyP-", "reviewer": "Reviewer_rg9d", "summary": "This paper studies an interesting problem in long-tailed recognition, i.e.,  the training class distribution is long-tailed while the test class distribution is agnostic rather than a uniform distribution as assumed in previous works. To deal with the problem, this paper proposes a new approach that outperforms existing methods in both vanilla and test-agnostic long-tailed recognition settings.", "questions": "1. Why does this paper choose Balanced softmax and a variant logit adjustment loss?\n2. How should we decide the number of experts?\n3. Is the classifier re-weighting strategy optimal? And can the Test-time Self-supervised Aggregation learn optimal weights?\n4. Why should we use the proposed method instead of existing long-tailed methods such as RIDE? The performance improvement is not significant and the proposed method incurs additional computational costs.", "limitations": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n\n1. The studies problem is interesting and under-explored.\n2. Extensive experiments are conducted to justify the effectiveness of the proposed method.\n3. The writing is clear and easy to understand.\n\n## Weakness\n\n1. The technical significance is not enough. Specifically, there are two aspects. First, **the** **skill-diverse expert learning** does not make new contribution to the field because multiple experts have been used in many existing literature, e.g., [1-3]. Moreover, the idea of aggregating multiple diverse models was also explored in [3] though the studied problem is different. Second, **the** **Test-time Self-supervised Aggregation** simply a re-weighting of three models. The key contribution might be the prediction stability maximization, but optimizing this objective does not ensure to obtain the optimal weights. \n2. This paper only considers the transductive setting where the entire test data are accessible at once. However, in many applications, the assumption is not satisfied.\n3. This paper incurs more computational cost than previous methods. The Test-time Self-supervised Aggregation has to be performed at each test time.\n4. nitpick: some bold numbers in Table 9 are not the best results.\n\n[1] Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification\n\n[2] Long-tailed recognition by routing diverse distribution-aware experts\n\n[3] Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classification", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OQJbmFlT1DJ", "reviewer": "Reviewer_uSnn", "summary": "The aim of this paper is to develop a mixture-of-expert (MOE) model to solve the test-agnostic long-tailed recognition problem, where the test class distribution may follow a uniform, forward or backward long-tailed distribution. The method is developed on the basis of RIDE with three experts and consists of two strategies. At training time, SADE utilizes skill-diverse expert learning strategies that require each expert to handle a different class distribution in order to solve distribution-agnostic long-tailed recognition problems. At test time, SADE utilizes a test-time expert aggregation strategy, which is based on a self-supervised learning approach, to determine expert aggregation methods that handle unknown class distributions. Experiments were conducted on various test-time training strategies dealing with class distribution transfer. SADE achieves state-of-the-art performance on multiple long-tailed datasets, including CIFAR100-LT, ImageNet-LT, Places-LT and iNaturalist 2018.", "questions": "I have a few questions on the fairness of the result comparisons and the setting of the test-time adaptation:\n- When comparing with the current SOTA method RIDE [46], the results of RIDE (https://github.com/frank-xwang/RIDE-LongTailRecognition/blob/main/MODEL_ZOO.md) is actually achieved by training the model for 100 epochs. However, the results reported in the paper is achieved by training the model for 200 epochs. Therefore, I am concerned about the fairness of the comparisons. \n- For test-time adaptation, have you tried using MC-Dropout [1] as a metric for expert aggregation? You can get the uncertainty of each expert and decide the weight of each expert based on the uncertainty. Does it save you from using all test data for expert aggregation? I think it might help if MC-Dropout is used with a strong data augmentation to produce different inputs.\n\n[1] Gal, Yarin, and Zoubin Ghahramani. \"Dropout as a bayesian approximation: Representing model uncertainty in deep learning.\" international conference on machine learning. PMLR, 2016.", "limitations": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work. The long-tail recognition algorithm aims to alleviate the problem of ignoring underrepresented minorities, which is necessary to obtain an unbiased model and facilitate the development of CNN models for social justice.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Evaluating on Forward-LT and Uniform test-class distributions can help us better understanding the performance of various long-tailed algorithms on different testing scenarios. It is shown that SADE can achieve SOTA performance on all these testing distributions.\n- This paper is well-written and easy to follow. \n- Authors conducted thorough experiments on various benchmarks and testing scenarios and achieved consistent performance improvements on all these benchmarks. \n\nFor weaknesses, my main concerns are twofold:\n- The biggest improvement comes from test data with Backward-LT distributions, however, I find it hard to believe that backward class distributions are common in real-world applications. The many-shot (few-shot) classes in the training data are often the many-shot (few-shot) classes in the testing data as well. Thus, Forward-LT and Uniform test class distributions make more sense, however, SADE achieves marginal improvements in these two testing cases. \n- The test-time self-supervised aggregation strategy requires the model to see all test data (unlabeled) before deployment, however, in real-world applications we more commonly see only 1 test image. This part is similar to my first concern, which is whether this setup is a practical setup that can be used in real-world applications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "CjoqrvlidJx", "reviewer": "Reviewer_eBs7", "summary": "This paper extends the conventional long-tailed learning to the \"test-agnostic one\", in which the model trained on a long-tailed class distribution should generalize to arbitrary testing distribution not necessarily being uniform. To handle such a problem, this paper proposes a novel method consisting of two modules: (1) diverse experts with different class expertise and (2) a self-supervised test-time weighting strategy that adaptively aggregates the experts to tackle unknown testing distribution. Extensive experiments have shown the efficacy of the proposed method on handling arbitrary class distributions in testing.\n", "questions": "Line 762: how exactly is the expertise-guided loss functions changed to suit different types of distributions?\n", "limitations": "The limitations are carefully discussed in the paper, which mainly encompass the extensibility to different tasks and the model complexity of the proposed method.\n", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The proposed test-agnostic setting is challenging and of great practical significance.\n- The paper is well-written and easy to follow.\n- The proposed test-time aggregation strategy is interesting and has proven to be useful.\n- The experiments and ablation studies are comprehensive and convincing.\n\n## Weaknesses\n- The weakness is mainly focused on the computation complexity. As mentioned in the paper, the three experts are independent in ResNet blocks (later stages) and fully-connected layers. Though it seems tolerable since it is a quite challenging problem after all, have the authors explored the trade-off between accuracy and complexity? For instance, is it a near-linear relationship between higher accuracy and experts with fewer shared modules? I would like to see how far can it go at the two extreme points: (1) when nothing is shared between experts and (2) everything is shared except the fully-connected layers.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept