Pareto Deep Long-Tailed Recognition: A Conflict-Averse Solution
Authors: Zhipeng Zhou
Liu Liu
Peilin Zhao
Wei Gong
Venue: ICLR 2024 poster (2024)
Keywords: Deep long-tailed recognition
Representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We provide a new angle in deep long-tailed recognition, i.e., addressing the optimization conflicts among categories during representation.

Abstract: Deep long-tailed recognition (DTLR) has attracted much attention due to its close touch with realistic scenarios. Recent advances have focused on re-balancing across various aspects, e.g., sampling strategy, loss re-weighting, logit adjustment, and input/parameter perturbation, to name a few. However, few studies have considered dynamic re-balancing to address intrinsic optimization conflicts. In this paper, we first empirically argue that the optimizations of mainstream DLTR methods are still dominated by some categories (e.g., major) due to a fixed re-balancing strategy. Thus, they fail to deal with gradient conflicts among categories, which naturally deduces the motivation for reaching Pareto optimal solutions. Unfortunately, a naive integration of multi-objective optimization (MOO) with DLTR methods is not applicable due to the gap between multi-task learning (MTL) and DLTR, and can in turn lead to class-specific feature degradation. Thus, we provide effective alternatives by decoupling MOO-based MTL from the temporal rather than structure perspective, and enhancing it via optimizing variability collapse loss motivated by the derived MOO-based DLTR generalization bound. Moreover, we resort to anticipating worst-case optimization with theoretical insights to further ensure convergence. We build a Pareto deep long-tailed recognition method termed PLOT upon the proposed MOO framework. Extensive evaluations demonstrate that our method not only generally improves mainstream pipelines, but also achieves an augmented version to realize state-of-the-art performance across multiple benchmarks.

Published as a conference paper at ICLR 2024
PARETO DEEP LONG-TAILED RECOGNITION: CONFLICT-AVERSE SOLUTION
A
Zhipeng Zhou', Liu Liu Peilin Zhao University of Science and Technology of China, zzp1994 194mmaim uste. edu. (Leonliuliu, weigengeuste.. edu.
Wei Gong
Tencent AI Lab
masonznao @wweeeet com
ABSTRACT
Deep long-tailed recognition (DLTR) has attracted much attention due its close touch with realistic scenarios. Recent advances have focused on re-balancing across various aspects, e.g., sampling strategy, loss re-weighting, logit adjustment, and input/parameter perturbation, etc. However, few studies have considered dynamic re- re-balancing to address intrinsic optimization conflicts, which are identified as prevalent and critical issues in this study. In this paper, we empirically establish the severity of the optimization conflict issue in the DLTR scenario, which leads a degradation of representation learning. This observation serves the motivation for pursuing Pareto optimal solutions. Unfortunately, a straightforward integration of multi-objective optimization (MOO) with DLTR methods is infeasible due to the disparity between multi-task learning (MTL) and DLTR. Therefore, we propose effective alternatives by decoupling 10O-based MTL from temporal perspective rather than a structural one. Furthermore, we enhance the integration of MOO and DLTR by investigating the generalization and convergence problems. Specifically, we propose optimizing the variability collapse loss, guided by the derived MOO- based DLTR generalization bound, to improve generalization. Additionally, we anticipate worst-case optimization to ensure convergence. Building upon the proposed MOO framework, we introduce novel method called Pareto deep LOng- Tailed recognition (PLOT). Extensive evaluations demonstrate that our method not only generally improves mainstream pipelines, but also achieves an augmented version realize state-of-the-art performance across multiple benchmarks. Code is available at httttts: /gtttpps///gthhb.... com/ wttpom///ppsttpPPOTTTT
INTRODUCTION
Nowadays success of machine learning (ML) techniques are largely attributed to the growing scale of the training dataset, as well as the assumption of being independent and identically distributed (i.i.d) with the test distribution. However, such an assumption can hardly hold in many realistic scenarios where training sets show an imbalanced or even long-tailed distribution, raising a critical challenge to the traditional ML community (Zhang et al. 2021b). To address this issue, recent researches devoted on deep long-tailes tailed recognition (DLTR) has gained increasing interests, which strives to mitigate the bias toward certain categories and generalize well on a balanced test dataset.
Plenty of approaches have been proposed to realize ballancing from various aspects in DLTR (Zhang et al.
2021b): sampling strategy (Zang al.) 2021 Cai et al. (021), ) loss function (Wang al.
2013
Ren et al.] 2020 Tan et al. 2020). logit adjustment (Cao et al. 2019: Li et al. 2022), data
augmentation (Kim al. 2020 Wang et 2021), input/parameter perturbation (Rangwani et
2022:
Zhou et al. 2023). decoupling learning regime (Kang et al. 2019), and diverse experts (Wang
al.
2020b; Guo & Wang 202T), etc. Usually, these works design fixed re-balancing strategies
according to the prior of the class frequency to ensure all categories are generally equally optimized.
Several very recent researches (Ma et al.) 2023) Sinha Ohashi 2023 Tan et al. (202)) empirically indicate that a dynamic re- oo-balancigg strategy is required, and achieve it by designing a quantitative
Corresponding authors. Work done when Z. Zhou works intern in Tencent AT Lah.

---

Published as a conference paper at ICLR 2024
8520308230
Saaaaassad
31898ase36
(a) ERM
(b) LDAM-DRW
(c) Bal. Softmax
(d) M2m
(e) MiSLAS
(f) GCL
Figure 1: Gradient conflicts among categories. 'Bal. Softmax' short for Balanced Softmax. The horizontal and vertical coordinates are for each category and the heat map represents the gradient similarity.
w
(a) ERM
(b) LDAM-DRW
(c) Bal. Softmax
(d) M2m
(e) MiSLAS
(f) GCL
Figure 2: Gradient similarities during optimization. *0', $4', and '9' denote the corresponding categories in CIFARIO-LT, belonging to the 'head', 'medium', 'tail' classes. Please refer to the gradient norm examination in Section 4.5 of the Appendix.
measurement semantic scale imbalance and a meta module to learn from logit, etc. All these works take the instant rather than prior imbalance into consideration, enabling them reach competitive performance across various imbalanced scenarios. Nevertheless, a question is naturally raised:
Is instant imbalance enough for designing dynamic re-balancing strategy?
We then delve into the optimization of representative DLTR models and present related observations from the perspective of multi-objective optimization (MOO) in Fig. and Fig. As depicted, intrinsic optimization conflicts among categories are prevalent and might be aggravated due to the dominated trajectories of certain categories, which would lead to sub-optimal solutions for the remaining Such an issue is rarely discussed for the above question and cannot be addressed by current dynamic strategies due to their lack of design nature (Refer to Section 3 for
details).
To fill this gap, we approach DLTR from a new angle, i.e., mitigate optimization conflicts via dynamic balannnnnnng which is usually neglected in past works. We first identify the existing intrinsic gradient conflicts among categories in the optimization of prevailing DLTR methods and show their connection with the adopted fix re-balancing strategies. To prevent the representation from being overwhelmed by dominated categories' properties, we introduce MOO in MTL to mine the shared features among categories. Unfortunately, naïve combination is not applicable due to the structure difference between MTL and DLTR illustrated in Fig. Specifically, MOO-based MTL usually assumes that the model architecture is consist a backbone network and several separate task-specific branches on top of the backbone network, and strives to learn task-shared features with backbone network via MOO algorithms. While DLTR targets only one task and owns only one branch. Hence a critical challenge appears:
How to engage MOO into DLTR?
As depicted in Fig. we tackle this challenge with two key enablers: (1) Regarding a multi- classification task as multiple binary classification tasks, (2) transform the shared feature extraction and task task-specific optimization from structural to temporal. Besides, by investigating several popular MOO approaches and choosing a stable one, we provide instructions on the integration of DLTR and MOO, propose variability collapse loss, and anticipate worst-case optimization ensure generaliza- tion and convergence. It should be noted that our goal is to provide a distinct angle of re- --bllancing rather than design new instant imbalance metric or MOO method, thus comparing approach with these counterparts is beyond the scope of this paper.
Contributions: Our contributions mainly be summarized as four-fold:
2

---

Published as a conference paper at ICLR 2024
Through the lens of MOO, empirically identify the phenomena of optimization conflicts among categories and establish its severity for representation learning in DLTR.
To mitigate the above issues, we endow prevailing re-balancing models with Pareto property via innovatively transforming the MOO- boo-- MTL from structural to temporal, enabling the application of MOO algorithm in DLTR without model architecture modifications.
Moreover, two theoretical motivated operations, i.e., variability collapse loss, and an- ticipating worst-case optimization are proposed to further ensure the generalization and convergence of MOO-based DLTR.
Extensive evaluations have demonstrated that our method, PLOT, can significantly enhance the performance of mainstream DLTR methods, and achieve state-of-the of-the-art results across multiple benchmarks compared to its advanced counterparts.
2 PRELIMINARIES
Taking a Wwy classification task for example, assume we are given a long-tailet training set S = (xy, ysli 1.... ,.) for the DLTR problem. And the corresponding per-class sample numbers are {m1, ??,..., nk }. Σx n,, Without loss of generality, we assume n, < nj if i < j. and usually n > Following the general DLTR setting, all models are finally evaluated on a balanced test dataset.
Problem Setup:
Our framework hinges on MOO-based MTI,, which strives to achieve the Pareto optimum under the MTI. situation. Formally, assume that there are N tasks at hand, and their
Pareto Concept:
differentiable loss functions are (0), E [N]. The weighted loss is Lw = -E' (,C,(0), w E W, where 0 is the parameter of the model and W the probability simplex on [N]. A point O' is said to Pareto dominate 0, only if Vi, (0)) < (0). And therefore the Pareto optimal is the situation that no O' can be found that holds Vi, (0") S (0) for the point O. All points that satisfy the above conditions are called Pareto sets, and their solutions are so-called Pareto fronts. Another concept called Pareto stationary, which requires min || = where IN is the weighted gradient. In this paper, since we regard the K-way classification task as K binary tasks, N is assigned as K.
Definition 2.1 (Gradient Similarity). Denote thj as the angle between two task gradients gi and gj. then we define the gradient similarity cos Pij and the gradients conflicting when cos Φiş < O. Definition 2.2 (Dominated Confficting). For task grudients gi and 9j. assume their average gradient as 9o. and [g: < || Then define the gradients dominated conflicting when dos O.
3 MOTIVATION AND EMPIRICAL OBSERVATIONS
Intrinsic Property in Definition and Difference with MTL: As outlined in Section 2 DLTR model is trained an imbalanced dataset but is expected to generalize well to all categories, which aligns with the motivation of Pareto optimality, i.c., improving all individual tasks (categories). However, unlike MTL, which employs distinct structure where the backbone and corresponding branches are responsible for shared feature extraction and task-specific optimization, respectively. the structures in DLTR are attributed to all categories. This difference impedes DLTR models from achieving Pareto properties. Therefore, in Section 4 we introduce the MOO-based DLTR pipeline. Optimization Conflicts under Imbalanced Scenarios: As depicted Fig. 3 evident that
Bi
Emeon
Breen
Uman
Fj
8
confictiin
conficting
non-confiicing
confiicting
conflicting
(a) Balanced Scenarios.
(b) Imbalanced Scenarios.
Figure 3: Illustration of gradient conflict scenarios.
each task exhibits improvement when optimized using the average gradient, i.e., Iwean in bal balanced
3

---

Published as a conference paper at ICLR 2024
Table Benefits of MOO methods for mainstream DLTR models on CIFARIO-LT. We re-implement all models via their publicly released code, and all results are reported over 3 random seeds experi- ments. //v indicates nunperfoms/unddeeeffomm their vanilla versions, while the early stop version is colored and the naive integration version is underlined.
cKT+Mixup
LDAM-DRW
Imb.
Vanilla
w/ EPO
w/ MGDA
w/ CAGrad
Vanilla
w/ EPO
w/ MGDA
w/ CAGrad
200
73.06
33:45
76.24
68.05
75.98a
75.15
76.02a
71.38
56,04
73.64a
67.18
74.08
55.80
73.28A
100
79.15
34.27
79.69
73.71
79.26
79.58
80.16A
77.71
66.49
77.25
73.70
77.79
66.49
76.86v
50
84.21
36.53
83.79
79.27
84.15
83.52
84.49
81.78
72.60
81.62v
78.24
81.58
69.26
81.85A
Balanced Softmax
M2m
Imb.
Vanilla
w/ EPO
w/ MGDA
w/ CAGrad
Vanilla
w/ EPO
w/ MGDA
w/ CAGrrad
200
81.33
45.37
81,40a
74.13
80.90
79.20
80.93v
73.43
$1,90
73.07
57.14
72,63
70.95
73.84A
100
84,90
44.33
85.30a
79.06
83,77
85.40a
77.55
57.89
76.57v
52.37
76.48v
76,24
77.95
50
89.17
41443
88.97
79.43
88.90
88.00
89.27
80.94
42.07
81.19A
46.38
80.66
78.19
81.11a
MISLAS
GCL
Imb.
Vanilla
w/ EPO
w/ MGDA
CAGrad
Vanilla
EPO
w/ MGDA
CAGrad
200
76.59
36.62
76.97
63 63.40
76.12v
76,30 30
77.43a
79.25
62.08
79.73a
75.43
80.03
78.73
80.08
81.33
39.92
81.22v
a2.47a
82.85
82.75
79.01
82.81
82.48
83.48A
50
85.23
44.78
84.60v
70,20
84.84
85,20
85.33a
86.00
78.42
84.55v
81.89
85.58v
85.31
85.90v
scenarios where conflicts arise. However, in imbalanced scenarios, the utilization of Jmeen tends to favor the dominant tasks. This preference becomes particularly pronounced in extreme cases, where even when Imean and gy are in conflict (referred to as Dominated Conflicting in Definition 2.2). the optimization of task i leads to an enhancement in its performance at the expense of task j. In order to investigate the presence of optimization conflict issues in DLTR, we meticulously analyze several re-balancing regimes: (1) Cost-sensitive loss approaches, such as LDAM-DRW (Cao et al. (019) and BalancedSoftmax (Ren et al. 2020); (2) Augmentation techniques, such as M2m (Kim al. 2020);; and (3) Decoupling methods, including cRT Mixup (Kang et al. 2019), MiSLAS (Zhogg et al. 202I), and GCL (Li et al. 2022). By computing the cosine similarities among gradients (referred to as Gradient Similarity in Definition 2.1 associated with different categories, we illustrate the conflict status of these methods in Fig. о As depicted, all the selected methods exhibit varying degrees of gradient conflicts, which persist in the early stages of training (refer to Section 4.2 in the Appendix). Furthermore, we examine the optimization preference of DLTR models in Fig. 2 revealing that certain categories dominate the overall optimization process. Additionally, we provide statistical analysis on the frequency of dominated confficting instances in Fig. 5 establishing roughly positive correlation between the frequency of dominated conflicting and the imbalance ratio.
54.90
613.69
Amaz 33.95
Ansz 472.6
Ante
Areos
Anate -2.00
Arun 77.75
Val Acc: 97.70
Val Acc:
Val Acc: 98.50
Val Acc: Ac0:30.0
155
104
20
(a) Class O in vanilla.
(b) Class 9 in vanilla.
(c) Class in addressed one.(d) Class 9 in addressed one.
Figure 4: Hessian spectrum analysis of before and after addressing optimization conflict issue.
The Benefit of Addressing Optimization Conflicts: Here we provide a preview of the advan- tages achieved by addressing optimization conflicts through the utilization of our temporal design, as outlined in Section 4.T We present the benefits from two perspectives: (1) representation analysis and (2) performance improvements. To gain insights into the impact of addressing the optimization conflict issue on representation learning, we conducted a Hessian spectrum analy-
Mainstream DLTR methods usually employ the SGD optimizer for implementation. Therefore, our analysis does encompass the results obtained by utilizing alternative optimizers such Adam.
4

---

Published as a conference paper at ICLR 2024
Shared Feature Extraction Task-specific Optimization
Shared Feature Extraction
Task-specific Optimization
branch
Backbone
Backbone
branch
epoch epoch
epoch E+1
(a) MOO-based MTL
(b) MOO-based DLTR
Figure 6: Comparison of MOO-based MTL and MOO-based DLTR.
sis (Rangwani et al. 2022) and visualized the results in Fig. Our analysis reveals that addressing optimization conflicts results flatter minima for each class, thereby mitigating the risk of be- ing trapped at saddle points and facilitating the acquisition of more generalized representations. Furthermore, we demonstrate the performance benefits achieved by employing various MOO ap- proaches in Table The results effectively showcase the potential of integrating MOO with DLTR. Urgency of Conflict-Averse Strategy: Currently, our MOO-based DLTR framework can primarily be classified as specialized dynamic re- re-balancing strategy, formally formulated as L(x. y) = >* wx Rm v.) Here B the batch size, B, and w. represent the frequency and dynamic re-weighting factor of class *, respectively, and
10 220
Dombuted Conffic 16
I(x", y.") is the average loss of class *. While there are
existing studies that explore the concept of dynamic balancing (Tan et al. 2023 Sinha & Ohashi 2023 Ma
al., 2023), none of them address the issue of optimiza- tion conflicts from comprehensive perspective. Conse- Figure 5: Statistical of dominated con-
quently, the intrinsic optimization conflicts among cate- flicts.
gories cannot be effectively mitigated (See Section 4.4 in the Appendix). Fortunately, our work bridges this gap and offers a solution to this problem.
4 PARETO DEEP LONG-TAILED RECOGNITION
Building the aforementioned analysis, present detailed design for integrating MOO into DLTR, which encompasses its adaptation from MTL to DLTR and an augmented version to ensure generalization and convergence. For complete proofs of the proposed theorems, please refer to Section 1 of the Appendix.
4.1 MOO: FROM MTL TO DLTR
As stated previously, straightforward integration of MOO is not feasible for DLTR scenarios due to differences in task properties and architectures. Upon revisiting the function of each component in MOO-based MTL, they be categorized into aspects: (1) Shared feature extraction (SFE) and (2) Task-specific optimization (TSO). SFE aims to extract shared representations among distinct tasks, while TSO is responsible for the corresponding task performance with the independent branch. task by treating it as multiple binary classification is considered as a single objective in MOO, and we re-design
In this study, we approach the tasks. Each binary classification SFE and TSO from a temporal perspective during the training stage, rather than structural one. Specifically, we implement SFE in the first E epochs by applying MOO algorithms in DLTR models, but release it in the subsequent stages, illustrated in Fig. 6 This approach can also be interpreted as an early stopping operation, inspired by previous research (Cadena et al. 2018 Hu et al. 202T) suggesting that the early layers of neural networks undergo fewer changes in the later stages of training. To validate the effectiveness this design, we employ three representative MOO algorithms, ie.. MGDA (Désidéri 2012), EPO (Mahapatra & Rajan 2020). and CAGrad (Liu al. 2021a), to
5

---

Published as a conference paper at ICLR 2024
equip them the aforementioned six DLTR models, and the results are presented in Table. | The results demonstrate that our design enables MOO to enhance DLTR models in most cases, which also highlights the potential benefits of addressing the optimization conflict problem. Moreover, the performance would significantly deteriorate without an early stop (naïve integration version), indicating class-specific feature degradation. It is also noteworthy that CAGrad exhibits relatively stable performance across various baselines, thus we select it as the fundamental framework for further enhancemen 2
Remark (No Modifications on Model Architecture). Once again, it is important to emphasize that our approach does not involve any modifications to the model architecture. Rather, our method represents an effective learning paradigm for the application of MOO in DLTR.
4.2 CHALLENGES OF CAGRAD UNDER DLTR
Generalization Problem: Prior to delving into the additional technical designs of PLOT, it is necessary to establish the learning guarantees for the MOO-based DLTR framework. To this end, we introduce we into the definition of Rademacher complexity (Cortes et al. 2020p, which can be reformulated as follows:
K Wk
mi o (((())))))))))))))))(())))))(( y.))))
(s
sup WEH
(1)
m&
where g is associated to the hypothesis set as H (9 (z, y) + (((()). 3) : vh E H}: og is the independent uniformly distributed random variables taking values in 1-1,+11 From this definition, we have the following theorem:
Theorem 4.1. (MOO-based DLTR Generalization Bound) If the loss function 4 belonging to Ku. category is Mg- -i-psccchitz, and v(x, y). (a', yy) E X x V, vh E H: [l[h(x), y [h(x') y'lll S Di. assume MdDu is bounded by M, then for any e 0 and 6 O, with probability at least 1 8, the following inequality holds for vh E H and tio E W:
K
log 3 2
(h) S (1) 29s w) + Σ W
The above derived generalization bound indicates that should minimize Lu (h) as well as constrain the intra-class loss variability M. With this theoretical insight, we design the following variability collapse loss:
K Σ Sud(i( v(
D.
K
where Std(-) is the standard deviation function and i( yn) is the loss of the kis category in mini-batch. It is worth noting that our proposed design shares a similar concept with recent study (Liu et al. 2023), which aims to induce Neural Collapse in the DLTR setting. However, our approach is distinct in that we propose it from the perspective of MOO with theoretical analysis.
Convergence Problem: On the other hand, although CA- Grad exhibits stability, it may not always yield improve- ments, as evidenced by Table Consequently, we delve deeper into CAGrad and demonstrate its limitations in the DLTR scenario. Based on the convergence analysis of CAGrad, we present the following theorem:
Loa Centour avurd Traned Mozol
Look Contecrtur arrnnd Tralned Mede
15
35
2
192
Theorem 4.2. (Convergence of CAGrad in DLTR) With fix step size a and the assumption of H-Lipschitz gradients, i.e., (;)) 7 ((')ll < H ||@ l'll for
(a) Head
(b) Tail
Figure 7: Loss landscape of LDAM.
i 1, 2, ..., K. Denote d (0x) as the optimization direction of CAGrad step then have:
((0(+1) [(..)
2 (1 - (0.)) - - 1) ||d
(8.))
Nonetheless, the objective of this paper is to explore the potential of the MOO framework in addressing the DLTR problem and to propose effective algorithm for enhancing mainstream DLTR methods. Therefore, we defer the development and integration of more advanced MOO algorithms to future research.

---

Published as a conference paper at ICLR 2024
where (0. and (0)) is the corresponding average gradient of 0₂. The convergence of CAGrad is inherently influenced by the value of H, as observed in our experiments. This observation is further supported by related study (Fernando et al. (022), which introduces random noise to the optimization trajectory of CAGrad and demonstrates convergence failures. Additionally, it is widely acknowledged that achieving small value of H for tail classes in DLTR models is often unattainable (Rangwani et al. 2022 Zhou et al. (023). To visually illustrate this point, we provide depiction of the LDAM-DRW loss landscape Fig. (for more visual illustrations, please refer to Section 7.2 in the Appendix). In this particular case, the loss landscape of the tail class exhibits sharp minimum, indicating large value of H and consequently posing challenges for achieving convergence when integrating CAGrad with DLTR models. To solve this problem, we constrain H by anticipating worst-case optimization (Foret et al. 2020). i.e., Sharpness aware minimization (SAM):
min max L(O + +(0)), where (e(()]l22 S. 0 <())
where the inner optimization can be approximated via the first order Taylor expansion, which results in the following solution (p is a syper-parametery
{()) /Vo((0)/ (IVoC(0)13* $
Overall Optimization Procedure: At the Enj step of the SFE stage, we first compute the original loss C()) for mini-batch, and obtain the perturbative loss LSAM = L ε(0) according Eqn. 4
as well as the variability collapse loss Lve defined in Eqn. 2 based on perturbative loss. Thus we
have the average gradient Do and the class-specific gradient gr, i E {1, 2, ...: K} by back propagating
fooo = LSAM + Lee- Finally, the dynamic class weights w = {W1, w2, -.. -kk is obtained by solving CAGrad (Liu et al. 202
min F F(w)= = 9-.g0 + 11911 where o = c² ||
(5)
WEW
and update the model via: O. 0.-1
go
Now gw
1//2
: 99w
Σx K
wi gz. The overall pseudo
algorithm is summarized in the Section 6.2 of the Appendix.
4.3 IMPLEMENTATION DETAILS
We implement our code with Python 3.8 and PyTorch 1.4.0, while all experiments are carried out on Tesla V100 GPUs. We train each model with batch size of 64 (for CIFARIO-LT and CIFAR100-LTT / 128 (for Places-LT) 256 (for ImageNet-LT and iNaturalist), SGD optimizer with momentum of 0.9.
EVALUATION
Table 2: Performance CIFAR datasets.
Following the mainstream protocols, we con- duct experiments on popular DLTR benchmarks: CIFARTO-/CIFAR 100-LL; Places-I.T (Liu et al., 2019) ImageNet-LT (Liu et al. 2019) and iNat- uralist2018 (Van Horn et al. 2018). To show the versatility of PLOT, aforementioned popular re- be-balancing regimes under various imbalance scenarios. Moreover, PLCT achieves the state-of-the-art (SOTA) per- formance by augmenting the advanced baseline across large scale datasets. Micro benchmarks are claborated show the effectiveness of each components finally. For fair comparison, we exclude ensemble pre-training models in our experiments.
CIFARIO-LT
CIPAR100-LT
Lmb.
200
100
50
200
100
50
LDAM-DRW
71.38
77.71
81.78
36.50
41 46.61
LDAM-DRW + +BLOT
74.32
78.19
82.09
37.31
42.31
47.04
equip it with the
M2m
73.43 77.55
80.94
33.81 40.77
45.73
M2m PLAT
74.48 78.42
81.79
38.43
43.00
47.19
cRT Mixup
73.06
79.15
84.21
(41.3
45.12
50.86
eRT Mixup + PLOT
78,99
80.55
84.58
43,80
47.59
51.43
Logit Adjustment
78.01
43.36
Logit Adjustment FLCT
MISLAS MISLAS PLOT
76.59
81.33
85.23
42.97
47.37
$1.42
77.73
81.88
85.70
44.28
47.91
52.66
79.25
82.85
44.88
52.85
GCL + PLOT
80.08
83,35
85.90
45.61
49,50
53.05
5.1 VERSATILITY VERIFICATION

---

Published as a conference paper at ICLR 2024
Given the inherent optimization conflicts in ad- vanced DLTR models, PLOT can serve as a valuable augmentation technique. Our experimental results, presented in Table 2 demonstrate that PLOT brings improvements in most scenarios by addressing the problem from new dimension that is orthogonal to current solutions. Notably, cRT Mixup and LDAM-DRW exhibit the most conflict scenarios and gain the most from PLOT. In fact, cRT Mixup even achieves competitive performance compared to the state-of-the-art under certain imbalance ratio settings, highlighting the efficacy of PLOT in ad- dressing optimization conflict problems. We also observe marginal effect of this augmentation, as GCL exhibits marginal improvement or even degra- dation, owing to the absence of significant optimiza- tion conflicts (see Fig.
Table Performance on large-scale Datasets.
Dataset
Method
Backbune
Overall
CE
ResNet-152
30.2
Decouple-7-non
ResNet-155
37.9
Balanced Softmax
ResNet-152
38.6
LADE
ResNet: 152
38.8
RSG
ResNet: 152
39.3
Places-LT
DisAlign
ResNet-152
39.3
ResL.T
ResNet: 152
39.8
ResNet-
40.6
cRT + Mixup
ResNet-152
38.5
cRT + Mixup + PLOT
ResNet- 152
41.0
MiSLAS
ResNet. 152
40.2
MISLAS + PTOT
ResNes- 152
40.5
CE
ResNeXt-50
44.4
Decouple-r-non
ResNet-50
46.7
Balanced Softmax
ResNeXi- 50
52.3
LADE
ResNeXt-50
$2.3
RSG
ResNcXt-50
51.8
DisAlign
ResNet-50
52.9
ResNeXt-50
53.4
ImageNet-LT
ResLT
ResNeXt-50
52.9
5.2 COMPARISONS WITH SOTA
LDAM-DRW + SAM
ResNet-50
53.1
GCL
ResNet-50
54.9
CRT Mixup
ResNet-50
51.7
We further evaluate the effectiveness of PLOT on large-scale datasets, i.e., Places-LT, ImageNet- LT. and iNaturalist, and compare it against main- stream methods in Table 3 Through augmenting two advanced baselines (cRT + Mixup and MiS- LAS), PLOT achieves state-of-the-ar performance. Specifically, our approach exhibits a substantial per- formance advantage over other DLTR models on Places-LT and iNaturalist, two recognized challeng- ing benchmarks due their high imbalance ratios.
Mixup
ResNeXt-30
54.3
MiSIAS
ResNet-50
52.7
MISLAS + PLOT
ResNet-50
53.5
CR
ResNet-50
61.7
Decouple-r-non
ResNet-50
65.6
Ralanced Softmax
ResNet-50
70.6
LADE
ResNet-50
70.0
RSG
ResNet-50
70.3
DisAlign
ResNet-50
70.6
iNa8-2018
ResLT
ResNet-50
70.2
LDAM-DRW SAM
ResNes-50
70.1
GCL
ReaNet-50
72.0
cRT Mixup
ResNet-50
69.5
cRT Mixup + PLOT
ResNet-50
71.3
MiSLAS
ResNet-50
71.6
5.3 ABLATION STUDY
MiSLAS + PLO:
ResNet 50
72.1
Our system comprises multiple components, and we aim to demonstrate their individual effectiveness. To this end, we conduct ablation studies and present the relationship between each component and the final performance in Table 4 Our results indicate that the proposed operations, i.c., temporal design (temp), variability collapse loss (var.) and anticipate worst-case optimization (anti.), can significantly enhance the system's performance.
Table 4: CIFAR10-LT when imbalance ra- tio is set 200.
Ablation studies on
cKT Mixup w/ temp w/ anti. w/ var. Acc.
73.06
76.02
77.79
78.99
5.4 OPTIMIZATION TRAJECTORY ANALYSIS
We capture the optimization trajectories of different cate- gories in LDAM-DRW + PLOT and present them in Fig. In the left figure, the gradient similarity is computed be- tween each category and the average gradient, while in the right figure, it is calculated between each category and the gradient aggregated by MOO. By comparison, the original LDAM-DRW approach exhibits dominance of head classes representation learning, resulting in deterioration of shared feature extraction. In contrast, our augmented version with PLOT demonstrates relatively comparable and stable similarities among categories, in- dicating the potential for effective extraction of shared features across categories.
Figure Gradient similarity with the ag- gregated gradient before after applying PLOT on LDAM-DRW.
(a) Gradient similari(b) Gradient similari- ties LDAM- DRW. ties PLOT.
Crdde Sy
802
200
400
D00
600
850
naningg Staps
8

---

Published as a conference paper at ICLR 2024
6 RELATED WORKS
6.1 DEEP LONG-TAILED RECOGNITION
Recent advancements in DLTR have been driven by three distinct design philosophies: (1) re-
balancing strategies on various aspects, (2) ensemble learning, and (3) representation learning.
In the first category, considerable efforts have been dedicated to re-sampling (Wang et al. 2019
Zang et al., 2021 Cai et al., [021] Wei et al. (222)). re-weighting (Park et al. [d.2 Kini al.
2021), re- m-rriinng (Feng et al. 2021; Cao et al., 2019, Koltchinskii & Panchenko 2002), logit adjustment (Menon et al. 2020; Zhang al.) 202 a), and information augmentation (Kim et al.) 2020, Yang & Xuj [020] Zang et 2021), etc. As anticipated, these approaches aim to manually re-balance the model by addressing sample number (through sampling and augmentation), cost sensitivity, prediction margin/logit, and other factors, thereby reducing the bias towards major classes, Ensemble learning- based approaches strive to leverage the expertise of multiple models. Generally, there several methods for aggregating these models. BBN (Zhou et al. (020)) and SimCAL (Wang it al. 2020a) train experts using both long-tailed and uniformly distributed data and aggregate their outputs. On the other hand, ACE (Cai et al. 2021), ResLT (Cui et al. 2022), and BAGS (Li et al., 2020) train experts on different subsets of categories. SADE (Zhang et al. 2022)) employs diverse experts, including long-tailed, uniform, and inverse long-tailed, and adaptively aggregates them using a self-supervised objective.
Recent efforts in representation learning have emerged in decoupling and contrastive learning, which employ distinct regimes to obtain general representations for all categories. Decoupling- -aaed methods (Kang al. [019] Zhong et al. 2021 Li et al., 2022) have shown that the representation learned via random sampling strategies is powerful enough, and additional effort devoted to the second stage, i.e., classifier adjustment, can help achieve advanced performance. A recent study (Liu et al., 2021b) has empirically found that contrastive learning-bar based methods are less sensitive to imbalance scenarios. Thus, these methods (Cui et al. 2021| Zhu et al., 2022) extract general representations via supervised contrastive learning and achieve competitive performance. Our work takes new approach to DLTR by developing a gradient coffict-averss. solution, which is almost orthogonal to current solutions and has been verified to be effective.
6.2 MOO-BASER MTL
Multi-task learning (MTL), particularly MOO-based MTL, has garnered significant attention in the machine learning community as a fundamental and practical task. MGDA-UB (Sener & Koltun 2018) achieves Pareto optimality by optimizing derived upper bound in large-scale MTL scenarios. PCGrad (Yu et al. (0200) mitigates conflict challenges by projecting gradients on the corresponding orthogonal directions. In contrast, CAGrad (Liu et al.) 2021a) develops provably convergent solution by optimizing the worst relative individual task and constraining it around the average solution. Additionally, EPO (Mahapatra Rajan (020) proposes a preference-guice
method that can search for Pareto optimality tailored to the prior. Our work represents the first attempt to integrate MOO into DLTR. We bridge the gap between MTL and DLTR and propose two improvements for further augmentation. Although evaluating PLOT under the MTL setting would be satisfactory choice, it is beyond the scope of this paper and is left for future investigation.
7 CONCLUSION
This paper have presented novel approach to bridging the gap between MTL and DLTR. Specifically, we proposed re-design of the MOO paradigm from structural to temporal, with the aim of addressing the challenge of optimization conflicts. To further ensure the convergence and generalization of the MOO algorithm, we optimized the derived MOO-based DLTR generation bound and seek flatter minima. Our experimental results demonstrated the benefits of injecting the Pareto property across multiple benchmarks. We hope that our findings provide valuable insights for researchers studying the integration of MOO and DLTR, which has been shown to hold great promise. Our future works lie in developing adaptive strategies apply MOO algorithm more efficiently.
9

---

Published as a conference paper at ICLR 2024
8 ACKNOWLEDGEMENTS
We thank anonymous reviewers for their valuable comments. This work was supported by the NSFC under Grants 61932017 and 61971390.
REPRODUCIBILITY STATEMENT
Further implementation details can found Section of the Appendix. Specifically, the supplementary material includes the attached code, which serves reference and provides additional information. The provided code demo consists of a prototype trained on the CIFARIO-/100-LT datasets.
REFERENCES
Santiago Cadena, Marissa A Weis, Leon A Gatys, Matthias Bethge, and Alexander Ecker. Diverse feature visualizations reveal invariances in early layers of deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 217 277, 2018.
Jiarui Cai, Yizhou Wang, and Jenq-Neng Hwang. Ace: Ally complementary experts for solving tailet recognition in one-shot. In Proceedings of the IÉFE/CVF International Conference on Computer Vision, pp. 112-121, 2021.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with aubel-distribution-awaa margin loss. Advances in neural information processing systems, 32, 2019.
Corinna Cortes, Mehryar Mohri, Javier Gonzalvo, and Dmitry Storcheus. Agnostic learning with multiple objectives. Advances in Neural Information Processing Systems, 33:20485-20495., 2020.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the JEEE/CVF international conference on computer vision, pp. 715-724, 2021.
Jiequan Cui, Shu Liu, Zhuotao Tian, Zhisheng Zhong, and Jiaya Jia. Reslt: Residual learning for
long-tailed recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Jean-Antoine Désidéri. Multiple-gradient descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Mathematique, 350(5-6):313-3) 2012.
Chengjian Feng, Yujie Zhong, and Weilin Huang. Exploring classification equilibrium in long-tailed object detection. In Proceedings of the IEEE/CVF International conference on computer vision, pp. 3417-3426, 2021.
Heshan Devaka Fernando, Han Shen, Miao Liu, Subhajit Chaudhury, Keerthiram Murugesan, and Tianyi Chen. Mitigating gradient bias in multi-objective learning: A provably convergent approach. In The Eleventh International Conference on Learning Representations, 2022.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awar minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
Hao Guo and Song Wang. Long-tailed multi-label visual recognition by collaborative training uniform and re- re-balanced samplings. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15089-15555 15098, 2021.
Jie Hu, Liujuan Cao, Tong Tong, Qixiang Ye, Shengchuan Zhang, Ke Li, Feiyue Huang, Ling Shao, and Rongrong Ji. Architecture disentanglement for deep neural networks. In Proceedings the JEFF/CVF international conference on compuler vision, pp. 672-681, 2021.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-taile tailed recognition. arXiv preprint arXiv: N010:0990009217 2019.
10

---

Published as a conference paper at ICLR 2024
Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major-to- minor translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13896-13905, 2020.
Ganesh Ramachandra Kini, Orestis Paraskevas, Samet Oymak, and Christos Thrampoulidis. Label- imbalanced and group-sensitive classification under overparamettrrizaton. Advances in Neural Information Processing Systems, 34:18970-1899 18983, 2021.
Vladimir Koltchinskii and Dmitry Panchenko. Empirical margin distributions and bounding the generalization error of combined classifiers. The Annals of Statistics, 30(1):1-50, 2002.
Mengke Li, Yiu-ming Cheung, and Yang Lu. Long-tailed visual recognition via gaussian clouded logit adjustment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2022.
Recognition, pp.
Yu Li, Tao Wang, Bingyi Kang, Sheng Tang, Chunfeng Wang, Jintao Li, and Jíashi Feng. Overcoming classifier imbalance for long-tail object detection with balanced group softmax. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10991-11000, 2020.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:18878-1888 2021a.
Hong Liu, Jeff Z HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. arXiv preprint 1 2021b.
Xuantong Liu, Jianfeng Zhang, Tianyang Hu, He Cao, Yuan Yao, and Lujia Pan. Inducing neural collapse in deep long-tailed learning. In International Conference Artificial Intelligence and Statistics, pp. 11534-1155 1154.. PMLR, 2023.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2019.
Yanbiao Ma, Licheng Jiao, Fang Liu, Yuxin Li, Shuyuan Yang, and Xu Liu. Delving into semantic scale imbalance. The Eleventh International Conference on Learning Representations, 2023.
Debabrata Mahapatra and Vaibhav Rajan. Multi-task learning with user preferences: Gradient descent with controlled ascent in pareto optimization. In International Conference on Machine Learning, pp. 6597-6607. PMLR, 2020.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint rrXiv:2007.07314 2020.
Seulki Park, Jongin Lim, Younghan Jeon, and Jin Young Choi. sefuunce-balancee loss for imbalanced visual classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 735-744, 2021.
Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, et al. Escaping saddle points for effective generalization on class- innslimmallneeed data. Advances in Neural Information Processing Systems, 35: 22791-22805, 2022.
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long-tailed visual recognition. Advances in neural information processing systems, 33:4175-4186, 2020.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018.
Saptarshi Sinha and Hiroki Ohashi. Difficulty-net: Learning to predict difficulty for long-tailed recognition. In Proceedings of the IEFF/CVE Winter Conference on Applications of Computer Vision, pp. 6444-6453 2023.
Jingru Tan, Changbao Wang. Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-taile object recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11662-11671, 2020.

---

Published as a conference paper at ICLR 2024
Jingru Tan, Bo Li, Xin Lu, Yongqiang Yao, Fengwei Yu, Tong He, and Wanli Ouyang. The equalization losses: Gradient-driven training for long-tailed object recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. -7788 2018.
Jialei Wang, Peilin Zhao, and Steven CH Hoi. Cost-sensitive online classification. IEEE Transactions on Knowledge and Data Engineering, 26( 26(10):2425-2)) 2-238, 2013.
Jianfeng Wang, Thomas Lukasiewicz, Xiaolin Hu, Jianfei Cai, and Zhenghua Xu. Rsg: A simple but effective module for learning imbalanced datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3784-3793 2021.
Tao Wang. Yu Li, Bingyi Kang, Junnan Li, Junhao Liew, Sheng Tang, Steven Hoi, and Jiashi Feng. The devil is in classification: simple framework for long-tail instance segmentation. In Computer Aision-ECCY 2020: 16th European Conference, Glasgow: UK, August 23-28. 2020, Proceedings, Part XIV 16, pp. 728-744 Springer, 2020a.
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu. Long-tailed recognition by routing diverse distribution-and aware experts. arXiv preprint arXiv:2010.01809, 2020b.
Yiru Wang, Weihao Gan, Jie Yang, Wei Wu, and Junjie Yan. Dynamic curriculum learning for imbalanced data classification. In Proceedings of the IEEE/CVF international conference computer vision, pp. 5017-5026, 2019.
Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A class-rebalaneing self-training framework for imbalanced semi-supervised learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10857-10866, 2021.
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning. Advances in neural information processing systems, 33:19290-19301, 2020.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824-5836, 2020.
Yuhang Zang, Chen Huang, and Chen Change Loy. Fasa: Feature augmentation and sampling adaptation for long-tailed instance segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3457-3466, 2021.
Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. Distribution alignment: A unified framework for long-tail visual recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2361-2370, 2021a.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. arXiv preprint arXiv:2 10.04596, 2021b.
Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition. Advances in Neural Information Processing Systems, 35:34077-340090, 2022.
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recogni- tion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16489-164988, 2021.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for
visual recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9719-9728, 2020.
12

---

Published as a conference paper at ICLR 2024
Zhipeng Zhou, Lanqing Li, Peilin Zhao, Pheng-Ann Heng, and Wei Gong. Class-conditional sharpness-awee aware minimization for deep Conference Computer Vision and Pattern Recognition (CVPR), pp. 3499-3509, June 2023.
recognition. In Proceedings of the IEEE/CVF
Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Balanced contrastive learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6908-6988800 1018-691777 2022.
13

---

Under review as a conference paper at ICLR 2024
APPENDIX FOR: PARETO DEEP LONG-TAILED RECOG- NITION: A CINFLICT-AVERSS SOLUTION
Anonymous authors Paper under double-blind review
CONTENTS
Theoretical proofs
Proof of Theorem
Proof of Theorem 4.2
2
Details of Implementations
DLTR Methods
MOO-base based MI MethodS
2.3
Datasets
5
Open Long-
Long-Tailed Tailed Recognition Evaluation
4
Additional Experiment Results
Identify Optimization Conflict under Various Conditions
Gradient Conflicts the Early Training Stage
Frequency vs. Loss Value
4.4
Gradient Conflicts Status of Dynamic Re- weighting Approach
4.5
Gradient Norm Examination
Gradient Similarity Examination after PLOT augmentation
Apply PLOT on Different Layers
4.8
Gradient Conflicts Status under the Balanced Setting
Limitation
6
Computational Complexity and Scalability Analysis
6.1
Computational Complexity
6.2
Scalability
7
Visualizations
10
7.1
Feature Representation Analysis
10
Loss Landscape of DLTR models
10
Embedding Visualization
11
7.4
Canonical Correlation Analysis
11
8
Detailed Results Mainstream Datasets
12

---

Under review as a conference paper at ICLR 2024
THEORETICAL PROOFS
1.1 PROOF OF THEOREM 4.1
Theorem 1. (MOO-based DLTR Generalization Bound) If the loss function ly belonging to kih
category is Mg-Lipschitz, and (a, y). (x',y') E X Y, vh E H: [l[h(x)) y [h(==') w'll I Dy, assume MdDx bounded by M, then for any e > and O, with probability at least the following inequality holds for vh E H and tie E W:
log 8
(h) S (h) 2 Σ wk W
Proof. Vo E W and sample $ = {(x1, y1,,..., (ÆN, yN }. let ((S) = suPhEM Lw (h) (h). And assume S' is another sample set that contain only one point (a', y²) different from S. Thus, we have
((S") - ((S) = supnen(Cwn (h) (h)) SuPnEH (h)) Lu (h))
I supnEM! C (h) (h) (h) Lu (h))
= suPhEr Lw (h) - E. (h)l
K
K
1
1
suPhE7
(a v()
(x yi)
mi
me
K
1
suphe7
Wk
[1(2%) v/t) 1(x0e, y:)l
mi
K
= =SUPheH
Wk
[1 v/1) -((04, (4))
m&
k=1
K
1
< su1hee7
Wk Mc [1[h(z), 31 [h(x'),3/7
k=
K
S <Supheh
Wk
k=1
K
< suphen
M
k=
According to McDiarmid's inequality, for any 8 with probability at least 8 for any h H. we have:
log x
(h) S (h) + E Cu (hi +
Σ
wk
k=1
K
log
$ (h) + (9 w)
M
2
1.2 PROOF OF THEOREM 4.2
Pseudo Code of CAGrad:
2

---

Under review as a conference paper at ICLR 2024
Algorithm 1: Training Paradigm of CAGrad
Input:
Initial model parameter 9, differentiable loss functions are L,(0), i E [N] constant E [0, 1] and learning rate Rt.
Output: Model trained with CAGrad
while not converged do
At the tu optimization step, define go = K Ex VoLi(0(-1) and Φ c² l19oll.
Solve
min WEW
F(w)= -N99 + v |1llll where & = =2
0:12 n 9w)
Update O2 - 0 go
Theorem 2. (Convergence of CAGrad in DLTR) With a fix step size and the assumption of H-
Lipschitz on gradients, i.e. ( V (6)) & H ||e 0'|l for i = 1, 2, ... K. Denote d' (0,)
as the optimization direction of CAGrad at step I, then we have:
[(0c+1) - L(0,) -/ [1 c²) (0:))
(Ha 1) d (0.)ll
Proof.
[(0++1) C(0,) C(0, ad (0.)) ((),)
S (0.) T d' (0.) + Ha²
|d (0+))l
2
S 2
a (190 + ||d (0+) (000((s))) d² d0=))l
+ Ha²
|| (0,)
2
2 (l96 - (0.) d" d()))) + Ha²
||d (0.)
ld" (0.))
(1 c²) (0:) (Ha 1) | (0:))l
2
DETAILS OF IMPLEMENTATIONS
We conduct all experiments according to their publicly released code if applicable, please refer to these code for more details. Our early stop hyper-parameter E is selected from {10, 30, 50, 80), while anticipating worst-case optimization hyper-paramete p is searched over (1.0e-3, 1.0c-4, 1.0e-5).
2.1 DLTR METHODS
LDAM-DRW. LDAM-DRW re-balances the model via logit adjustment. enforces the theoretical derived margins that is class rrequency-relatee to achieve cost-sensitive learning. Its publicly released code can be found at htttss://githhu www.//oo/oiiiiiiccoooeeeo
DRN
Balanced Softmax. Balanced Softmax is another cost-sensitive learning approach, which ce-formulate softmax function in with the combination of link function and Bayesian infer- ence. Its publicly released code can be found at htttp://ggituue com/ jiawei-ren/ BalancedMet SawnnceeeeeaaSotttaaaaeleeeee.
M2m. M2m takes advantage of generating adversarial samples from major to minor classes and thus com/
re-balance via augmentation. Its publicly released code can be found at
alinlab/ M2m
cRT + Mixup. cRT is a milestone decoupling method, which re- re-trains the classifier with balanced sampling strategy in the second stage. Despite the official implementation is available, it does not
3

---

Under review as a conference paper at ICLR 2024
include version that utilizes ResNet-32 and is evaluated on CIFAR SFFERRO-/CIFARIOO the mainstream protocols. Therefore, we have re-implemented the method and achieved similar performance that reported in GCL. In our implementation, decay strategy as GCL, which involves multiplying the learning rate by 0.1 after the 160 and 80th epochs.
which
have adopted the same learning rate
MiSLAS. MiSLAS follows the decoupling regime and adopts a class-frequency related label smooth- ing operation achieve both the improvement of accuracy and calibration. Its publicly released code can be found at /
com dv research www.aaaeaaaareeccaeeaaaaaaaaaaae
SLAS
GCL. Likewise, GCL is also pioneer two-stage method, which observes the problem of softmax satu- ration and proposes to tackle it by Gaussian perturbation of different class logits with varied amplitude. Its publicly released code can be found htt /tttt///gtttbbb. com Keke 921 /CLLLoss
difficultyNet.. As dynamic re-balancing method, difficultyNet employs meta learning to learn the adjustment of class re- weiietntting from logits. Its publicly released code can be found https
//github.
Net
(2020) takes care of both representation learning and classifier learning by equipping with novel cumulative learning strategy on two branches. Its publicly released code can be found at https: git..b....uu..ggggg www.m/gvmmmmiieccoc/mm/ec rww.errerrchheee
BBN. BBN Zhou
PaCo. PaCo Cui et al. (2021) introduces a set of parametric class-wise learnable centers to re- balance from an optimization perspective, addressing the bias on high-frequency classes su- pervised contrastive loss. Its publicly released code can be found at httts:://gtthtt com/
BCL. BCL. Zhu et al. (2022) proposes class-averaging and class-complement methods to help form a regular simplex for representation learning in supervised contrastive learn- ing. Its publicly released code can be found at httts:://gitae com/ htt//FaamieZhu..
2.2 MOO-BASED MTL METHOODS
MGDA. MGDA is a classical baseline for MOO-based MTL. This approach is particularly appealing due to its ability to guarantee convergence to Pareto stationary point under mild conditions. Building upon this foundation, MGDA-UP introduces an upper bound on the multi-objective loss, aiming to optimize and thereby achieve the Pareto optimal solution. practice, suggests task weighting based on the Frank- Wolfe algorithm (Jaggi 2013). We eonluceeaallutiion with the re-implementation in the publicly released code of CAGrad.
EPO. Different from the general MOO-based MTL, EPO provides preference--pecific MOO frameworks, which can effectively finds the expected Pareto front from the Pareto set by carefully controlling ascent to traverse the Pareto front in a principled manner. Generally, the re-balancing purpose also requires a preference for MOO. Its publicly released code can be found at https //githube com/ hetmm/ddmptr//// wbwptr/EPOSeaaae.... EPOOSeachh
CAGrad. CAGrad improves MGDA mainly by the ideas worst-case optimization and convergence guarantee. It strikes a balance between Pareto optimality and globe convergence by regulating the combined gradients in proximity to the average gradient.:
min g d dER"WEW"
s.t. l|t.I/d goll s |lgooll
where d represents the combined gradient, while go denotes the averaged gradient, and e is the hyper- parameter. publicly released code can be found at //ftttss///gthhu.. www.e/owao/aaaaaeee
CAGrad
Why we choose CAGrad as the baseline? CAGrad widely recognized as a robust baseline in MOO -based MTL. In contrast to MGDA, which consistently favors individuals with smaller gradient norms, CAGrad achieves a delicate balance between Pareto optimality and global convergence. This unique characteristic allows CAGrad to preserve the Pareto property while maximizing individual progress. Conversely, EPO necessitates manual tuning of the preference crucial role in its performance but proves challenging to optimize in practical scenarios, particularly
which plays
4

---

Under review as a conference paper at ICLR 2024
for classification tasks with large number of categories. In comparison, CAGrad requires less effort in terms of hhper--arameee parameter tuning.
2.3 DATASETS
CIFARIO.CIEARROOLII CIFARIO-CCIEARIOG is subset of CIFARIO/CIFAR 100, which formed by sampling from the original 50,000 training images to create a long-tailed distribution. In our evaluation, set the imbalance ratio 3 Pmez (200. 100, 50), where Tymar and Amin represent the sample numbers of the most and least frequent classes, respectively.
7miin
Places-LT & ImageNet-LL. Places-LT and ImageNet-LT are long-tailed variants of Places-365 and ImageNet, respectively. Places-LT comprises total of 62.5K training images distributed across 365 classes, resulting in an imbalance ratio of 996. Similarly, ImageNet-LI consists of 115.8K training images spanning 1000 classes, with imbalance ratio of 256.
iNaturalist 2018. iNaturalist 2018 is a naturally occurring long-tailed classification dataset that comprises 437.5K training images distributed across 8142 categories, resulting in imbalance ratio of 512. In our evaluations, we have followed the official split.
OPEN LONG-TAILED RECOGNITION EVALUATION
To further demonstrate the robustness of the learned representations by PLOT, conduct an evaluation known as open long-tailed recognition (OLTR) (Liu et al.) 2019) on Places-LT and ImageNet-L.T datasets. The results are presented in Table I Based on the comparison of F-measures, our method achieves state-of-the-art performance in OLTR. It is worth noting that we employ cosine similarity measurement between incoming representations and prototypes proposed by (Liu et al. 2019), which allows us compete favorably against bells and whistles OLTR methods (e.g.. LUNA (Cai et al. ). This highlights the superiority of the mechanism proposed by PLOT for representation learning.
Table 1: Open long-tailed recognition on Places-LT and ImagetNet-LT.
memassue f
Lifed
Focal
Range
OpenMax
CC-SAM
Mixup
ImageNet-LT 0.295
0.374
0.371
0.373
0.368 0.458
0.474
0.525
0.579
0.552
0.563
Places LT
0.366
0.459
0.453
0.457
0.464
0.486
0.491
0.510
0.516
ADDITIONAL EXPERIMENT RESULTS
4.1 IDENTIFY OPTIMIZATION CONFLICT UNDER VARIOUS CONDITIONS
Generally, the optimization objectives of samples from different classes tend to exhibit some conflicts, they need learn class-specific features in addition the shared features across classes. In the case of a balanced training set, this conflict does not significantly impact the simultaneous optimiza- tion of individual classes. However, in an imbalanced scenario, the model optimization becomes dominated by the majority class, exacerbating this conflict issue. Consequently, the performance of the minority class compromised favor of optimizing the majority class. This outcome hinders the effective learning of sahagir features. Importantly, this problem is intuitively independent of hyperparameters, optimizers, and other related factors.
To validate the aforementioned hypothesis, we have conducted an extensive investigation into the gradient conflict and dominated categories issues by thoroughly examining LDAM-DRW across various experimental settings. These settings encompassed diverse mini-batch sizes, learning rates, and optimizers. The meticulous presentation our findings is shown in Fig. D and Fig. 2 Notably, Fig. provides snapshot of the instantaneous gradient conflict status at early stage, while Fig 2 illustrates the continuous gradient similarity status throughout the optimization process. It is important to emphasize that Figs. 2 (a) and (b) are presented on a larger scale in their axes, resulting in a less apparent discrepancy. However, it should be noted that these subfigures do not differ significantly from the other subfigures when placed the same scale.
5

---

Under review as a conference paper at ICLR 2024
(a) SGD-1228-01 (b) Adam- Adam-128-0.1 (c) SGD-256-0.1 (d) SGD-512-0.1 (e) SGD-IGSDS 128-0.01 (1) 128--128-1 -e-3
Figure 1: Gradient conflict status of LDAM-DRW under various conditions. Each sub-figure is
named as (Optimizer) (Batch Size) (Learning-Ratee
(a) Adam- Adam-128-0.1 128-0.1 (b) Adam- Adam-128-1c- 128-1c- (c) SGD-256-0.1 (d) SGD-512-0.1 (c) 12DD-1280001 (f) SGD-128-1c-3
3
Figure 2: Continual gradient similarity status of LDAM-DRW under various conditions. Each
sub-figure named as (Optimizen (Batch-Size) Size) (Gearning-Rata) Rate).
As anticipated, have observed the presence of the gradient conflict and dominated categories issues in all tested scenarios. These findings significantly contribute to validating the universality of the gradient conflict issue. If deemed applicable, we fully committed to providing additional evidence and conducting further analysis in the updated version to reinforce the robustness of our findings.
4Ko
(a) ERM
(b) LDAM-DRW
(c) Bal. Softmax
(d) M2m
(e) MiSLAS
(f) GCL
(g) ERM
(h) LDAM-DRW (i) Bal. Softmax
(j) M2m
(k) MiSLAS
(l) GCL
Figure 3: Gradient conflicts among categories at step 50 (a-f) and step 84(g-l).
4.2 GRADIENT CONFLICTS IN THE EARLY TRAINING STAGE
Here we further present the existing of optimization conflicts at different steps in Fig. As depicted, such conflicts remain for long time in the early stage.
4.3 FREQUENCY VS. LOss VALUE
In Fig. we present our observations of class-wise weighted loss, denoted as Ba-T(a". vx)) the norms Bf / and T(= y Our results indicate that BalancedSoftmax, which functions as a re-balancing method by adjusting the loss function, is still dominated by class frequency, i.c., B./ B. Conversely, LDAM- -DD exhibits a balanced weighted loss for both head and tail classes, thereby demonstrating the effectiveness of its re-balancing strategy.
compare

---

Under review as a conference paper at ICLR 2024
Sleed
Woggted Log
Wegghee Lowe
2.3
Weg stoc Lpos
0.2
200
400
850
810
402
592
407
810
Stops
Thaningg Stips
(a) cRT Mixup
(b) LDAM-DRW
BalancedSoftmax
Figure 4: wlass-wise weighted loss comparison on CIFARIO-LT when imbalance ratio is set as 200 and batch size as 64.
4.4 GRADIENT CONFLICTS STATUS DYNAMIC RE- WEIWEIGHTING APPROACH
As form of dynamic re-weighting method, further investigate the optimization conflicts associated with other dynamic weighting approaches to showease distinct this regard, we employ difficul-
Figure 5: Optimization conflicts of diff. Net.
research perspective. tyNet the baseline for conducting a verification exper- iment, and the results are depicted in Fig. 5 It is evident that difficultyNet fails to address optimization conflicts during its early training stage, primarily due to its inherent lack of design nature.
(a) Epoch 0
(b) Epoch 30
4.5 GRADIENT NORM EXAMINATION
In this section, we expand our analysis by conducting a thorough investigation into the phenomenon of gradient norm domination within mainstream DLTR approaches. To quantify this domination, calculate the mean gradient the corresponding category and present the results our examination in Figure 6 aff). As depicted, all the examined approaches demonstrate consistent pattern, mirroring the tendency observed in Figure of the main text. Specifically, we observe that the tail class extremely under-optimized, indicating the explicit dominance of specific categories during the optimization process. These compelling findings provide strong support for the motivation underlying our proposed method, which aims to effectively mitigate such domination and its associated adverse effects. For comparative purposes, we also provide the PLOT-augmented gradient norm examinations in Figure gl)). As expected, the results reveal that no individuals (categories) exhibit explicit domination during the early stage of representation learning.
Stepe
Stepe
Tumig Sena
Takkinccm 2001
Tento
Tak
(a) ERM
(b) LDAM-DRW
(c) Bal. Softmax
(d) M2m
MiSLAS
(l) GCL
100
1:00
3mm
400
Tanil Sepa
(g) ERM
(h) LDAM-DRW
(i) Bal. Softmax
(i) M2m
(k) MiSLAS
()) GCL
Figure 6: Gradient norm examination for mainstream DLTR approaches (a-f). 'Norm Ratio" is
calculated between the individuals and the mean gradient. (g-l) the PLOT-augmented ones.
7

---

Under review as a conference paper at ICLR 2024
4.6 GRADIENT SIMILARITY EXAMINATION AFTER PLOT AUGMENTATION
To substantiate the effectiveness of the PLOT, which facilitates simultaneous progress for all in- dividuals, compute the cosine similarities between individuals and the gradient derived from MOO. These cosine similarities are then compared with the corresponding vanilla results presented in Figure 2 of the main text. The comparative results are depicted in Figure 7 As observed, the DLTR methods augmented with PLOT exhibit more balanced cosine similarity, indicating that all individuals achieve more equitable progress and improvements.
Takig
Stepe
Tamin Stipe
(a) FRM
(b) L.AM-DDW
(c) Bal. Softmax
(d) M2m
(c) MiSLAS
(f) GCL
Figure 7: Gradient similarity examination for PLOT -o0T-augmetted mainstream DLTR approaches.
"Cosine Similarity' is calculated between the individuals and the MOO derived gradient.
4.7 APPLY PLOT ON DIFFERENT LAYERS
Figure 8: Impact of different layers.
MOO algorithms recognized be computationally intensive, particularly when the number of tasks scales up, To address this issue, we propose applying PLOT to select specific layers rather than the entire model. In this study, we examine the impact of different layers to demonstrate the robustness of our approach. Specifically, we apply PLOT to different layers, i.e., layerl, layer2, and layer3 in ResNet-32, and present their final performance in Figure 8 Our results indicate that the last layer is the optimal choice across various evaluations.
74
Imbatance
4.8 GRADIENT CONFLICTS STATUS UNDER THE BALANCED SETTING
We investigate the gradient conflict and similarity status in a balanced setting to motivate the integration of MOO and DLTR. To this end, we train ERM with the full (roughly balanced) CIFAR10 dataset for 200 epochs and achieve final accuracy of 92.84%. We record the class-wise gradient and compute their similarities. As shown in Fig[9] gradients among categories also exhibit conflict cases but play roughly equal role during optimization
Figure Gradient conflicts and similar- ity of ERM.
LIMITATION
The limitations of PLOT be attributed to main fac- tors. Firstly, the approach incurs additional computational overhead due multiple gradient backpropagation and MOO. To address this issue, we propose applying PLOT to select layers rather than the entire model, which serves an efficient alternative. Additionally, we randomly sample small of categories to participate in MOO for large- scale datasets. Secondly, the early stopping mechanism currently controlled by a dlyper-paramete which introduces additional tuning complexities. To simplify this process, select the paprm-errrettrr from fixed set of values.
(a) Gradient conflicts.
(b) Gradient sim.
We utilize WideResNets (Foret et al. 2020 the backhone network without ShakeShake regularization additional augmentations. Therefore, our reported result slightly lower than the generally reported accuracy. Notably, our focus is on observing the class-wise gradient status rather than realizing a re-implemennation. Whiile this conclusion straightforward, it is worth comparing with DLTR results.
8

---

Under review as a conference paper at ICLR 2024
COMPUTATIONAL COMPLEXITY AND SCALABILITY ANALYSIS
6.1 COMPUTATIONAL COMPLEXITY
As noted in the Limitations section, our method is more complex than its baselines. However, we would like emphasize that our approach only applies MOO to subset of the model parameters in a mini-batch, which contains fewer classes. This results a lower time cost than expected. Additionally, we only apply MOO in the early stage of DLTR training for shared feature extraction. For large-scale datasets such as ImageNet-LT, we further adopt a class sampling strategy to reduce the computation cost. We believe that these optimizations help to mitigate the complexity of our approach while maintaining its effectiveness in addressing the long-tain tailed recognition problem. Here we provide simple comparison of the time cost of running on Tesla T4 GPU:
Table 2:
Training time cost comparison running on Tesla T4.
Method
CE
LDAM DRW
cRT Mixup
cRT
Mixup + ELOT
LDAM LDAM-DRW FLOT
GPU Time (b)
0.62
0.68
0.65
0.83
0.86
6.2 SCALABILITY
As previous research has indicated (Cadena et al., 2018 Hu et al. [222]] Zimmermann et al. 2021 Liu et al., 2019), the parameters of backbone layers tend be more stable the later stages of training. Therefore, our temporal-style MOO learning paradigm is designed to address the gradient conflict problem that occurs during the early stages of representation learning. This approach enables compatibility with mainstream DLTR models. Here, provide examples of how approach can be integrated with Decoupling using pseudo code, as shown Algorithm (Please refer Eqns. 2, 4 in the maintext):
Algorithm 2: Representation Training Paradigm of Decoupling + PLOT
Input: Training Dataset ~ pa y) P. (x
Output: Model trained with PLOT
Stage SFE:
Initialize 0 raanooly while epoch < do
foreach baich B; in S do
Compute empirical loss I.s with B; and obtain its gradient gs
Perturb with according to Eqn. 3 and Eqn.
Compute the perturbative loss CSAM and the variability collapse loss Lue according to Eqn. 2.
Estimate the class class- specific gradient G {g1. 92. with respect
Lmoo LSAM + Lye
Update by solving Eqn. 6: 0 +- 0 a (go ||aw|| 9w
1/2
Stage TSO:
while epoch > do
foreach batch B, S do
Computing empirical loss Ls with B;
Update 0: 0 0 (OL

---

Under review as a conference paper at ICLR 2024
7 VISUALIZATIONS
7.1 FEATURE REPRESENTATION ANALYSIS
More class-specific Hessian spectral analysis are provided in Figs. 10 11 conclusion in the main text, i.e. PLOT leads to flatter minima for each class, thereby reducing the risk of getting stuck at saddle points.
which align with the
Amaa $1.19
Araa 69.95
Assa 117.67
Arpax 127.42
-7-
Amic 4.94
Avate 8.56
Ania -14.35
Arsts -17-7-7-00 17.46
33
10
152
150
(a) Class 1
(b) Class 2
(c) Class 3
(d) Class 4
204.61 Aruuz
290.67 Arsax
Arsax 261.15
Aesax
Avazn
24.07
Arais
36.00
Arais 43.63 30.60
Aeste
85.29
250 Liga Apenestton
27
135
140
200
240
220 0iper//l0a
140
200
252 Alper
.25
(c) Class 5
(f) Class 6
(g) Class 7
(h) Class 8
Figure 10: Hessian spectrum analysis of vanilla cRT + Mixup.
Amza
30.50
Amaax 74.69
Aruzz 83.85
Arsce 03.0
Aruin 3.72
Araus 7783
Aramn
Apin 11.50
Demvly
(a) Class 1
(b) Class 2
(c) Class 3
(d) Class 4
Aveay 161.51
Aeso 223.23
Anay 192.56
409.00 Avain 63.87
Avsin - 23.86
Avan 30.89
Aysic -39.20
(e) Class 5 (f) Class
(g) Class 7
(h) Class 8
Figure 11: Hessian spectrum analysis of PLOT- cRT Mixup.
7.2 LOSS LANDSCAPE OF DLTR MODELS
To further illustrate the unattainable small value of H in the DLTR models, we have included additional loss landscape visualizations in Fig. [2] These visualizations reveal that the tail classes of the models exhibit sharp minima, with the exception of M2m. Additionally, we observe that PLOT displays flat minima in both head and tail classes.
https://github. http:///htthh..ccommaaiiiiaaaa// l-
longTail
10

---

Under review as a conference paper at ICLR 2024
Law Chrtoass anound Trased Foold
Less Conture smund Thahrd Mede
Contont sound Traired Moda
Lotn Conmure Traned Model
Conture amund Tranne
2030
24
13
20
30
2
10
20
(a) Head of ERM
(b) Head of LDAM- (c) Head of M2m
(d) Head of MiSLAS
(e) Head of PLOT
DRW
Traved
343
333
13
28
(1) Tail of ERM
(g)
Tail of LDAM- (h) Tail M2m
(i) Tail of MiSLAS
(i) Tail of FLOT
DRW
Figure 12: Loss landscapes of head and tail classes ERM, LDAM-DRW, M2m, MiSLAS, and PLOT.
7.3 EMBEDDING VISUALIZATION
In addition, we present visualizations of the extracted embeddings from both the vanilla and PLOT- augmented approaches by projecting them into 2D plane using t-SNE Van der Maaten & Hinton (2008). The corresponding visualizations can be found in Figure L3 (a) As anticipated, the tail classes of the cRT Mixup PLOT approach exhibit increased separability compared to the vanilla approach. This observation suggests that the incorporation LOT enhances the representation of all categories, intended.
PLOT
coof value
0.0
12.5
CCA Coef
(a) Mixup.
(b) cRT + Mixup PLOT
(c)
Figure 13: (a) and (b) are the t-SNE visualization results. (c) Representation similarity between head and tail classes.
7.4 CANONICAT. CORRELATION ANALYSIS
To further investigate the shared feature representation across categories in the early stage, we examine the Canonical Correlation Analysis (CCA) scores (Raghu et al. 2017) of representations between head and tail categories CIFAR1O-LA as learned by both the standard and PLOT enpowered DOAM-DRWW. As illustrated in Fig. D (c), the PLOT augmented version exhibits greater degree of similarity among categories in comparison to the conventional model, thereby substantiating the efficacy of our conflict-averse solution.
11

---

Under review as a conference paper at ICLR 2024
8 DETAILED RESULTS ON MAINSTREAM DATASETS
In order to demonstrate the generalization of P PLOT and its impact on different subsets, we provide more detailed results in this section. Specifically, compare PLOT with two state-of-the-art DLTR approaches, namely BBN and PaCo, by augmenting them with PLOT. The consistent improvements observed in Table indicate that PLOT can effectively enhance various types DLTR approaches. Furthermore, present detailed results on large-scale datasets in Table which empirically illustrate that PLOT successfully augments medium and tail classes without significantly compromising the performance of major classes.
Dalasel
Methed
Backbone
Many
Medium
Few
Overall
ResNet-112
45.7
27.3
8.20
30.2
Decouple-T-oom/ Kang et
2019
ResNet-152
37.8
40.7
31.8
37.9
Balanced Softmax
ReaNct-152
30.5
38.6
LADE et 204
ResNet-152
42.8
39.0
31.2
38.8
PLaces-ITT
Wany
ResNet-152
39.3
DisAlrn Zmning 00
202
ResNet-152
40.4
42.4
30.1
39.3
Cur
ResNet-152
39.8
31.4
39.8
GC1 [b at
ResNet-152
40.6
cRT + Mixup Kang ot
2019
RosNet-132
44.1
38.3
27.1
38.1
Mixup P4:0
MiSLAS DhssiA Zhong el al.
2021
ResNet- 152
39.2
43.2
36.5
40.2
MiSI.AS
36.9
CB
ResNcXt-50
65.9
37.5
7.7
44.4
Decouple-7-some
ReaNet-50
56.6
27.4
46.7
Balanced Soltmaa
Ken et
RexNeXt-50
64.1
48.2
33.4
52.3
LADE
RSG Wang e
ResNeXt-50
63.2
48.2
32.2
51.8
DisAlna Zhang
ReaNet-50
61.3
52.2
31.4
52.9
ImageNet-LTT
ResLT Cui al
2022
ResNeXt-50
RosNcXt-50
63.0
62.7
50.5
52.1
35.5
31.4
53.4
52.9
LDAM-DRN
RangRaai
2022
GCL Li al 2022
ResNet-50
54.9
Mixup Kang et
KesNeXXi300
cRT Mixup + PL0T
ResNeXt-50
62.6
52.0
39.4
54.3
MiSLAS Zhong et 2021
ResNet-50
61.7
51.3
35.8
52.7
MiSLAS PT.T
ResNet-50
61.4
37.5
53.5
BCL et al. 2022
RosNet-50
63.7
53.7
37.3
56.0
BCL
ResNet-50
38.8
57.2
ReaNet-50
72.2
57.2
61.2
Decouple-r-nom
2019
ResNet-50
65.6
65.3
65.9
65.6
Balanced Softmax Kea et
2021
RosNot-50
70.6
LADE/Hong
ReaNet-50
RSG Wang au 2022
ResNet.50
70.3
BicAlinnZhargg
ResLT 2022
ResNet-50
70.2
INat-2018
LDANFDRA FSAM Rangwani et
2022
ResNet-50
64.1
70.5
71.2
70.1
Gi (2022)
ReaNet-50
72.0
cRT + Mixup Kang ot al. 2019
ReaNet-50
74.2
71.1
68.2
70.2
Mixup
ResNet-50
MiSIAAS Shiii.AShhhnngee
Zhong et al.
2021
ResNet.50 50
73.2
72.4
70.4
71.6
MiSLAS PT.01
ResNet-50
73.1
71.2
Table 3: Detailed Top-1 Accuracy on Places-LT, ImageNet-LT, and iNaturalist-201s
12

---

Under review as a conference paper at ICLR 2024
Table Top-1 Accuracy on CIFAR datasets.
CIFARIO-LT
CIFAR 100-LT
Imb.
200
100
200
100
LDAM-DRW LDAM-DRW PLOT
71.38
77.71
81.78
36.50
41.40
74.32
78.19
82,09
37.31
42.31
47.04
73.43
35.81
45.73
M2m + PLOT
74.48
78.42
81.79
38.43
43.00
47.19
cRT Mixup
73.06
79.15
84.21
41.73
45.12
50.86
eRT + Mixup + PLCT
78.99
80.55
84.58
43.80
47.59
51.43
Logit Adjustment
78.01
43.36
Logit Adjustment PLOT
79,40
44.19
BBN
73.52
77.43
80.19
36.14
39.77
45.64
MiSLAS
76.59
81.33
85.23
42.97
47.37
51.42
MISLAS PLOT
77.73
81.88
85.70
44.28
47.91
52.66
GCL
79.25
82.85
86.00
44.88
48.95
$2.85
GCL FLOT
80,08
83,35
85,90
45.61
49.50
53.05
PaCo
47.28
$1.71
55.74
PaCo + PLOT
47.75
52.60
56.61
REFERENCES
Santiago Cadena, Marissa A Weis, Leon Gatys, Matthias Bethge, and Alexander Ecker. Diverse feature visualizations reveal invariances in early layers of deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 217-232, 2018.
Jiarui Cai, Yizhou Wang, Hung-Min Hsu, Jenq-Neng Hwang. Kelsey Magrane, and Craig S Rose. Luna: Localizing unfamiliarity near acquaintance for open-set long-tailed recognition. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 131-139, 2022.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF international conference computer vision, pp. 715-724, 2021.
Jiequan Cui, Shu Liu, Zhuotao Tian, Zhisheng Zhong, and Jiaya Jia. Reslt: Residual learning for
loig-tailed recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010001412. 2020.
Youngkyu Hong. Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang.
Disentangling label distribution for loig-tailed visual recogmition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6626-6636, 2021.
Jie Hu, Liujuan Cao, Tong Tong, Qixiang Ye, Shengchuan Zhang, Ke Li, Feiyue Huang, Ling Shao, and Rongrong Ji. Architecture disentanglement for deep neural networks. In Proceedings of the IEEE/CVF international conference on computer vision. pp. 672-681, 2021.
Martin Jaggi. Revisiting frank- wolfe: Projection free sparse convex optimization. In International
conference on machine learning, pp. 427 435. PMLR, 2013.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis, Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.092177 2019.
Mengke Li, Yiu-ming Cheung, and Yang Lu. Long-tailed visual recognition via gaussian clouded logit adjustment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6929-6938. 2022.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella X Yu. Large- scale long- tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
13

---

Under review as a conference paper at ICLR 2024
Maithra Raghu, Justin Gilmer, Jason Yosinski. and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Harsh Rangwani, Sumukh Aithal, Mayank Mishra, et al. Escaping saddle points for effective generalization on class-imbalanced data. Advances in Neural Information Processing Systems, 35: 22791-22803 2022.
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, al. Balanced meta-softmax for long-tailes visual recognition. Advances in neural information processing systems, 33:4175-4186, 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Jianfeng Wang, Thomas Lukasiewicz, Xiaolin Hu, Jianfei Cai, and Zhenghua Xu. Rsg: A simple but effective module for learning imbalanced datasets. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3784-3793, 2021.
Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. Distribution alignment: A unified framework for long-tail visual recognition. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2361-2370, 2021.
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recogni- tion. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 16489-16498, 2021.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: blaterral-branh network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9719-9728, 2020.
Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Balanced contrastive learning for tailet visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Roland S Zimmermann, Judy Borowski, Robert Geirhos, Matthias Bethge, Thomas Wallis, and Wieland Brendel. How well do feature visualizations support causal understanding of cnn activa- tions? Advances in Neural Information Processing Systems, 34:11730 100--114444 1744, 2021.
14


Reviews:
[{"review_id": "2HMUqOcBZs", "reviewer": "Reviewer_yXjC", "summary": "This paper focuses on the problem of Deep Long-Tailed Recognition (DTLR) and highlights the importance of dynamic re-balancing to address optimization conflicts in this domain. The authors empirically demonstrate that existing DTLR methods are dominated by certain categories due to fixed re-balancing strategies, preventing them from effectively handling gradient conflicts. To address this, they introduce an approach based on multi-objective optimization (MOO) to decouple the problem from a temporal perspective, avoiding class-specific feature degradation. Their method, named PLOT (Pareto deep long-tailed recognition), was conducted on several benchmarks for evaluation.", "questions": "Please see the paper weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The long-tailed recognition problem studied in this paper is a fundamental task that deserves further study.\n- The paper is well organized and easy to follow.\n- The technique contributions of this paper are novel and reasonable, which are tailored for the challenges of long-tailed recognition.", "weaknesses": "- My main concern with this paper is that the authors primarily demonstrate the effectiveness of their approach by augmenting existing methods with PLOT. However, they lack comprehensive performance comparisons by integrating PLOT with state-of-the-art methods like BBN, SADE, PaCo, etc. This omission makes it challenging to assess the method's effectiveness in comparison to the latest advancements in the field, raising questions about its overall impact and general applicability.\n- Another potential weakness of this paper is the insufficient coverage of long-tailed recognition methods in the references. The paper may not thoroughly discuss some classical methods in the field, which can be essential for providing a comprehensive understanding of the long-tailed recognition landscape.\n- The conclusion of the article should be written in the past tense. Additionally, the conclusions require to be added future work of this paper.\n- The authors are encouraged to carefully proofread the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5NhvzCm09T", "reviewer": "Reviewer_KQKE", "summary": "This paper considers an interesting and important issue in long-tailed learning, that is the optimization conflicts. To solve this problem, the authors introduce Pareto optimization for long-tailed learning. First, the authors observe that existing widely-used fixed re-balancing strategies can uniformly lead to gradient conflict. Then, they introduce Multi-Objective Optimization (MOO) to enhance existing long-tailed learning methods. Moreover, the authors find that directly integrating MOO-based methods can lead to performance degradation. To solve this, they propose to decouple the MOO-based methods from the temporal rather than structural perspective to enhance the integration. Experimental results demonstrate that the proposed temporal MOO method can boost the baseline method and achieve an improvement by a large gap.", "questions": "(This is not a question, but a suggestion). I found a very similar work that also studies long-tailed learning with multi-objective optimization [1]. Maybe you can publicly release your work considering its timeliness.\n\n[1] Long-Tailed Learning as Multi-Objective Optimization, in https://arxiv.org/abs/2310.20490", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper studies an interesting problem, which is the optimization conflict in long-tailed learning.\n2. The authors propose to utilize MOO to solve the conflict issue, which is reasonable and makes sense.\n3. This paper is clearly written and easy to understand.\n4. The authors conduct multiple empirical studies to demonstrate the effectiveness since the results show that the proposed method can boost the performance of the baseline methods obviously.\n5. The source code is released for reproduction.", "weaknesses": "1. Figure 1 may cause misunderstandings. How is Figure 1 computed? Is it cosine similarities between the mean of the gradient of different classes? Since the diagonal is red (1.0), it seems that the values represent gradient similarities. And what is the connection between \"gradient conflicts\" and \"gradient similarities\"?\n2. You mainly consider the directions of gradients. Have you considered the impacts of the L2 norms of the gradients?\n3. As you have mentioned MOO is applied during the early stages of representation learning, how to select the applied stage?\n4. The iNaturalist dataset has multiple versions. You should highlight it with \"iNaturalist 2018\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RKlErwpEtX", "reviewer": "Reviewer_wNAW", "summary": "In this paper, the authors mainly tackle the long-tailed learning from the optimization perspective. Specially, this paper first reveals the optimization conflicts among categories in long-tailed learning, and proposes to integrate multi-objective optimization (MOO) with long-tailed learning. Temporal design on MOO, variability collapse loss and sharpness-aware minimization are then employed. Experiments on four long-tailed benchmark datasets are conducted to validate the effectiveness of the proposed method.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-\tLeveraging MOO method to enhance long-tailed learning seems reasonable.\n-\tThis paper reveals the phenomena of optimization conflicts among categories, which is an importance topic in the long-tailed learning from the optimization perspective. \n-\tThe proposed method does not introduce additional computational cost.\n-\tThe paper is generally easy to follow.", "weaknesses": "-\tThe paper introduces some trivial factors, e.g. sharpness-aware minimization, which makes the contribution of this paper ad-hoc. In my humble opinion, sharpness-aware minimization does not show strong connection to the proposed MOO framework, and can be integrated to almost all the long-tailed learning baselines.\n-\tThe technical contribution of this paper seems not strong. This paper mainly adopts CAGRAD with slight modification on the training schedule (temporal design). Besides, sharpness-aware minimization is directly applied and the variability collapse loss is only a regularization term on the standard deviation of the loss.\n-\tThe effectiveness of the proposed PLOT on top of baseline methods shows significant variability, see Table 1-2. However, the authors only conduct experiments on top of cRT + Mixup/MiSLAS on ImageNet-LT, Places-LT, iNaturalist, thus casting doubts on the confidence of the results.\n-\tIn Table 1, MOO underperforms in nearly half the cases when compared with baseline methods (24/54). It cannot support the claims that MOO is beneficial for long-tailed learning. However, in Table 4, the authors present the most favorable performance enhancement with cRT + Mixup, thus making the results less confident.\n-\tAccording to Theorem 4.1, the generalization bound is bounded by the weighted intra-class loss variability, i.e., $w_KM$. It raises the question of whether uniformly constraining the intra-class loss variability is the optimal solution. Besides, in the neural collapse theory, minimizing loss function (e.g. cross-entropy loss) can lead to the intra-class collapse at the optimal case, why there should be an explicit constraint on intra-class variability?\n-\tThis paper lacks background and discussions on the MOO methods, including MGDA, EPO and CAGrad. The reason for choosing CAGrad is solely performance-driven, lacking a connection to its inherent compatibility with the long-tailed learning paradigm.\n-\tIn this paper, multi-classification task is regarded as multiple binary classification tasks. Does the proposed approach yield effective results when applied to imbalanced multi-label classification tasks? The empirical study on the multi-label tasks will benefit a lot to improve the generalizability of the proposed method.\n-\tThis paper lacks disjoint analysis on the classification performance wr.t. different groups (many/medium/few).\n-\tSome results on logit adjustment are missing in Table 2.\n-\tThere is no clue that the proposed method solves the gradient conflicts issue depicted in Figure 1.\n-\tIn figure 8, it's unclear whether the proposed method actually surpass the performance of the baseline method LDAM-DRW in terms of gradient similarities. The gradient similarities of the proposed method exhibit a similar trend in the second stage.\n-\tIt is suggested to provide some qualitative results (e.g. t-SNE) to validate the effectiveness of the proposed method on the embedding space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)