Breaking Long-Tailed Learning Bottlenecks: A Controllable Paradigm with Hypernetwork-Generated Diverse Experts

Authors: zhe  zhao
haibin  wen
zikang  wang
pengkun  wang
fanfu  wang
song  lai
qingfu  zhang
yang  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 15:13:44

Abstract: Traditional long-tailed learning methods often perform poorly when dealing with inconsistencies between training and test data distributions, and they cannot flexibly adapt to different user preferences for trade-offs between head and tail classes. To address this issue, we propose a novel long-tailed learning paradigm that aims to tackle distribution shift in real-world scenarios and accommodate different user preferences for the trade-off between head and tail classes. We generate a set of diverse expert models via hypernetworks to cover all possible distribution scenarios, and optimize the model ensemble to adapt to any test distribution. Crucially, in any distribution scenario, we can flexibly output a dedicated model solution that matches the user's preference. Extensive experiments demonstrate that our method not only achieves higher performance ceilings but also effectively overcomes distribution shift while allowing controllable adjustments according to user preferences. We provide new insights and a paradigm for the long-tailed learning problem, greatly expanding its applicability in practical scenarios. The code can be found here: https://github.com/DataLab-atom/PRL.

Main paper: 
 introduction: 
 In many real-world tasks such as object detection and image classification, we face the challenge of long-tailed distributions. Since the samples of the head classes account for the vast majority in the datasets while the tail class samples are extremely scarce [21, 16, 6, 17] , this extreme imbalance in the data makes the model prone to overfitting towards the head classes during training, resulting in poor performance on the tail classes [34, 8, 30, 20, 39] .To address the long-tailed distribution problem, existing research has proposed a series of methods such as re-sampling [25, 5, 24, 10] and modifying the loss function [17, 6] , with the common idea of focusing on improving the performance of the tail classes. However, these methods typically assume that the distributions of the training and test data remain invariant, and thus cannot well handle the Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We describe the contributions of this paper in detail. 
related work: 
 Long-tailed distributions are prevalent in real-world data, leading to imbalanced datasets that pose challenges for machine learning models [30, 20] . To address this issue, researchers have proposed various methods, including re-sampling, loss function modification, and multi-expert models.Re-sampling methods balance class distributions by oversampling tail classes [24, 3] or undersampling head classes [7] . Loss function modification approaches assign higher weights to tail class losses [27, 26] or use meta-learning to alleviate undersampling issues [14, 32] . Multi-expert models train multiple experts on different class distributions and combine their outputs, adapting to various test distributions [37, 38, 31] . Most existing methods assume specific distributions during training or testing, limiting real-world applicability with distribution shifts, and cannot accommodate varying user needs for head and tail class trade-offs. We propose an approach to overcoming distribution assumptions and achieve interpretable, controllable trade-offs in long-tailed learning.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of this paper in the appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theory: 
 In this section, we analyze the distribution shift problem from a theoretical perspective and provide the definition and properties of the environment's total variation distance, laying the theoretical foundation for the methods section. Traditional empirical risk minimization (ERM) methods on a single training distribution struggle to handle distribution discrepancy, which can affect generalization. This limitation can be characterized by the following theorem:Theorem 1 (Limitation of ERM). Let f (x; θ) be a classifier learned via ERM on E m , then its risk on the test environment E test is:EQUATIONwhere R m (f ) and R test (f ) are the risks of f on E m and E test , respectively, and π test is the class prior of the test environment.To measure the distribution discrepancy across environments, we introduce the environment's total variation distance (ETVD): Definition 2 (Environment Total Variation Distance). The total variation distance between environments E i and E j is defined as:δ(E i , E j ) = 1 2 K k=1 |π i k -π j k |, and the ETVD of M environments is defined as: ∆(E 1 , . . . , E M ) = max i,j∈{1,...,M }δ(E i , E j )Using ETVD, we can further bound the risk of the ERM-learned classifier on the test environment: Corollary 1. Under the assumptions of Theorem 1, let M = max i,x ℓ(f (x; θ), i), thenEQUATIONThis corollary shows that the test risk of the ERM-learned classifier is affected not only by the distribution discrepancy between the training environment and the test environment but also by the distribution discrepancy among training environments (i.e., ETVD). To overcome the diversity shift, we propose minimizing the empirical risks across multiple training environments to capture the distributional characteristics of different environments, thereby learning a set of diversity experts.Next, we provide a theoretical analysis of the domain adaptation algorithm based on diversity experts proposed in this paper. To characterize the generalization performance of this algorithm, we first introduce the following notations:Let {f 1 , . . . , f N } be the N experts learned via ERM on the N training environments {E 1 , . . . , E N }, respectively, and f be the final classifier obtained by ensembling these N experts. Define the empirical risk of the ensemble classifier f on environment E m as:EQUATIONWe can obtain the following theorem regarding the generalization performance of the ensemble classifier: Theorem 2. Under the above notations and definitions, the risk of the ensemble classifier f on the test environment E test satisfies:EQUATIONwhereM = max i,x ℓ( f (x), i).Theorem 2 shows that the test risk of the ensemble classifier consists of three parts: the average empirical risk of all experts, the average total variation distance between the training environments and the test environment, and the weighted average of ETVD among the training environments. Compared to single-environment ERM, the diversity experts method learns a set of experts to capture the distributional characteristics of different environments, which can reduce the distribution discrepancy between the training environments and the test environment, thereby achieving better generalization performance.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Answer: [Yes] Justification: We provide the complete code as well as the details of the publicly available datasets used.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
problem formulation: 
 Consider a K-class classification problem with a training set D = {(x i , y i )} N i=1 , where each class k has N k samples. Let P train denote the empirical distribution over D. The goal is to learn a classifier f : X → R K that generalizes well across various test distributions P test . Traditional empirical risk minimization (ERM) methods optimize the loss under P train , but may fail to adapt to changes in P test , especially in long-tailed scenarios.To improve the robustness of f , we optimize the losses under multiple importance-weighted distributions. Define an M -dimensional simplex:EQUATIONEach α ∈ ∆ M corresponds to an importance-weighted distribution P α :EQUATIONwhere P k (x | y) and P k (y) = N k N are the conditional distribution and prior for class k, respectively. The objective is to learn a set of classifiers F := {f (i) } M i=1 that achieve low risk simultaneously across all P α , forming the Pareto optimal solution:EQUATIONwhereR Pα (F) := E (x,y)∼Pα 1 M M i=1 ℓ(f (i) (x), y). Pursuing an approximate Pareto solution across all distributions leads to models with stronger generalization capabilities.Let X and Y denote the input and output spaces, respectively. We introduce T = 3 classifiers {f i } T i=1 as diverse experts. These experts share a feature extractor ϕ θ : X → R d , but use different classifier heads {g wi } T i=1 :EQUATIONTo generate diverse experts, we introduce a hypernetwork h ψ that takes random noise z ∈ R k as input and outputs the classifier head parameters w i ∈ R d×C :EQUATIONwhere Dir(α) is the Dirichlet distribution with parameter α ∈ R k + . The hypernetwork h ψ consists of three linear layers with ReLU activations.During training, we sample {z i } T i=1 from Dir(α) and use h ψ to generate {w i } T i=1 . The loss function for a training batch is:EQUATIONwhere L i is the classification loss for the i-th expert f i , defined the same as in SADE: L 1 is the standard cross-entropy loss; L 2 is the balanced softmax loss, where the logits are adjusted by adding the log of the prior probabilities of each class; L 3 is the inverse softmax loss, where the logits are adjusted by adding the log of the prior probabilities and subtracting the scaled log of the inverse prior probabilities.Let L i (Θ, D) denote the loss function of the i-th expert f i on dataset D, where Θ = {θ, ψ} represents all trainable parameters. The objective is to jointly optimize the losses of all T experts:EQUATIONTo promote diversity among experts, we introduce the Stochastic Convex Ensemble (SCE) strategy, which aims to minimize the worst-case loss of the convex combination of experts:EQUATIONwhere p = (p 1 , • • • , p T ) ⊤ ∈ ∆ T is the weight vector, and∆ T := {p ∈ R T + | T i=1 p i = 1} is the T -dimensional simplex.Inspired by the max-min inequality, we relax the SCE objective to:EQUATIONwhere λ > 0 is a hyperparameter. As λ → 0, the relaxed objective approaches the original SCE objective. The term λ • log T i=1 exp 1 λ L i (Θ, D) promotes diversity among experts.During testing, we can control the trade-off between head and tail classes using a preference vectorα * = (α * 1 , α * 2 , α * 3 ) ⊤ ∈ ∆ 3, where ∆ 3 is the 3-dimensional simplex. Given a trained preference vector r = (r 1 , r 2 , r 3 ) ⊤ ∈ ∆ 3 , we compute the test-time preference vector r ∈ ∆ 3 as:EQUATION)where ⊙ denotes the Hadamard product. The test-time preference vector r is then input to the hypernetwork h ψ to generate the classifier head parameters for each expert:EQUATIONwhere Ŵi ∈ R d×C is the weight matrix for the i-th expert's classifier head. For a test sample with feature vector x ∈ R d , the output of the i-th expert is:EQUATIONwhere ∥ • ∥ F is the Frobenius norm and bi ∈ R C is the bias vector for the i-th expert. By adjusting α * , we can control the model's focus on head or tail classes, enabling flexible trade-offs to suit different application needs. An observation on our method. To better understand this part, we use Figure 2 to demonstrate the effectiveness of preference control in overcoming distribution shifts, as well as the flexibility of our method. For preferences, the coordinate system is a three-dimensional orthogonal coordinate; for accuracy, the coordinate system represents the performance on the farward50, uni., and backward50 splits of the CIFAR100-LT dataset. The dark plane represents the plane formed by different preference vectors, and the outer surface represents the corresponding performance on the three distributions for these preference vectors. The yellow dots are the results of running SADE, whose preferences are uncontrollable, so the results of each run are random dots, lying below our purple plane, indicating that their performance is lower than our method (i.e., being dominated in the Pareto optimal set). This figure illustrates that our method can cover unknown distributions without additional training, and unlike previous methods, it can trade off performance by adjusting the preference vector. We will analyze this in more depth in the experimental section.  
experiments: 
 In this section, we first evaluate the superiority of PRL in terms of both standard and test-agnostic long-tailed recognition to demonstrate that our method has a higher performance ceiling under the traditional setup. Then, we analyze the effectiveness of our method in changing the trade-off for long-tailed classes through input preferences. Furthermore, we conduct necessary ablation studies.Datasets. We evaluate our method on four benchmark datasets: ImageNet-LT [20] , CIFAR100-LT [4] , Places-LT [20] , and iNaturalist 2018 [29] . These datasets have varying imbalance ratios, ranging from 10 to 256. CIFAR100-LT has three versions with different imbalance ratios. Detailed statistics are in Appendix D.Baselines. We compare PRL with various state-of-the-art long-tailed recognition methods, including two-stage methods (MiSLAS [41] ), logit-adjusted training (Balanced Softmax [15] , LADE [12] ), ensemble learning (RIDE [32] , SADE [38] ), causal inference (Causal [28] ), representation learning (LSC [33] ), and balanced posterior averaging (BalPoE [1] ). These methods address the long-tail problem from different perspectives. Further details are provided in the appendixA.Evaluation protocols and implementation details. We evaluate the models on multiple test datasets with different class distributions using micro accuracy. We report the accuracy of many-shot, mediumshot, and few-shot classes. We use the same setup for all methods, including ResNeXt-50 for ImageNet-LT, ResNet-32 for CIFAR100-LT, ResNet-152 for Places-LT, and ResNet-50 for iNaturalist 2018 as backbones. We employ hypernets (MLPs) to output trainable parameters of experts and adopt the cosine classifier for prediction. Unless specified, we use α = 1.2 for the Dirichlet distribution, µ = 0.3 for stochastic annealing, SGD with momentum 0.9, train for 200 epochs, and set the initial learning rate to 0.1 with linear decay. During test-time training, we train aggregation weights for 5 epochs with a batch size of 128, using the same optimizer and learning rate as in training. Other details please refer to Appendix G.We conduct extensive experiments on four widely-used long-tailed datasets, including CIFAR100-LT, Places-LT, iNaturalist 2018, and ImageNet-LT, to evaluate the performance of our proposed PRL method in comparison with state-of-the-art approaches.Results on standard long-tailed recognition. Table 1 demonstrates the effectiveness of our proposed method, PRL, on four benchmark datasets under the standard long-tailed recognition setting, where the test class distribution is uniform. PRL consistently achieves the highest top-1 accuracy across all datasets, outperforming the previous state-of-the-art methods, LSC [33] and BalPoE [1] . On CIFAR100-LT, PRL improves the accuracy by 0.6% to 0.8% compared to LSC and BalPoE, showcasing its robustness to different imbalance ratios (IR=10, 50, and 100). Similarly, on Places- LT, iNaturalist 2018, and ImageNet-LT, PRL obtains the best performance, surpassing the existing methods by a clear margin. The superior performance of PRL in the standard long-tailed recognition setting validates the efficacy of our approach in mitigating the bias towards head classes and improving the recognition accuracy of tail classes.Results on distribution-shift long-tailed recognition. We evaluate the performance of PRL and other methods in the distribution-shift long-tailed recognition setting, where the test class distribution is unknown and different from the training distribution. Tables 2 and 3 show the Top-1 accuracy results on various test class distributions (including forward LT, uniform, and backward LT) for CIFAR100-LT (IR=100) and ImageNet-LT, respectively.Figure 4 : A more comprehensive example of how preference influences performance.On different test distributions of both datasets, PRL consistently outperforms all compared methods. On CIFAR100-LT (IR100), PRL achieves the highest accuracy across all settings, surpassing LSC [33] , BalPoE [1] , and SADE [38] .Even under the most challenging backward LT distribution, PRL can maintain its outstanding performance. On ImageNet-LT, PRL obtains the best results across all test distributions, significantly outperforming LSC, BalPoE, and SADE.The consistent improvements achieved by PRL highlight its higher performance ceiling, indicating the effectiveness of our method design in overcoming distribution shifts. We further conduct distribution-shift experiments on Places-LT and iNaturalist 2018, where PRL also achieves impressive results. Please refer to the AppendixF for detailed results. User preference control. We evaluated the model's performance on many-shot, medium-shot, and few-shot classes on CIFAR100-LT under different preference settings (R=(1.0, 2.7), R=(0.5, 2.5), R=(1.9, 1.1)). Table 4 shows that by adjusting the preference value R, we can effectively control the trade-off between many-shot and few-shot classes. When R=(1.0, 2.7), the model performs best on many-shot classes; when R=(1.9, 1.1), the model performs better on few-shot classes at the cost of a slight drop in performance on many-shot classes. R=(0.5, 2.5) achieves the best performance on medium-shot classes, indicating that our method can balance performance across different classes with an appropriate setting. As shown in Figure 3 , we analyze the performance trade-offs between head and tail classes across three different distributions, demonstrating how our preference control mechanism allows flexible adjustment of model behavior. The results clearly show how adjusting preferences affects accuracy across different class frequency groups. Figure 4 provides a more comprehensive visualization of the performance on the head classes under the forward50 distribution. The plane represents the performance of the head classes without inputting any preference, while the red dots indicate the preference positions in polar coordinates that can improve the performance of the head classes, and the green dots represent the preference positions that may degrade the performance. These experimental results demonstrate the effectiveness of our method in controlling the trade-off for long-tailed classes based on user preferences. By adjusting the preference without the need for retraining the model, we can flexibly adapt to different application scenarios and requirements, achieving a desired trade-off in long-tailed recognition tasks that aligns with practical needs.Ablation study. We conduct ablation studies on CIFAR100-LT to evaluate the impact of removing the hypernetwork (w.o. hnet) and removing the Chebyshev polynomial (w.o. stch) on the model's performance under different unknown test class distributions (as shown in Figure 5 ). The complete model (ours) performs best across all distributions. Removing either the hypernetwork or the Chebyshev polynomial leads to performance degradation, highlighting their importance in dynamically adjusting the model behavior to adapt to distribution shifts and learning preference-aware representations. This ablation study verifies the effectiveness of different components in our method, which work together to better handle unknown test distributions and data imbalance issues in long-tailed recognition. 
conclusion: 
 This study introduces a novel long-tailed learning paradigm to address distribution shifts between training and testing datasets. Our hypernetwork-based approach generates adaptable classifiers, achieving Pareto optimality for real-time adaptation. During inference, the model adjusts based on user-defined trade-offs between head and tail classes, enhancing flexibility. Empirical results show improved accuracy and adaptability to class imbalances and distribution shifts. Our work establishes an interpretable, generalizable, and controllable framework for long-tailed learning, meeting diverse user needs.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provide the relevant details and analysis of the experimental results.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We reported the average results of multiple tests in the experiment.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We have provided the code for easy reproduction.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We are fully qualified.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the implications in the appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We give references to the datasets and code used.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: • Oversampling [5, 9] : Generates synthetic examples for minority classes.-Alleviates imbalance by increasing tail class samples.-Can lead to overfitting and high computational cost.• Undersampling [7, 2] : Removes examples from majority classes.-Simple and efficient approach to balance classes.-Discards potentially valuable head class information.• Focal Loss [17] and variants [6] :-Imposes larger penalties on well-classified examples, encouraging focus on hard samples.-Requires careful tuning of focusing parameter.• Class-Balanced Loss [6] :-Re-weights loss based on effective number of samples per class.-Assumes equal importance of classes, which may not hold in practice.• LDAM [4] : -Explicitly models each example's contribution to the gradient direction.-Requires additional hyperparameters and complex optimization.• Decoupled Learning [16, 43] :-Separates representation and classifier learning for better feature extraction.-Requires architectural changes, may not generalize well.• Few-Shot Experts [32] :-Employs additional experts to handle few-shot classes.-Increased model complexity and training difficulty.• Self-Supervised Pretraining [14] :-Leverages self-supervision to improve feature representations.-Requires additional pretraining, benefits may be task-specific.• Data-Based Transfer [20, 14] :-Knowledge distillation and feature transformation can transfer head knowledge to tails.-Assumes head and tail distributions are related, may suffer negative transfer.• Model-Based Transfer [36] :-Utilizes models pretrained on heads to facilitate tail class learning.-Again assumes related head and tail distributions.Despite progress, existing LTL methods face limitations in addressing the inherent head-tail trade-off, handling distribution shifts, and accommodating varying user preferences. To overcome these issues, we formulate LTL as a multi-objective optimization problem and propose a hypernetwork-based diverse expert learning paradigm, achieving interpretable and controllable solutions tailored to user needs under test distribution shifts.Let X ⊆ R n be the decision space and consider m objective functions f i : X → R, i = 1, . . . , m to be minimized simultaneously. The multi-objective optimization problem (MOP) can be stated as:EQUATIONIn general, there does not exist a single solution x * ∈ X that minimizes all objectives simultaneously due to the conflicting nature of the objectives. Instead, the solution concept is that of Pareto optimality [22] . Definition 3 (Pareto Optimality). A solution x * ∈ X is Pareto optimal if there does not exist anotherx ∈ X such that f i (x) ≤ f i (x * ) for all i = 1, . . . , m and f j (x) < f j (x * ) for at least one j.The set of all Pareto optimal solutions is called the Pareto set, and its image in the objective space is the Pareto front. The goal in MOPs is to approximate the Pareto front as well as possible.A common approach to approximate the Pareto front is through scalarization methods that transform the MOP into a scalar optimization problem [22, 19] . The weighted Chebyshev scalarization is defined as:EQUATIONwhere w = (w 1 , . . . , w m ) T ∈ R m + is a weight vector with m i=1 w i = 1, and z * = (z * 1 , . . . , z * m ) T is a utopian reference point [11] . By varying w, different Pareto optimal solutions can be obtained.Hypernetworks [18] provide a promising approach for multi-objective optimization of neural networks. A hypernetwork h ϕ : Z → Θ is a neural network that takes a low-dimensional input z ∈ Z and outputs the parameters θ ∈ Θ of a target neural network f θ : X → Y. By sampling different z ∈ Z, the hypernetwork generates an ensemble {f θi } i where θ i = h ϕ (z i ). This ensemble can approximate the Pareto front of the multi-objective optimization problem:EQUATIONwhere L i : Θ → R are loss functions corresponding to the m objectives. The hypernetwork parameters ϕ can be optimized via scalarizations like the Chebyshev method:EQUATION)where p(w) is a distribution over weight vectors w. This enables learning a diverse set of target networks approximating the Pareto front in a flexible and controllable manner.Proof. The risk of classifier f on the test environment E test is defined as:EQUATIONUsing the law of total expectation, we can decompose the risk as:EQUATIONSimilarly, the risk of classifier f on the training environment E m can be expressed as:EQUATIONSince the classifier f is learned via ERM on E m , we have P m (x|y = i) = P test (x|y = i) for all i ∈ {1, . . . , K}. Therefore,EQUATION= K i=1 π m i • E x∼Ptest(x|y=i) [ℓ(f (x; θ), i)] + K i=1 (π test i -π m i ) • E x∼Ptest(x|y=i) [ℓ(f (x; θ), i)] = R m (f ) + K i=1 (π test i -π m i ) • E x∼Ptest(x|y=i) [ℓ(f (x; θ), i)]This completes the proof.Proof. From Theorem 1, we have:EQUATIONUsing the definition of M , we can bound the expectation term:E x∼Ptest(x|y=i) [ℓ(f (x; θ), i)] ≤ M (26) Therefore, R test (f ) ≤ R m (f ) + K i=1 (π test i -π m i ) • M = R m (f ) + M • K i=1 |π test i -π m i | = R m (f ) + 2M • δ(E m , E test )where the last equality follows from the definition of the total variation distance δ(E m , E test ) =1 2 K i=1 |π m i -π test i |. Next, we use the triangle inequality to bound δ(E m , E test ): δ(E m , E test ) ≤ δ(E m , E j ) + δ(E j , E test ) ≤ max i,j∈{1,...,M } δ(E i , E j ) + δ(E j , E test ) ≤ ∆(E 1 , . . . , E M ) + δ(E j , E test )for any j ∈ {1, . . . , M }. By taking the average over all j, we obtain:EQUATIONCombining this with the previous bound on R test (f ), we have:EQUATIONwhich completes the proof.Proof. By definition, the risk of the ensemble classifier f on the test environment E test is:R test ( f ) = E (x,y)∼Ptest [ℓ( f (x), y)] = 1 N N i=1 E (x,y)∼Ptest [ℓ(f i (x), y)] = 1 N N i=1 R test (f i )Applying Corollary 1, we have:R test ( f ) ≤ 1 N N i=1 R m(i) (f i ) + 2M • (δ(E m(i) , E test ) + ∆(E 1 , . . . , E N )) = 1 N N i=1 R m(i) (f i ) + 2M N N i=1 δ(E m(i) , E test ) + 2M ∆(E 1 , . . . , E N )where m(i) denotes the index of the training environment used to learn expert f i . Now, we bound the second term:1 N N i=1 δ(E m(i) , E test ) = 1 N N m=1 i:m(i)=m δ(E m , E test ) ≤ 1 N N m=1 N m • δ(E m , E test ) ≤ 1 N N m=1 N • δ(E m , E test ) = N m=1 δ(E m , E test )where N m is the number of experts learned from environment E m , and we used the fact that N m=1 N m = N . Finally, we have:R test ( f ) ≤ 1 N N m=1 R m (f m ) + 2M • 1 N N m=1 δ(E m , E test ) + ∆(E 1 , . . . , E N ) = 1 N N m=1 R m (f m ) + 2M • 1 N N m=1 δ(E m , E test ) + N -1 N ∆(E 1 , . . . , E N )where the last equality follows from the definition of ETVD.Next, we will explain the connection between the theoretical part of the paper and the proposed method, to aid better understanding The above theoretical analysis demonstrates that, in the long-tailed learning domain, introducing multiple training environments and minimizing the empirical risks on these environments to learn a set of diverse experts can effectively address the problem of distribution shift between the training and test environments, leading to better generalization performance. These theoretical insights provide important guidance for further improving our algorithm.To evaluate the effectiveness of our proposed method, we conduct experiments on four long-tailed datasets: CIFAR100-LT, ImageNet-LT, iNaturalist 2018, and Places365-LT. These datasets cover a diverse range of domains and exhibit varying degrees of class imbalance, providing a comprehensive testbed for long-tailed learning algorithms. ImageNet-LT [20] is a long-tailed subset of the ImageNet dataset, containing over 115,000 images spanning 1,000 classes. The class cardinalities follow a Pareto distribution with α = 6, leading to a maximum imbalance ratio of 256.iNaturalist 2018 [29] is a real-world dataset with a natural long-tailed distribution, comprising approximately 450,000 images across 8,142 species. The number of images per species varies drastically, with an imbalance ratio of up to 500, posing a significant challenge due to the extreme class imbalance and high intra-class variation.Places365-LT is a long-tailed version of the Places365 dataset [42] , which consists of over 1.8 million images spanning 365 scene categories. We induce a long-tailed distribution by randomly subsampling the images for each class, resulting in an imbalance ratio of approximately 50. This dataset is particularly challenging due to the large number of classes and the inherent visual ambiguity present in scene recognition tasks.Here are pseudo codes explaining the core aspects of the method: for k = 1 to K do 10:ϕ k = H θ (z k ) {Generate expert weights ϕ k from hypernetwork}11:E k = E ϕ (f ) {Obtain expert predictions using ϕ k } 12:L k = L(E k , y, extra_info) {Compute expert losses} 13:end for 14:L div = DiversityLoss(E) {Encourage expert diversity} 15:L total = k L k + λL div 16: θ = θ -η∇ θ L total 17:end for 18: end for=0 Total loss for updating hypernetwork weightsOn the representative Places-LT dataset, our PRL method achieves the best Top-1 accuracy under various unknown test class distributions. Specifically, in the Forward-LT setting, as the proportion of unknown classes decreases from 50% to 2%, the Top-1 accuracy of PRL drops from 47.9% to 42.8%, but still significantly outperforms other baseline methods. Under the Uniform distribution, PRL reaches the highest accuracy of 41.9%. In the Backward-LT setting, PRL's accuracy gradually increases from 41.7% to 44.1%, again surpassing all counterpart methods. These results thoroughly validate the outstanding performance and robustness of our method in handling various unknown class distributions.On the iNaturalist 2018 dataset, PRL also exhibits excellent performance. In the Forward-LT setting, when the proportion of unknown classes decreases from 3 to 2, PRL's Top-1 accuracy slightly increases from 73.7% to 73.8%, and reaches the best performance of 74.3% under the Uniform distribution. In the Backward-LT setting, although PRL's accuracy slightly decreases from 74.0% to 73.9%, it still outperforms all comparison methods. These results further confirm the broad effectiveness of our method across different datasets and scenarios.Overall, by successfully tackling the challenges of long-tailed distributions and unknown class distributions, the PRL method demonstrates superior performance on two representative long-tailed datasets, thereby validating the superiority of our method. 9 : Model size and computational cost with and without hypernetworks.The hypernetwork is responsible for outputting the trainable parameters D for each expert classifier head. In this case, the number of parameters in the hypernetwork becomes E × D × K, where E is the number of input channels to the output layer of the hypernetwork, and K is the number of experts. Thus, the time complexity of the hypernetwork is O(D).If the total number of parameters in the model without the hypernetwork is O(N ), and the computational complexity of the model's main operations is O(M × N ), where M > 1 is a complexity factor, then when N ≫ D, the overall time complexity becomesO(D) + O(M × N ) = O(M × N ).Therefore, while the hypernetwork increases the total number of parameters, its impact on the overall computational complexity is relatively small, especially when the number of parameters D generated by the hypernetwork is much smaller than the total number of parameters N in the main model. As can be seen from the table, the introduction of hypernetworks results in almost no increase in GFLOPs (floating-point operations) across different models.While the proposed novel approach of using a hypernetwork to generate multiple diverse expert models shows great potential in enabling controllable adjustment of head and tail class weights for long-tailed datasets, as well as improving robustness to distribution shifts, the introduction of the hypernetwork also brings new challenges to model training and convergence. As an additional neural network module, the hypernetwork needs to generate the weight parameters for the classifier heads of each expert, thereby significantly increasing the total number of trainable parameters in the model, which may affect training stability. We analyze this issue in Section G, nevertheless, further research into more efficient training and stable controllability is still necessary.The proposed novel approach of generating multiple expert models via hypernetworks enables dynamic adjustment of head and tail class weights for long-tailed datasets, and improves model robustness to distribution shifts. This flexibility and robustness are of significant value in many practical applications. The present work provides a new viable solution to the important challenges of long-tailed distributions and distribution shifts, holding promise to enhance the generalization capabilities and practical applicability of existing models, thereby contributing to the technological advancement in relevant fields.• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper proposes a novel long-tailed learning method. It can generate a set of models to cover all distributions, and flexibly output a model that matches the user's preference. Empirical results validate the effectiveness of the proposed method.

The studied problem is an important and practical problem. The paper provides meaningful theoretical results and motivation for the new diversity expert learning method. The proposed method is novel enough. The paper provides an extensive set of experiments to support the claims.

In addition, the authors can further improve the paper according to the detailed reviews.

Reviews: review: paper_summary: paper_summary: The paper addresses the problem of learning long-tailed distributions, with the imbalance of head and tail classes. The paper introduces a long-tail learning paradigm based on diverse set of experts and hypernetworks. The proposed method can meet personalized user preferences and can adapt to wide range of distribution scenarios. The paper proposes this problem as multi-objective optimization, with the goal to learn whole pareto front. The authors also propose theoretical aspects considering distribution shifts and show that diversity experts methods learns a set of experts to capture the distributional characteristics of different environments, hence reducing the distribution discrepancy between the training and test environment.
strength_weakness: strength_weakness: 1. The paper mentions the proposed method to be interpretable and controllable long-tail learning method whereas I don’t think a model with an ability of adapting to  preference vectors can be considered interpretable. More explanation of this could be helpful to understand why authors believe the interpretability of the model. 

2. While the main problem is centered around preference based long-tail learning method, theoretical proofs just talking about distribution shift is not fitting very well. While the proofs look correct (to the best of my knowledge) and I agree that multiple experts can reduce the distribution discrepancy between training and test environment, why will this be useful for long-tailed distribution is not clear to me. Is there any explanation on why all the experts won’t still focus on head classes ?

3. Chebyshev polynomial has suddenly been introduced in ablation study, a brief mention about it would be beneficial for the readers. 

4. There are works which use hypernetworks for MOO [1], I think inclusion of such works would be useful. 

5. Inclusion of proofs considering long-tailed distribution would have been more beneficial for this problem setup. 

6. Given that this work considers optimizing for the whole pareto front, mention of hypervolume values would be useful for future works in this direction. 

Typos 
Line 36: requirement. [16, 29, 36] -> requirement [16, 29, 36] 

[1] Navon, A., Shamsian, A., Fetaya, E. and Chechik, G., Learning the Pareto Front with Hypernetworks. In International Conference on Learning Representations.
questions: questions: 1. The paper mentions the proposed method to be interpretable and controllable long-tail learning method whereas I don’t think a model with an ability of adapting to  preference vectors can be considered interpretable. More explanation of this could be helpful to understand why authors believe the interpretability of the model. 

2. While the main problem is centered around preference based long-tail learning method, theoretical proofs just talking about distribution shift is not fitting very well. While the proofs look correct (to the best of my knowledge) and I agree that multiple experts can reduce the distribution discrepancy between training and test environment, why will this be useful for long-tailed distribution is not clear to me. Is there any explanation on why all the experts won’t still focus on head classes ?

3. Additional information about how unknown test class distribution is created would be helpful.

4. How preference vector for testing has been decided ? 

5. Hypervolume calculation usually needs information on reference vector, how is that taken into consideration here ?

6. While the proposed method learns entire front in a single model, are the competing methods trained multiple models to cover the pareto front ? 

7. What are your comments on scalability of the proposed method for large target networks ?
limitations: limitations: Yes
paper_summary: paper_summary: This paper addresses the crucial and challenging problem of long-tailed learning under distribution shifts between training and testing data, which is highly relevant to real-world applications. The authors propose a novel and insightful learning paradigm that aims to obtain a set of diverse expert classifiers to adapt to arbitrary test distributions while allowing flexible control of the trade-off between head and tail classes based on user preferences. This new perspective greatly expands the applicability and practicality of long-tailed learning methods, making it a significant contribution to the field.
strength_weakness: strength_weakness: 1.	I agree with the novelty of this paper. The method proposed by the authors brings a new research perspective to the field. My doubts lie in part of the theoretical interpretation. Regarding the role of the Dirichlet distribution in generating diverse experts, can the authors analyze the effect of the hyperparameters of the Dirichlet distribution on the performance of the algorithm?
2.	As for the above question, I hope the author carries out some experimental analysis to support his theory.
3.	I think the explanation of Figure 2 seems a bit obscure, and the author would be better off giving a concise explanation.
4.	Some symbol errors need to be corrected.
questions: questions: Please answer the questions in the Weaknesses, which helps me better understand the theoretical contribution of this paper.
limitations: limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
paper_summary: paper_summary: This paper addresses long-tailed learning with a focus on tackling distribution shift and accommodating different user preferences for the trade-off between head and tail classes. The authors propose a method called PRL, which generates a set of diverse expert models via hypernetworks to cover all possible distribution scenarios, optimizing the model ensemble to adapt to any test distribution. Experiments across various long-tailed datasets validate the method's effectiveness.
strength_weakness: strength_weakness: - The paper uses $\theta$ and $\phi$ in Figures 3 and 4, but they are not defined beforehand.
- In Section 5.2, the paper investigates the model's performance under different preference values $R$ but does not explain their meaning.
- In Section 4.4, it is unclear how the trained preference vector is obtained.
- I am confused about how to ensemble the experts after training is complete.
questions: questions: Please see above.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper aims to overcome potential distribution shifts from a single training distribution to any testing distribution and adapt to different user preferences for trade-offs between head and tail classes. The proposed method leverages hypernetworks to generate a diverse set of expert models, enabling the system to adapt to any test distribution and cater to user-specific preferences. Extensive experiments demonstrate the method's superiority in performance and adaptability, providing new insights and expanding the applicability of long-tailed learning in practical scenarios.
strength_weakness: strength_weakness: 1. Some important concepts require further explanation. For example, what is the meaning of "environment" in Sec.3?  It is unclear why to minimize empirical risks across multiple training environments since the training set is typically fixed. Besides, I wonder how to ensemble the experts when the test distribution varies. It seems that the authors did not explain this key issue. 
2. In Section 5.1, the paper states "use $\alpha = 1.2$ for the Dirichlet distribution". However, according to Eq.(5), $\alpha$ should be a vector. It is recommended to provide a more detailed explanation.
3. Some more competitive baselines such as PaCO$^1$, DDC$^2$, and DirMixE$^3$ should be considered for comparison.
4. There are some typos, such as in line 36, where "Simply pursuing the overall optimal solution may not meet this flexibility requirement. [16, 29, 36]" should be "Simply pursuing the overall optimal solution may not meet this flexibility requirement [16, 29, 36]." In line 95, $\pi_m^k$ should be $\pi_k^m$.
5. Some figures, such as Figure 1 and Figure 3, are not referenced in the text.

------

$^1$ Parametric contrastive learning, ICCV 2021 

$^2$ A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning, NeurIPS 2023

$^3$ Harnessing Hierarchical Label Distribution Variations in Test Agnostic Long-tail Recognition, ICML 2024
questions: questions: Please refer to the questions in the weakness section.
limitations: limitations: No.

Comments: title: Thanks for your efforts
Official Comment by Reviewer qXeH
Thanks
Thanks for your efforts.
comment: We are delighted to know that our response has addressed your concerns and has led to an increase in your assessment score.
Your recognition of the improvements made in the revised manuscript is highly appreciated, and it motivates us to continue refining our work to meet the highest standards of quality and clarity.
Thanks a lot for the rebuttal. I have raised the rating to 7.
I appreciate the authors' efforts in the rebuttal. My concerns are addressed. I will raise my rating to 7.
Thank you very much for your thoughtful feedback on our paper. We are truly grateful for your careful consideration of our rebuttal and the time you've taken to reassess our work.If you have any further thoughts or suggestions, we would be more than happy to hear them.
We are delighted to see that our response has addressed the concerns you previously raised. We are deeply grateful for your decision to increase the rating to 7, which is a significant encouragement for us.

Your constructive feedback has played a crucial role in improving the quality of our work. If you have any further comments or suggestions, we welcome your feedback at any time.
I appreciate the authors' efforts in the rebuttal. My concerns are addressed. I will raise my rating to 7.
Thank you for your efforts on the rebuttal! The authors have clarified some important details about my concern. Hence, I decide to raise my rating to 7.