Title: TS-MOF: Two-Stage Multi-Objective Fine-tuning for Long-Tailed Recognition
Venue: NeurIPS 2025
Accepted: yes
Keywords: Long-tailed learning

Multi-objective optimization
Abstract: Long-Tailed Recognition (LTR) presents a significant challenge due to extreme class imbalance, where existing methods often struggle to balance performance across head and tail classes. Directly applying multi-objective optimization (MOO) to leverage multiple LTR strategies can be complex and unstable. To address this, we propose TS-MOF (Two-Stage Multi-Objective Fine-tuning), a novel framework that strategically decouples feature learning from classifier adaptation. After standard pre-training, TS-MOF freezes the feature backbone and focuses on an efficient multi-objective fine-tuning of specialized classifier heads. The core of TS-MOF's second stage lies in two innovations: Refined Performance Level Agreement for adaptive task weighting based on real-time per-class performance, and Robust Deterministic Projective Conflict Gradient for stable gradient conflict resolution and constructive fusion. This approach enables effective synergy between diverse LTR strategies, leading to significant and balanced performance improvements. Extensive experiments on CIFAR100-LT, ImageNet-LT, and iNaturalist 2018 demonstrate that TS-MOF achieves state-of-the-art results, particularly enhancing tail class accuracy (e.g., +3.3\% on CIFAR100-LT IR=100 tail) while improving head class performance, all within a remarkably short fine-tuning period of 20 epochs.


Peer reviews:
comments: SUMMARY:
This paper proposes a practical solution to address the long-tailed distribution problem in machine learning. The TS-MOF framework, through its ingenious two-stage design, enables multiple long-tailed learning strategies to work synergistically, achieving excellent performance across multiple benchmark datasets.

STRENGTHS AND WEAKNESSES:
S1: The design of decoupling feature learning from classifier adaptation greatly simplifies the optimization of complex long-tailed problems, allowing the MOO process to act more stably and efficiently on classifier heads.

S2: R-PLA and RD-PCGrad are the core highlights of this work. R-PLA's adaptive task weighting mechanism can dynamically adjust task weights based on real-time performance patterns, while RD-PCGrad ensures stable resolution of gradient conflicts and constructive fusion, effectively promoting synergy among different LTR strategies.

S3: The paper achieves significant results on multiple benchmark datasets, demonstrating its efficiency.

W1: Although the MOO components enhance robustness, there are still some hyperparameters in R-PLA and RD-PCGrad. I want see some sensitivity analysis of how these parameters affect model performance and convergence.

QUESTIONS:
Q1: Does $ε_{sim}$ have a role other than ensuring numerical stability when computing $s_{j,e}$? For example, in cases of extremely low similarity, does it affect weight allocation preferences?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes TS-MOF, a two-stage framework for long-tailed recognition (LTR). Stage 1 pre-trains a feature backbone using standard cross-entropy loss. Stage 2 freezes the backbone and fine-tunes multiple classifier heads via multi-objective optimization (MOO). In Stage 2, the authors proposed two interesting ideas: R-PLA and RD-PCGrad for long-tailed recognition.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The method underwent rigorous evaluation across three benchmark datasets under varying imbalance ratios.\n2. R-PLA and RD-PCGrad are novel and interesting.\n\nWeaknesses\n1. While the two strategies (R-PLA and RD-PCGrad) in the fine-tuning stage are interesting, the two-stage framework itself appears non-novel. What specific problems do these two strategies solve? The writing logic suffers from a weak connection between the paper’s motivation and these strategies. The authors should significantly reorganize the Introduction. Since the core effectiveness stems from the two MOO strategies, the Abstract and Introduction should highlight them as primary innovations rather than overemphasizing the two-stage framework.\n\n2．The paper lacks schematic diagrams, reducing accessibility. Adding visual illustrations (e.g., workflow of R-PLA/RD-PCGrad) would significantly improve intuitive understanding for readers.\n\n3. The multi-head structure resembles multi-expert models (e.g., SHIKE). However, the reported performance of SHIKE (46.9% on CIFAR100-LT IR=100 in Table 1) contradicts its original paper (56.3%). The authors must:\n(i) Clarify this discrepancy,\n(ii) Analyze how TS-MOF fundamentally differs from multi-expert ensembles in methodology and objectives.\n\n4. Critical implementation details are missing — specifically, which re-balancing strategies (e.g., LDAM? BCL?) were integrated into the multi-head MOO framework.\n\n5. Ablations are critically underdeveloped. The authors should: (1) Isolate the impact of individual components (R-PLA, RD-PCGrad); (2)Compare variants (e.g., alternative task-weighting schemes, gradient conflict resolution methods); (3) Benchmark performance across different strategy combinations.\n\nQUESTIONS:\nSee Strengths And Weaknesses\n\nLIMITATIONS:\nSee Strengths And Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents TS-MOF (Two-Stage Multi-Objective Fine-tuning), a novel decoupled-training expert-based framework for long-tailed recognition. TS-MOF decouples training into two stages: (1) Feature Learning: standard supervised training to learn good representations ; (2) Multi-head Classification Learning: freeze the learned representation and spawn multiple specialized “expert” heads, each fine-tuned with a different long-tailed method (e.g., BCL [26], LOS [20]). Key contributions in the second stage include: (a) Refined Performance Level Agreement (R-PLA): Dynamically weights each expert’s task loss based on per-class performance on the training set ; (b) RD-PCGrad: a robust deterministic variant of PCGrad [23] that resolves gradient conflicts between task objectives via gradient modulation. At inference time, a dynamic expert aggregation module weights and combines expert predictions per class, using performance on a validation set. Experiments on standard long-tailed benchmarks show that TS-MOF achieves higher balanced accuracy than single-objective baselines, with only a brief fine-tuning stage.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- S1 - Novel decoupled-training framework: introduces a two-stage learning approach that leverages multiple existing long-tailed approaches in parallel experts, with training-time adaptive task weighting and inference-time aggregation.\n\n- S2 - Theoretical insights: Provides analysis on gradient stability, Pareto improvements, convergence behaviour, and robustness of R-PLA and RD-PCGrad.\n\n**Weaknesses**\n\n- W1 - Reliance on validation set: Requires a (probably) balanced holdout set to select Stage 1 weights (L160–161) and to compute per-class aggregation weights (Eq. 8–9). In extreme imbalance scenarios, such a validation set may be impractical/unfeasible, and it isn't a common assumption in the LTR literature.\n\n- W2 - Missing component ablations: Lacks experiments isolating the contributions of R-PLA, RD-PCGrad, and the expert aggregation module. \n\n- W3 - Comparison to expert-based SOTAs: Does not discuss or compare with recent skill-diverse multi-expert models (e.g., BalPoE [A], SADE [B], MDCS [C]) that specialize experts with multiple objectives end-to-end.\n\n- W4 - Task Selection Strategy: While Appendix shows a large task combination search, the paper picks the “best” combination based on test-set performance. A more scalable, principled selection mechanism is missing from the paper.\n\n- W5 - Deterministic task ordering: The method for defining the fixed task order in RD-PCGrad is not specified.\n\n- W6 - Computational Overhead: No empirical comparison of Stage 2’s compute or memory cost versus vanilla multi-task optimization.\n\nAdditional references:\n\n[A] Sanchez Aimar, Emanuel, et al. Balanced Product of Calibrated Experts for Long-Tailed Recognition. CVPR 2023.\n\n[B] Zhang, Yifan, et al. Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognition. NeurIPS 2022.\n\n[C] Zhao, Qihao, et al. MDCS: More Diverse Experts with Consistency Self-Distillation for Long-Tailed Recognition. ICCV 2023.\n\nQUESTIONS:\n- Q1 - RD-PCGrad vs. PCGrad [23]: Why choose a fixed task ordering rather than PCGrad’s random ordering? What evidence shows that RD-PCGrad changes improve robustness?\n\n- Q2 - Validation-Set requirement: In the absense of a balanced validation set, can TS-MOF work with an imbalanced holdout set matching training distribution? If so, have you tested sensitivity to validation set imbalance?\n\n- Q3 - R-PLA computation: In Eq. 4, how do you estimate early performance vectors (index j) before computing cosine similarity with the final vector (index N) within the same epoch?\n\n- Q4 - Expert-head architectures: What is the architecture of each classification head (linear layer, MLP, etc.)? How does this compare to expert-based models such as RIDE [22]?\n\n- Q5 - Hyperparameter sensitivity: Which TS-MOF hyperparameters require careful tuning (as indicated in Section 10, L600–601)?\n\nWhile the TS-MOF framework presents a novel take into decoupled-learning and expert-based frameworks, the absence of key ablations, the impractical balanced-validation-set assumption, unclear task-selection and ordering procedures, and missing comparisons to relevant expert-based methods leave important questions unanswered. A focused rebuttal that supplies the above ablations/comparisons and clearly quantifies practical costs will significantly strengthen confidence in TS-MOF’s utility—and could increase my score. Conversely, if these gaps remain unaddressed, my score will stay at 3.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes TS-MOF, a two-stage multi-objective fine-tuning framework for long-tailed recognition. The approach integrates multiple loss functions via a multi-head architecture and resolves gradient conflicts using a modified version of PCGrad (RD-PCGrad), while dynamically reweighting tasks based on a refined performance-level agreement mechanism (R-PLA). Experiments on several long-tailed benchmarks (CIFAR10, ImageNet- iNaturalist) demonstrate consistent improvements over competitive baselines.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n1. The proposed two-stage framework is well-motivated and aligns with prior findings that feature decoupling improve long-tailed generalization;\n2. RD-PCGrad introduces a deterministic and task-aware projection strategy, offering a practical solution to inter-task gradient interference;\n3. R-PLA dynamically adjusts task importance in a label- and batch-aware manner, enabling adaptive optimization focus across head and tail classes;\n4. The method shows consistent gains across CIFAR100-LT, ImageNet-LT, and iNaturalist, with a relatively lightweight fine-tuning phase.\n\n### **Weaknesses**\n1. While Prop. 1 states that the final gradient $g_{final}$ is Pareto-improving or non-worsening, the paper does not analyze how far this direction can deviate from the true Pareto front in the proof. In complex objective landscapes, the projection step may inadvertently suppress beneficial gradient components, and a discussion of such trade-offs is missing;\n2. The use of the cube root in Eq. (5) is not well justified. Based on the proof of Prop. 2, any concave, increasing function could preserve the robustness properties. The choice of the cube root lacks explanation on whether it confers specific benefits (e.g., smoother gradients or empirical stability);\n3. Although the proposed method is evaluated in a two-stage training pipeline, the core techniques (RD-PCGrad and R-PLA) are not inherently limited to such a setup. Gradient conflict resolution might be even more critical in single-stage end-to-end training. The paper would benefit from a more apparent justification for the choice of a two-stage framework or a discussion of broader applicability.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations, and no potential negative impact is found.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-05"}