Multiplicative Logit Adjustment Approximates Neural-Collapse-Aware Decision Boundary Adjustment

Authors: Naoya Hasegawa, Issei Sato
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: long-tailed recognition, imbalanced learning, neural collapse, logit adjustment, multiplicative logit adjustment, machine learning, learning theory

Abstract:
Real-world data distributions are often highly skewed. This has spurred a growing body of research on long-tailed recognition, aimed at addressing the imbalance in training classification models. Among the methods studied, multiplicative logit adjustment (MLA) stands out as a simple and effective method. What theoretical foundation explains the effectiveness of this heuristic method?
We provide a justification for the effectiveness of MLA with the following two-step process. First, we develop a theory that adjusts optimal decision boundaries by estimating feature spread on the basis of neural collapse. Second, we demonstrate that MLA approximates this optimal method. Additionally, through experiments on long-tailed datasets, we illustrate the practical usefulness of MLA under more realistic conditions. We also offer experimental insights to guide the tuning of MLA hyperparameters.

Decision: accept (poster)
Reviewer scores: 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729844253589

thread_type:
official_review

<thread_object>
creator: reviewer_k7rv

<content>
summary:
The paper addresses the problem of class imbalance in classification models and analyzes the effectiveness of the Multiplicative Logit Adjustment (MLA) method. The authors propose a theory on optimal decision boundaries based on neural collapse, demonstrating that MLA approximates these optimal boundaries. The paper is supported by experiments across various modalities, providing insights into MLA’s performance under different conditions.

strengths:
* The problem of class imbalance is clearly motivated, and the investigation into MLA is a valuable contribution to this area of research.
* The mathematical framework is well-developed, offering a solid theoretical foundation for understanding the method.
* The analysis extends beyond image datasets to include other modalities, such as text, enhancing the paper's applicability.
* Extensive experiments are conducted, which reinforce the theoretical claims and provide a comprehensive evaluation of MLA’s performance.

weaknesses:
* While the theoretical concepts are accurately communicated, the writing could be improved by breaking up overly long sentences to enhance readability (e.g., lines 098-104).
* The results in Tables 6-10 are intriguing, but more explanation is needed to clarify the implications of "Many," "Medium," and "Few" categories. Additionally, the reasons for the Baseline outperforming others in the "Many" category should be discussed in more depth.
* The descriptions for tables and figures are too brief, making it difficult for readers to grasp the full context of the results. Adding more detailed captions would help in understanding the findings.
* The paper contains numerous formal definitions and formulas, which can disrupt the reading flow, especially for readers who are not deeply familiar with this specific research area.

questions:
1. Why did you primarily focus on ResNet models for the image analysis? Were there specific reasons for this choice over other architectures?
2. Why did you create imbalanced versions of balanced datasets instead of using inherently imbalanced datasets? For example, datasets that exhibit natural class imbalances could offer more realistic evaluations.
3. Although the paper focuses on post-hoc LA methods, did you compare the results with loss-function-based approaches? How does MLA's performance compare to other logit adjustment methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713825273

thread_type:
official_review

<thread_object>
creator: reviewer_eiub

<content>
summary:
The paper aims to provide a theoretical justification for Multiplicative Logit Adjustment (MLA) in long-tailed recognition by linking it to Neural Collapse theory. The authors develop a theoretical framework for optimal decision boundary adjustment based on feature spread estimates from Neural Collapse and demonstrate that MLA approximates this optimal method.

strengths:
The framework that connects MLA with Neural Collapse is novel.  The assumptions and conditions are clearly stated, and the notation is consistent and well-defined.

weaknesses:
1. **Motivation**:
   The paper fails to establish a compelling motivation for the study. The claim that "MLA has demonstrated significant empirical success" is not sufficiently substantiated. There is no systematic analysis of the current strengths and challenges of MLA in long-tailed recognition.

2. **Theory-Practice Gap**:
   The connection between optimal decision boundary adjustment and actual performance improvements in long-tailed scenarios is weak. While the paper seems to demonstrate the effectiveness of MLA as a close approximation to the 1vs1adjuster from both theoretical and experimental perspectives, and contrasts it with Additive Logit Adjustment (ALA), it does not thoroughly analyze how MLA addresses the specific challenges of long-tailed learning from a theoretical standpoint. Additionally, there is no comparison with state-of-the-art long-tailed baselines or discussion on how MLA could be integrated with them. The experimental section focuses more on hyperparameter tuning for MLA rather than examining its impact on long-tailed learning.

questions:
The paper mentions that "MLA has demonstrated significant empirical success," but it does not provide sufficient evidence to support this claim. Could more literature or experimental results be added to demonstrate the current advantages of MLA in long-tailed recognition?

The theoretical comparison between MLA and ALA is addressed in some aspects, but why is there no comparison with other advanced baselines in long-tailed recognition (such as the latest contrastive learning methods or distribution-balanced loss)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720247133

thread_type:
official_review

<thread_object>
creator: reviewer_hejv

<content>
summary:
This paper provides a theoretical foundation for the Multiplicative Logit Adjustment (MLA) method used in long-tailed recognition tasks. First, the authors develop a theory for optimally adjusting decision boundaries by leveraging feature spread estimates derived from neural collapse (NC). They then demonstrate that MLA effectively approximates this optimal adjustment method. Experiments conducted on various long-tailed datasets validate the practical applicability of this approximation, showing that MLA performs well under realistic conditions.

strengths:
- This paper presents a theory for optimally adjusting decision boundaries based on NC and links it to the MLA method, providing a strong theoretical explanation for MLA and adding depth to its empirical success.
- The authors conduct a series of experiments on long-tailed datasets to validate the theory and demonstrate the practical utility of MLA.
- The authors offer both theoretical and empirical comparisons between MLA and ALA, clarifying their differences and highlighting the advantages of MLA.

weaknesses:
- The iNaturalist dataset is a widely used large-scale long-tailed dataset. Conducting experiments on this dataset would be beneficial to examine the behavior of MLA, ALA, and the 1vs1 adjuster in a larger-scale real-world context. Additionally, testing on CIFAR-LT with more extreme imbalance ratios would further evaluate the effectiveness of the proposed method.

- The theoretical framework relies on some strict assumptions that may not always hold in practice, such as the assumption of conditions under which NC occurs, which may not be valid in highly imbalanced real-world datasets. Similarly, the assumption that $\psi \rightarrow \pi/2$ does not hold when $K$ is small.
- There are some typos, such as on line 373, where "the optimal decision boundaries during inference is effected by" should be corrected to "the optimal decision boundaries during inference are affected by".

questions:
Please refer to the weakness section.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies the multiplicative logit adjustment (MLA) method proposed recently for the long-tail recognition (LTR) problem. The paper formally shows that MLA can be seen as an approximation of a theoretically-guided approach based on a neural collapse (NC) based adjusting of the optimal decision boundaries. The theoretical analysis also results in some practical benefits, such as better performance in more realistic LTR settings and hyperparameter tuning.

The paper received generally positive reviews from the reviewers. There were some concerns such as (1) lack of experiments on very large-scale LTR datasets such as iNaturalist (Reviewer Hejv), and (2) concerns about some of the theoretical assumptions (Reviewer Hejv), (3) motivation and theory-practice gap (Reviewer EiUb). There were some additional concerns that the experiments are not strongly demonstrating the effectiveness of MLA but are rather focused more on how it guides hyperparameter tuning

The authors provided a detailed response to these concerns. In the end, while two reviewers leaned towards acceptance with scores of 6, Reviewer EiUb remained a bit cautious with a borderline score of 5.

Considering the reviews, the author rebuttal, the discussion, and my own reading of the paper, my assessment is that the paper does offer a nice understanding/justification of MLA, along with some practical guidelines on how to use it effectively in realistic real-world LTR problems. I therefore recommend the paper for acceptance. At the same time, the authors should incorporate the reviewer feedback in the final version.