Stochastic Order Learning: An Approach to Rank Estimation Using Noisy Data

Authors: Chaewon Lee, Seon-Ho Lee, Chang-Su Kim
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: rank estimation, label noise, order learning

Abstract:
A novel algorithm, called stochastic order learning (SOL), for reliable rank estimation in the presence of label noise is proposed in this paper. For noise-robust rank estimation, we first represent label errors as random variables. We then formulate a desideratum that encourages reducing the dissimilarity of an instance from its stochastically related centroids. Based on this desideratum, we develop two loss functions: discriminative loss and stochastic order loss. Employing these two losses, we train a network to construct an embedding space in which instances are arranged according to their ranks. Also, after teaching the network, we identify outliers, which are likely to have extreme label errors, and relabel them for data refinement. Extensive experiments on various benchmark datasets demonstrate that the proposed SOL algorithm yields decent rank estimation results even when labels are corrupted by noise.

Decision: reject
Reviewer scores: 6, 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1730342532057

thread_type:
official_review

<thread_object>
creator: reviewer_g1n2

<content>
summary:
This paper solves the rank estimation problem under noisy data. The main novelty of this work is handling the input data with label noise, which seems limited, as other works can also handle noise data. Moreover, the motivation of the adopted techniques is also not clear. The experimental evaluation of this paper is good.

strengths:
This paper proposes a model to handle the noisy rank learning problem. The experimental evaluation is good.

weaknesses:
Many descriptions are not clear. For example: 
What is the difference between rank estimation and order learning? 
In Eq. (1), the ground truth label is represented by consecutive integers, while the observed noisy rank is assumed to be the sum of the true rank with some Gaussian noise. When the Gaussian noise is added, will the observed rank still be integers?
Besides, why does the noise follow a Gaussian distribution? 
In line 137, what is the relationship between e and e_x?

questions:
Please see the weakness.
Also, please describe the motivation for the proposed method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730387966643

thread_type:
official_review

<thread_object>
creator: reviewer_en6x

<content>
summary:
This paper introduces a novel algorithm called Stochastic Order Learning (SOL) for reliable rank estimation in the presence of label noise. The proposed method models label errors as random variables and formulates a desideratum to minimize dissimilarity from stochastically related centroids. By employing discriminative loss and stochastic order loss, the algorithm constructs an embedding space that effectively arranges instances according to their true ranks. Additionally, it incorporates outlier detection and relabeling schemes to refine noisy data, demonstrating robust performance across various benchmark tasks.

strengths:
1. The proposed algorithm in this paper demonstrates reasonable formulation for rank estimation in the presence of label noise, with clear mathematical derivations. 


2. The experimental section is adequately designed, covering multiple benchmark datasets to validate the algorithm's performance.

weaknesses:
1. The novelty of the paper is relatively moderate. While it builds upon existing research, it lacks significant innovative contributions. 

2. The Dataset Section can be placed in the Appendix to allow more space for detailed analysis of experimental results.


3. The paper lacks comparisons with algorithms from the past two years. Including recent benchmarks would provide a more comprehensive evaluation for the proposed method. Besides, the number of comparison algorithms is too small. Adding more comparison algorithms will further highlight its advantages.

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730550605282

thread_type:
official_review

<thread_object>
creator: reviewer_nnr1

<content>
summary:
This paper presents a novel algorithm to address noise in rank ordering within learning-to-rank tasks. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method.

strengths:
This paper addresses a practical, real-world setting, and the experiments effectively demonstrate the method's effectiveness.

weaknesses:
1. The problem of noisy classification is well-studied, particularly in image classification (e.g., [1]). However, the authors do not clearly explain how their approach to learning with rank labels differs from or improves upon standard classification methods.
2. The writing quality requires improvement. Several notations are undefined, and some equations are presented without adequate derivation or verification.


[1] Xiao, Tong, et al. "Learning from massive noisy labeled data for image classification." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.

questions:
1. Where is the definition of $d(\cdot, \cdot)$ in Eq.(3)
2. How do the authors ensure that the noise $e_x$ remains within the range of true labels in Equations Eq.(1) and Eq.(2)
3. In line 155, could the authors clarify why $\mu_{r_{x + s}}$ the probablity $p_s$?
4. According to Equation (5), does the algorithm require scanning all data?
5. Is the probability in Equation (2) known or assumed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730620809340

thread_type:
official_review

<thread_object>
creator: reviewer_2xn2

<content>
summary:
To solve the rank estimation problem with label noise, this manuscript proposes the stochastic order learning (SOL) method. Specifically, label errors are represented as random variables that follow a Gaussian distribution. Based on this, the stochastic dissimilarity of instance x from rank r is defined and the corresponding discriminative loss and stochastic order loss are designed to train a network to build an embedding space where instances are arranged according to their ranks. Furthermore, after network training, the SOL method identifies outliers that are likely to have extreme label errors and relabel them to refine the dataset.

strengths:
1. This manuscript models label errors as stochastic variables and derives embedding space constraints to arrange instances based on their probabilistically associated ranks.
2. This manuscript proposes outlier detection and relabeling schemes to pinpoint and mitigate instances with extreme label errors, thereby diminishing the dataset's overall noise level.

weaknesses:
1. The reason for setting formulas 12-14 lacks description. Specifically, why is the formula defined in this way? It is necessary to explain the meaning of setting $r_x-r_y-s+t$. In other words, whether I can consider that if the label noise of samples x and y is s and t, respectively, the distance between the true labels of the two samples is $r_x-r_y-s+t$.  
2. In the experiments, there are few comparative methods and no comparison with the state-of-the-art methods. If there are few recent papers about rank estimation with label noise, it can be compared with related methods mentioned in this paper. For example, if rank estimation is regarded as a classification task, is the SOL method better than the approach in the paper [1]? Besides, on datasets with a rank less than 5, whether the SOL method is still performing significantly and better than the method in the paper [2].

[1] Xichen Ye, Xiaoqiang Li, Songmin Dai, Tong Liu, Yan Sun, and Weiqin Tong. Active negative loss functions for learning with noisy labels. In NIPS, 2023.

[2] Huan Liu, Jiankai Tu, Anqi Gao, and Chunguang Li. Distributed robust support vector ordinal regression under label noise. Neurocomputing, pp. 128057, 2024.

questions:
1. The proposed method settings tend to have a risk of labeling errors for each sample. More likely, only a small amount of samples in the dataset is mislabeled. In this situation, is the performance of the method still effective? In addition, the proposed method is based on the key assumption that the label noise of each sample follows the same discrete Gaussian distribution. It is more natural that the standard deviation of noise variables varies among different samples. In the future, a fine-tuning mechanism for setting standard deviations for each sample can be considered while taking into account computational complexity.
2. There are many hyperparameters in the manuscript. Can an adaptive mechanism be designed to select the optimal values of these parameters for different datasets?
3. In the relabeling mechanism, the rank of each detected outlier is adjusted by the same magnitude, that is, half of the average absolute difference between noisy and estimated rankings overall training instances, which seems counterintuitive. It is reasonable to combine outlier detection with the relabeling mechanism to adjust the rank of the outlier.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731473096640

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to thank all reviewers for their time and efforts for providing constructive reviews. We will upload our response to each question or comment as soon as possible.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Thanks for your submission to ICLR.

This paper received four reviews, which were borderline tending to reject.  On the positive side, the reviewers appreciated that this paper addresses a practical problem.  But they had various concerns, including i) issues with the presentation / missing descriptions, ii) issues with the experiments, including missing comparisons to other methods, iii) concerns about the effectiveness / novelty of the approach.

The author response clarified some of these issues, but the negative reviewers were unwilling to raise their scores.  I think the rebuttal did address several of these issues, but I also think (particularly given the overall lack of enthusiasm with the current paper) that the paper could use some additional refinement to address these issues.  With more work it seems that the paper could definitely be publishable.