RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
E.3 Transition Matrix based Methods
Methods based on transition matrices constitute one of the primary approaches for addressing the issue of noisy labels.

Driven by the observation that the clean class posterior, denoted as p(y^{\text{gt}}|x) ,
can be inferred from the transition probability and the noisy class posterior, p(y|x)=T(y|y^{\text{gt}})p(y^{\text{gt}}|x) ,
the modification of the loss function enables the construction of a risk-consistent estimator using the estimated transition matrix [Yao et al., 2020 ] .

There are many approaches aiming to enhance the estimation of the transition matrix.
These include factorizing it into the product of two matrices by introducing an intermediate class [Yao et al., 2020 ] , training the Bayes label transition network [Yang et al., 2022a ] ,
learning the transition matrix within a meta-learning framework [Wang et al., 2020 ] , down-weighting less informative features based on f -mutual information [Zhu et al., 2022 ] ,
and adopting a two-head architecture. The latter involves a noisy classifier for simultaneous transition matrix estimation and a clean classifier for statistically consistent training [Kye et al., 2022 ] .

Moreover, Xia et al. [ 2020 ] explores the utilization of part-dependent transition matrices, combining them to approximate the instance-dependent transition matrix.

In an extended context, Li et al. [ 2022c ] broadens the problem to include noisy multi-label learning and suggests considering label correlations.
E.4 Combination with Contrastive Learning
Incorporating unsupervised learning methods proves effective in alleviating label noise, prompting the integration of noisy label mitigation techniques with unsupervised learning, particularly contrastive learning.

Zhang et al. [ 2021a ] show that the combination of contrastive loss and semi-supervised loss yields successful mitigation of the noisy label problem.

Beyond the application of contrastive learning, other approaches involve selecting confidence pairs and confidence samples [Li et al., 2022b ] ,
leveraging clean probability estimation derived from the relationship between representation clusters and labels [Huang et al., 2023 ] ,
employing class prototypes for weakly-supervised loss [J. Li, 2021 ] , and implementing soft-labeling based on the relation between representations and labels [Ortego et al., 2021 ] .

Additionally, an approach introduces a contrastive regularization function aimed at preventing adverse effects stemming from noisy labels [Yi et al., 2022 ] .
F.1 Dataset Curation Details
Table 7 provides a comprehensive overview of the statistics for the six benchmark datasets meticulously curated for the task of noisy label regression.
Detailed descriptions of the dataset tailoring process are presented below for clarity.

Age prediction datasets (IMDB-Clean-B, AFAD-B, IMDB-WIKI-B, UTKFace-B) : These datasets are harmonized by achieving a balance across distinct age values.
This equilibrium is established using a bin sample count threshold (clip value) of 1000 for IMDB-Clean-B and IMDB-WIKI-B, 1251 for AFAD-B, and 200 for UTKFace-B.
Image inputs are resized to dimensions of (128\times 128) .
For the regression task, we consistently employ a ResNet-50 backbone across all models.

SHIFT15M-B : Achieving data balance in this dataset involves a two-step process.
First, the label space is binned based on a price threshold of \stackon [.4ex] \stackon [.65ex]Y 2000.
Subsequently, data points exceeding the maximum price of \stackon [.4ex] \stackon [.65ex]Y 40000 are clipped to remove outliers.
The binning threshold is set at 16084 sample counts to further ensure balanced representation.
To standardize the label currency, it is pegged to the U.S. dollar, referencing exchange rates from 2010 to 2020, which coincides with the period when the original clothing item data is collected.
Notably, this dataset is provided as the penultimate feature of the ImageNet pre-trained VGG-16 model.
Consequently, we opt for a three-layer MLP architecture with a hidden layer size of [2048, 1024, 512], aligning with recommendations from Papadopoulos et al. [ 2022 ] and Kimura et al. [ 2021 ] .

MSD-B : Achieving balance in the Million Song Dataset involves setting a threshold of 550 samples per year.
For all regression models in this context, we adopt a regression backbone rooted in the tabular ResNet structure proposed by Gorishniy et al. [ 2021 ] , featuring a hidden dimension of 467.

Licenses of existing datasets. IMDB-Clean dataset [Yiming et al., 2021 ] is under MIT license. 2 2 2 https://github.com/yiminglin-ai/imdb-clean SHIFT15M dataset [Kimura et al., 2021 ] is under CC BY-NC 4.0 and MIT license. 3 3 3 https://github.com/st-tech/zozo-shift15m MSD [Bertin-Mahieux et al., 2011 ] song year prediction dataset is under CC BY 4.0 license. 4 4 4 https://archive.ics.uci.edu/dataset/203/yearpredictionmsd UTKFace dataset [Zhifei et al., 2017 ] is available for non-commercial research purposes only. 5 5 5 https://susanqq.github.io/UTKFace/ IMDB-WIKI dataset [Rothe et al., 2015 , 2018 ] is available for academic research purpose only. 6 6 6 https://data.vision.ee.ethz.ch/cvl/rrothe/imdb-wiki/ Unfortunately, the license of AFAD [Niu et al., 2016 ] dataset could not be found. 7 7 7 https://github.com/John-niu-07/tarball
F.3 ConFrag Training Details
ConFrag employs the Cosine Annealing Learning rate [Loshchilov and Hutter, 2017 ] with a minimum learning rate of \eta_{min}=0 .
The optimization is carried out using the Adam optimizer [Kingma and Ba, 2015 ] .
For the K -nearest neighbors-based prediction, we experiment with various values of K , specifically choosing from the set [3,5,7] .
The number of fragments, denoted as F , remains constant at four throughout all the experiments.
To determine the buffer range for jittering, we conduct a search over values within the range [0,0.05,0.1] .

Some dataset-specific hyperparameters exist:

Age prediction task datasets, IMDB-Clean-B [Yiming et al., 2021 ] , AFAD-B [Niu et al., 2016 ] , IMDB-WIKI-B [Rothe et al., 2018 ] , and UTKFace-B [Zhifei et al., 2017 ] , train for 120 epochs
with a learning rate of 0.001.
Each feature extractor employs the ResNet-18 architecture, which contains only 48% of the parameters found in ResNet-50, the architecture utilized for the regressor.

Clothing price estimation task dataset SHIFT15M-B [Kimura et al., 2021 ] trains for 40 epochs
with a learning rate of 0.0001.
MLP with hidden dimensions [1024, 512, 256] is deployed for feature extractors, and the parameter size is 44% of the regressor.

Music year production task dataset MSD-B [Bertin-Mahieux et al., 2011 ] trains for 20 epochs
with a learning rate of 0.0001.
Similar to the regression backbone, the feature extractor model is the tabular ResNet structure [Gorishniy et al., 2021 ] , and the hidden dimension is reduced to 256.
F.4 Random Gaussian Noise
Fig. 9 illustrates the application of random Gaussian noise within the label space of IMDB-Clean-B [Yiming et al., 2021 ] .
The procedure for injecting noise is akin to the approach employed by Yao et al. [ 2022 ] , where Gaussian noise is applied to every unique label within the training samples.
Specifically, Yao et al. [ 2022 ] sets the standard deviation of the Gaussian noise as a fixed 30% of the range of the label space corresponding to the dataset.
In contrast, our noise injection method introduces an element of stochasticity, allowing for variable levels of deviation for each unique label.

To achieve this variability, we employ uniform sampling from the minimum and maximum values specific to each label’s domain.
For instance, in the context of an age prediction task, we assume minimum and maximum values of 0 and 100, respectively.
However, in cases where the label domain lacks clarity ( e.g ., for a variable like ‘price’), we utilize the minimum and maximum label values provided by the dataset itself.

It is important to highlight that baselines with known noise rates, such as CNLCU-S/H, Sigua, and Selfie, are incapable of dealing with Gaussian noise.
Given that these baselines employ a heuristic approach to control selection rates through (1-\text{noise rate}) ,
they prove ineffective when exposed to Gaussian noise, as it introduces noise to all samples, thereby resulting in a nearly 100% noise rate.
Hence, we create a soft noise rate to be used by them for selection.
This is done by calculating an updated noise rate, assuming that the Gaussian noise injected samples that fall within an acceptable variance of the original ground-truth label are clean
(the acceptable variance is set to equal the label length/size of a single fragment).
F.5 Computation Resource
For implementation, we use Python 3.9 and PyTorch 1.13.1.
All experiments are conducted using NVIDIA Quadro 6000 24GB RAM GPUs.
The required computation time for experiments differs depending on the dataset.
Below, we report the computation time for each dataset.

AFAD-B. On the AFAD-B dataset, the average computation time of ConFrag and Co-ConFrag are approximately 2.5 GPU hours and 5 GPU hours, respectively.
The computation time of baselines ranges from 1.25 GPU hours to 4 GPU hours.
About 650 GPU hours were required to produce the AFAD-B part of Tab. 1 .

IMDB-Clean-B. On the IMDB-Clean-B dataset, the average computation time of ConFrag and Co-ConFrag are approximately 3.5 GPU hours and 5.5 GPU hours, respectively.
The computation time of baselines ranges from 2 GPU hours to 5 GPU hours.
About 970 GPU hours were required to produce the IMDB-Clean-B part of Tab. 1 .

SHIFT15M-B. On the SHIFT15M-B dataset, the average computation time of ConFrag and Co-ConFrag are approximately 1 GPU hour and 1.2 GPU hours, respectively.
The computation time of baselines ranges from 6 GPU minutes to 1 GPU hour.
About 100 GPU hours were required to produce the SHIFT15M-B part of Tab. 1 .

MSD-B. On the MSD-B dataset, the average computation time of ConFrag and Co-ConFrag are approximately 4 GPU minutes and 5 GPU minutes, respectively.
The computation time of baselines ranges from 1 GPU minute to 4 GPU minutes.
About 9 GPU hours were required to produce the MSD-B part of Tab. 1 .

Additionally, further computation was required for analysis and experiments in § 4.3 and Appendix G .
G.1 UTKFace Results
Table 8 presents a comparison of MRAE values between various baseline methods and our proposed approach on the balanced UTKFace dataset [Zhifei et al., 2017 ] , UTKFace-B.
We experiment under four different symmetric noise conditions of symmteric 20%, 40%, 60% and 80% noise rates.
Both ConFrag and Co-ConFrag demonstrate superior performance across all experiments when compared to the fourteen baseline methods.
G.2 Fragment Number Analysis
In Fig. 10 and 11 , we undertake an examination of various fragment numbers within the context of symmetric 40% noise, using the IMDB-Clean-B and SHIFT15M-B datasets as benchmarks.
Our evaluation criteria encompass the Selection rate, Error Residual Rate (ERR), and Mean Relative Absolute Error (MRAE).
The number of fragments is chosen from F\in[4,6,8,10] .
To address scenarios with a smaller fragment number, we examine cases where F=1 or 2 .
When F=2 , a fragment f that satisfies self-agreement (Eq. 3 ) does not meet the criteria for neighbor-agreement ( \alpha^{\text{ngb}}_{f} in Eq. 4 ),
as the agreement relies on comparing the distribution of fragment f and its contrasting pair f^{+} .
Consequently, the unified neighborhood agreement (Eq. 4 ) consistently yields a value of 0 .
On the other hand, defining a contrasting pair is not feasible when F=1 .
Instead, we present a plot of the vanilla baseline to illustrate the case when F=1 without utilizing ConFrag.

The results reveal that the MRAE of the vanilla model initially decreases during the early epochs as it learns patterns from clean samples.
However, as the model starts to memorize noisy samples, the MRAE degrades.
In contrast, ConFrag consistently mitigates the impact of noisy samples across all plots ( F\in[4,6,8,10] ) when compared to the vanilla baseline.

We also observe a declining trend in performance as the number of fragments increases in the case of IMDB-Clean-B.
In contrast, SHIFT15M-B exhibits relatively stable performance across different fragment numbers.
This decrease in performance with an increased number of fragments is likely attributed to a finer division of the training data among feature extractors, ultimately leading to overfitting and reduced generalization capabilities.
G.3 Hyperparameter Analysis
The hyperparameter K is used for K -nearest neighbor classification when assessing self/neighbor agreement from a representational perspective.
As shown in Fig. 12 , 13 , with an increase in the value of K , the criteria for agreement become more stringent.
Consequently, as the value of K increases, a greater number of confident samples are selected, resulting in a reduction in the Selection rate and ERR.

The hyperparameter J controls the buffer range for jittering, which, in turn, determines the level of regularization applied via neighborhood jittering.
Increasing the value of J results in stronger regularization, effectively preventing overfitting.
However, excessive regularization, as observed when J=0.10 , may result in adverse effects during training.
Specifically, in Fig. 14 (a), the feature extractors exhibit similar convergence patterns when J=0.05 or J=0.10 .
Consequently, comparable performance is observed in Selection Rate and MRAE.
Yet, in Fig. 14 (b), the ERR of J=0.05 is smaller than that of J=0.10 , leading to improved MRAE performance for J=0.05 .
Similar effects are observed in the SHIFT15M dataset, as depicted in Fig. 15 (SHIFT15M-B).
G.4 Fragment Pairing Analysis
In Fig. 1 (c), we offer deeper insights into our approach by comparing contrastive fragment pairing ( [1,4],[2,5],[3,6] ) against all-fragments ( [1,2,3,4,5,6] ).
In Fig. 6 (a), we show the importance of contrastive fragment pairing by comparing contrastive fragment pairing to alternative pairings.
In Fig. 16 – 17 , we present the extended results with Selection rate, ERR, and MRAE alongside other pairing methods.

The experiments involve training the feature extractors using either contrastive fragment pairing, all-fragments, or alternative pairings.
Notably, a single feature extractor is employed for all fragments, whereas the fragment pairing (contrastive or alternative) uses a smaller feature extractor for each individual pair.
Subsequently, sample selection is executed in accordance with the Mixture of Neighboring Fragments approach (§ 2.3 ).

In an optimal selection algorithm, the Selection rate should approach 100-\text{noise rate}(\%) , with ERR and MRAE minimized.
Across all evaluation metrics, the contrastive fragment pairing demonstrates superior performance compared to other methods.
It is important to highlight that performance is poorest when the pairing is least distinguishable ( [1,2],[3,4],[5,6] when F=6 , [1,2],[3,4] when F=4 ) and moderate when the pairing is partially distinguishable ( [1,6],[2,5],[3,4] when F=6 , [1,4],[2,3] when F=4 ).

Furthermore, in Fig. 18 , we utilize t-SNE to compare the feature extractors trained using contrastive pairing, alternative pairings, and all-fragments.
The visual comparison validates that representations trained with contrastive pairs exhibit significantly more distinguishable features.
G.5 Closed-Set versus Open-Set Noise
To explore the impact of closed-set and open-set noisy samples, as depicted in Fig. 1 (b) in the main manuscript, we conducted an analysis of Selection rate, ERR, and MRAE performance while gradually introducing closed-set and open-set noisy samples into the IMDB-Clean-B dataset.
Our study employs the IMDB-Clean-B dataset, comprising a fixed set of clean samples that represent 40% of the total dataset, alongside varying amounts of noisy samples.
These noisy samples are classified into two distinct categories: closed-set and open-set noise [Wei et al., 2021 , Wan et al., 2024 ] .
For example, consider an example with 4 fragments, whose contrastive fragment pairs are \{(1,3),(2,4)\} .
When training a feature extractor on binary classification between fragment 1 and 3, noisy sample whose ground truth fragment id is either 2 or 4 but mislabeled as fragment id of either 1 or 3 is an open-set noisy sample.
On the other hand, a noisy sample whose ground truth fragment id is 1 but mislabeled as 3 (and vice versa) is a closed-set noisy sample.

Fig. 19 demonstrates that closed-set noisy samples have a considerably more adverse impact on ERR and MRAE compared to open-set noisy samples.
Our contrastive fragment pair-based learning approach is advantageous in this regard, as it introduces open-set noisy samples in lieu of many closed-set noisy samples, thereby facilitating learning with reduced interference.
G.6 Analysis of Samples on the Bounday versus Center of Fragments
Table 9 presents a comparative analysis of the selection rate and error reduction rate (ERR) between samples located at the boundary and the center of fragments across eight experimental configurations.
The results indicate an average difference of 2.29% in selection rates and 2.43% in ERR between the two groups.
These findings substantiate the robustness of ConFrag’s sample selection process, demonstrating consistent performance irrespective of the sample’s positional location within the fragment.
G.7 Ablation & Combination Analysis
In Table 10 , we present a comprehensive study comparing the performance of Cross-Entropy (CE) and Symmetric Cross Entropy (SCE) [Wang et al., 2019 ] losses in various ablation and combination experiments conducted on the IMDB-Clean-B dataset [Yiming et al., 2021 ] , considering scenarios with 40% symmetric noise and two variations of Gaussian random noise, each having a maximum standard deviation of 30 and 50.

Firstly, we illustrate the impact of jittering regularization through ablation on each of the losses.
Notably, jittering regularization emerges as a crucial component of ConFrag’s performance, preventing the model from overfitting to the noisy labels.

The next ablation experiment entails replacing the ResNet-18 architecture of the feature extractors with ResNet-34.
The performance is enhanced when trained with SCE but decreases when trained with just CE.
This suggests that ConFrag could potentially benefit from a more powerful architecture, but it is not a necessity.

A significant advantage of ConFrag lies in its compatibility with other approaches.
We showcase its performance when combined with an additional technique: Co-teaching [Han et al., 2018 ] , which is also employed by CNLCU and Co-Selfie in our baseline.
Co-teaching involves training the regression model while heuristically assuming that 25% of the original noise still exists in the data ( e.g ., 40% original noise implies an assumption of 10% noise during Co-teaching regression).
Empirical observations reveal that Co-teaching consistently provides significant benefits.

Upon comparing CE and SCE for feature extractor training loss, we observe that CE, when combined with jitter regularization, synergizes better to exhibit much stronger performance compared to SCE.
G.8 Discretized Baselines
In Table 11 , we present a discretized version of several strong baselines, including Sigua [Han et al., 2020 ] , CNLCU [Xia et al., 2022 ] , BMM [Arazo et al., 2019 ] , Selfie/Co-Selfie [Song et al., 2019 ] , MD-DYR-SH [Arazo et al., 2019 ] , and CRUST [Mirzasoleiman et al., 2020 ] .

The discretization process aligns with our fragmentation approach used for ConFrag. We obtain selected samples at the end of every epoch to independently train the regression model.
Additionally, we report performance with mixup [Zhang et al., 2018 ] , a technique that proves beneficial for some baselines like Sigua [Han et al., 2020 ] .

Notably, most baselines exhibit a deterioration in performance following discretization.
However, Selfie/Co-Selfie [Song et al., 2019 ] stands out as the exception, showing an improvement in performance after discretization.
Interestingly, Sigua is the sole method that benefits from mixup [Zhang et al., 2018 ] training.
G.9 Comparison of Neighborhood Jittering and Other Regularization Methods
In Fig. 20 , we compare neighborhood jittering with other regularization methods
that can be applied to classification-based feature extractors
(SCE with weight decay [Wang et al., 2019 ] , mixup [Zhang et al., 2018 ] , and their combinations).
In conclusion, neighborhood jittering exhibits the strongest performance in
feature extractor test accuracy, ERR, and MRAE, among other regularization methods.
It is observed that ERR and MRAE improve in line with the performance of the feature extractor.
G.10 Extended Selection Rate/ERR/MRAE Comparison and Analysis
In addition to presenting the Selection rate, ERR and MRAE
for symmetric 40%, Gaussian 30, and Gaussian 50 noise experiments on the IMDB-Clean-B dataset in the main manuscript,
we have included results for all noise types, along with additional baselines (CNLCU-H, Sigua, BMM, DY-S, AUX, Selfie, Coselfie),
in both Fig. 21 and Fig. 22 .

As mentioned in § 4.2 , the ideal scenario for selection and refurbishment methods involves achieving a high selection rate while maintaining a low ERR,
resulting in a reduced MRAE. We examine the relationship between the selection rate, ERR, and MRAE based on Fig. 21 (b).
As training progresses, ConFrag and other selection methods (CNLCU-H, Sigua, BMM, DY-S) approach the ideal condition, resulting in an improving trend in MRAE.
ConFrag, in particular, comes closest to the ideal scenario, resulting in superior MRAE performance.

The most unfavorable scenario arises when there is a low selection rate coupled with a high ERR.
Selfie exemplifies the scenario in Fig. 21 (b), which is connected to a relatively worse MRAE.

The scenarios of the low selection rates with low ERR and the high selection rates with high ERR can be further examined using CNLCU-H and BMM.
CNLCU-H demonstrates superior selection quality in terms of ERR, while BMM exhibits a higher quantity in the selection rate.
This quality/quantity trade-off is linked to the observation that CNLCU-H and BMM show similar MRAE performance in Fig. 21 (b).
Additionally, Fig. 22 (a) reveals that the selection rate gap widens, while the ERR gap narrows when compared to Fig. 21 (b).
This is associated with BMM outperforming CNLCU-H in terms of the MRAE.

It’s important to note that, rather than employing the selection rate and ERR as indicators for MRAE,
these metrics offer valuable insights when assessing selected or refurbished samples
directly independent of any potential regularizing effects introduced by the underlying regression model.

In addition, upon a detailed analysis of the figures, it becomes evident that Co-ConFrag consistently achieves the lowest ERR across a wide range of noise types.
Notably, it maintains a Selection rate of above 40% while maintaining low ERR even in the presence of severe noise conditions, which leads to outstanding MRAE performance.
G.11 Variance Across Random Seeds
In Fig. 23 , we plot the variance of three unique random seed experiments on all six noise types (symmetric 20%/40%/60%/80%, Gaussian 30/50)
on the IMDB-Clean-B dataset. To declutter the graph, we compare it against the top two best-performing baselines under each noise type.

Tab. 12 – 13 show the main experimental results of Tab. 1 with standard deviation.