RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
ConFrag: Contrastive Fragmentation
In the noisy label regression problem, we are presented with a dataset denoted as \mathcal{D}=\{\mathcal{X},Y\} ; in each sample (x,y) , x\in\mathbb{R}^{d} is an input, and y\in\mathbb{R} is the observed label, which can be possibly noisy. We use y^{\text{gt}} to denote the groundtruth label.
The objective of ConFrag is to sample a clean subset of the data as \mathcal{S}\subset\mathcal{D} .
By training on \mathcal{S} , we aim to enhance the performance of the regression model.

An overview of our ConFrag framework is shown in Fig. 3 (a).
The framework has the following steps.
We divide the dataset into what we refer to as contrastive fragment pairs (§ 2.1 ), which collectively enhance the training of the feature extractors (§ 2.2 ).
We then select clean samples \mathcal{S} from dataset \mathcal{D} based on neighborhood agreements, utilizing a fragment-based mixture model (§ 2.3 ). A regression model is trained on the clean samples \mathcal{S} .
We also propose neighborhood jittering as a regularizer for further improved training (§ 2.4 ).
ConFrag is noise rate-agnostic unlike prior methods as it operates without knowing a pre-defined noise rate.
Training Feature Extractors for Contrastive Pairs
Once we obtain the contrastive fragment pairs \mathcal{P} , we train F/2 number of expert feature extractors on binary classification p(y|x;\theta_{i,j}) with its respective contrastive pair (\mathcal{D}_{i},\mathcal{D}_{j})\in\mathcal{P} , where \theta_{i,j} denotes the parameter of an expert.
That is, it is trained to predict whether a data x is in \mathcal{D}_{i} or \mathcal{D}_{j} .
Later, the feature extractors play a crucial role in determining whether a sample (x,y) is clean.
Neighborhood Jittering
A potential limitation of mixture models is that the individual expert feature extractor may not fully benefit from the full dataset as they model their own disjoint subsets (Dukler et al., 2023 ) .
Our neighborhood jittering mitigates this limitation as a robust regularizer that expands the effective coverage of each contrastive fragment pair during learning.
The process is visualized in Fig. 3 (d).

We bound the ratio of the jittering buffer range within [0,\frac{1}{2(F-1)}] , where F is the fragment number.
For every epoch, we shift the label coverage of each fragment by randomly sampling the value in this range. Jittering leads to a partially overlapping mixture model (Heller and Ghahramani, 2007b ; Hinton, 2002 ) as some data belong to multiple, neighboring fragments and thus the effective coverage per each expert is expanded.
Such regularization inhibits feature extractors from overfitting to potentially noisy samples and promotes learning of more robust features, even those that can be generalizable to overlapping parts of neighboring fragments.

Fig. 4 (a) shows that with jittering, the feature extractor exhibits higher accuracy on the clean test data due to its regularization effect.
In the sample selection stage (Fig. 4 (b)),
the feature extractor trained without jittering easily overfits the noise, resulting in over-selection and higher ERR (§ 4.2 ).
In contrast, the jittered feature extractor achieves a relatively low selection rate with halved ERR, indicating that the noisier samples are filtered out.
Better sample selection due to jittering subsequently leads to significantly better performance in regression (Fig. 4 (c)).
In Appendix G.9 , we compare neighborhood jittering to other regularizations, demonstrating its efficacy.
Related Works
We review prior works on learning with noisy labels and defer a comprehensive survey to Appendix E . We organize them into those utilizing prediction, representation, and combination of the two.

Prediction-based Methods .
This approach has been the focus of much existing research and covers a wide array of topics: (i)
the small loss selection by exploring the pattern of memorization in neural networks (Han et al., 2018 ; Arazo et al., 2019 ) ,
(ii) relying on the consistency of predictions to select or refurbish the samples (Liu et al., 2020 ; Huang et al., 2020 ) ,
(iii) estimating the noise distribution (Patrini et al., 2017 ; Hendrycks et al., 2018 ) ,
(iv) introducing auxiliary parameters or labels (Pleiss et al., 2020 ; Hu et al., 2020 ) ,
(v) using unlabeled data with semi-supervised learning (Li et al., 2020a ; Bai et al., 2021 ; Karim et al., 2022 ) ,
and (vi) designing a noise-robust loss function (Menon et al., 2020 ; Wang et al., 2019 ) .

Representation-based Methods .
This approach has seen a recent surge in interest, including
(i) clustering based selection (Mirzasoleiman et al., 2020 ; Wu et al., 2020a ) ,
(ii) feature eigendecomposition filtering (Kim et al., 2021 ) , (iii) using neighbor information to sample and refurbish with clean validation (Li et al., 2022a ; Gao et al., 2016 ) ,
and (iv) generative models of features for sampling (Lee et al., 2019 ) .

Combination .
Some works have also studied the combination of representation and prediction spaces. Wang et al. ( 2022 ) formulate a penalized regression between the network features and the labels for selection,
and Ma et al. ( 2018 ) use intrinsic dimensionality and consistent predictions to refurbish.
Other important approaches include (i) regularization via MixUp (Zhang et al., 2018 ) along with its regression version (Yao et al., 2022 ) ,
(ii) model-based methods that discourage large parameter shifts (Hu et al., 2020 ) , and (iii) importance discrimination of parameter updates (Xia et al., 2021 ) .

The majority of previous works have studied noisy labels for classification. Hence, a large portion of these works may not be directly applicable to regression tasks due to the restricted usage of class-wise information. In § 4 , we empirically compare our method with some of these works that are expandable to the regression task with some or minor technical adaptation.
Experiments
We compare ConFrag with fourteen strong baselines adapted for noisy label regression.
Due to the scarcity of benchmark datasets, we update existing datasets for the study of noisy labels.
Settings
Curation of Benchmark Datasets. We create six benchmark datasets for noisy labeled regression to encompass a sufficient quantity of balanced data, span multiple domains, and present a meaningful level of complexity. (i) Age Prediction from an image is a well-studied regression problem (Li et al., 2019 ; Shin et al., 2022 ; Lim et al., 2020 ) .
To address this domain, we acquire four datasets of AFAD (Niu et al., 2016 ) , IMDB-Clean (Yiming et al., 2021 ) , IMDB-WIKI (Rothe et al., 2018 ) , and UTKFace (Zhifei et al., 2017 ) .
Notably, IMDB-WIKI contains real-world label noise stemming from the automatic web crawling process (Yiming et al., 2021 ) .
We use a ResNet-50 backbone for all datasets. (ii) Commodity Price Prediction is a vital real-world task (Wen-Huang et al., 2021 ) .
We opt for the SHIFT15M dataset (Kimura et al., 2021 ) due to the diversity and scale of this domain.
This dataset is provided as the penultimate feature of the ImageNet pre-trained VGG-16 model.
Consequently, we use a three-layer MLP architecture for all experiments (Papadopoulos et al., 2022 ; Kimura et al., 2021 ) .
(iii) Music Production Year Estimation uses the tabular MSD dataset (Bertin-Mahieux et al., 2011 ) .
This dataset is identified as one of the most intricate and challenging datasets, based on the test R2 score (Grinsztajn et al., 2022 ) .
We adopt a tabular ResNet proposed by Gorishniy et al. ( 2021 ) .
The suffix “-B” is appended to the dataset name ( e.g ., AFAD-B) to indicate that it is a curated version of the original dataset.
To focus on the noisy label problem, we take measures to balance the datasets as elaborated in Appendix F.1 .

Experimental Design. For all datasets except for IMDB-WIKI-B which contains real-world label noise,
we inject symmetric and Gaussian noise into the labels, as done in prior literature (Yao et al., 2022 ; Yi and Wu, 2019 ; Wei et al., 2020 ) .
These types of noise can simulate a low-cost (human-free) controlled setting.
Symmetric noise mimics randomness such as Web crawling or annotator errors, and
Gaussian noise is often used for modeling the regression label noise. While Yao et al. ( 2022 ) inject a fixed 30% standard deviated Gaussian noise for every label ,
we make it more realistic by randomizing the standard deviation up to 30% or 50% of the domain’s range.
For our ConFrag experiments, we fix the fragment number ( F ) as four. See Appendix F.3 for further training details.

Baselines. There are many existing methods of noisy labeled learning for classification.
We assess fourteen baselines from the three branches that are naturally adaptable to regression with minor or no updates. (i) Small loss selection: CNLCU-S,H (Xia et al., 2022 ) , Sigua (Han et al., 2020 ) , SPR (Wang et al., 2022 ) , BMM (Arazo et al., 2019 ) , DY-S (Arazo et al., 2019 ) , SuperLoss (Castells et al., 2020 ) .
(ii) Regularization: C-mixup (Yao et al., 2022 ) , RDI (Hu et al., 2020 ) , CDR (Xia et al., 2021 ) , D2L (Ma et al., 2018 ) .
(iii) Refurbishment: AUX (Hu et al., 2020 ) , Selfie (Song et al., 2019 ) , Co-Selfie (Song et al., 2019 ) .
Appendix F.2 comprehensively details these baselines.
Evaluation Metrics
We mainly report the Mean Relative Absolute Error (MRAE) following prior works.
The MRAE is computed as (e/\rho)-1 , where e is the model’s Mean Absolute Error (MAE) performance under varying conditions (noise type, severity) and \rho is the noise-free model’s MAE. We express MRAEs in percentage for better comprehensibility.
The traditional MAE values are also reported in Appendix G.12 .
In addition, we report the Selection rate (a.k.a prevalence), which is a metric often seen in noisy labeled classification to quantify the coverage of the total dataset, |\mathcal{S}|/|\mathcal{D}| where \mathcal{S} and \mathcal{D} are the selected and total set, respectively.

Error Residual Ratio .
To better assess selection and refurbishment approaches, we introduce a new metric termed Error Residual Ratio (ERR).
Unlike classification, noisy labels in regression can show the diverse severity of the noise present in each label y ( i.e ., various degrees of deviation from the ground truth y^{\text{gt}} ).
This cannot be addressed when using conventional metrics, which are primarily designed for classification and tend to treat all instances of noise as equally severe.
Our proposed ERR considers the varying severity of noise and is defined as

ERR=\frac{1/|\mathcal{C}|\sum_{c}^{|\mathcal{C}|}|y_{c}-y^{gt}_{c}|}{1/|\mathcal{D}|\sum_{d}^{|\mathcal{D}|}|{y}^{ }_{d}-y^{gt}_{d}|}, (5)

where \mathcal{C} is a set of cleaned (selected or refurbished) samples.
The numerator is the average cleaned error that serves as an indicator of the precision of the cleaned data, while the denominator is the average dataset error that normalizes it for standardized assessment.
The ERR, along with the selection rate and regression metrics ( e.g ., MSE, MRAE), provides a deeper insight into the model performance.
Ideally, a method with a high selection rate coupled with low ERR and regression error can be deemed as closer to the upper bound.
Conclusion
To address the problem of noisy labeled regression, we introduce the Contrastive Fragmentation framework (ConFrag).
The framework partitions the label space and identifies the most contrasting pairs of fragments, thereby training a mixture of feature extractors over contrastive fragment pairs.
This mixture is leveraged for clean selection based on neighborhood agreements.
Extensive experiments on six curated datasets on three domains with different levels of symmetric and Gaussian noise demonstrate that our framework performs superior selection and ultimately leads to a better regression performance than fourteen state-of-the-art models.
Given its foundation in the Mixture of Experts model, the parameter size of ConFrag linearly grows with an increase in the number of fragments.
We acknowledge this as a potential avenue for future research.
Acknowledgement
We express our gratitude to the members of the Vision and Learning Lab at Seoul National University for their valuable feedback on the manuscript.
In particular, we would like to acknowledge Jaekyeom Kim, Soochan Lee, Jinseo Jeong, Wonkwang Lee, and Sehun Lee.
This work was supported by
LG AI Research,
Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2019-II191082, SW StarLab), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2022-II220156, Fundamental research on continual meta-learning for quality enhancement of casual videos and their 3D metaverse transformation), the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2023R1A2C2005573), and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2021-II211343, Artificial Intelligence Graduate School Program (Seoul National University)). Gunhee Kim is the corresponding author.
D.1 Classification versus Regression for Feature Learning
During the learning process, deep neural networks aim to maximize the mutual information between the learned representation, denoted as Z , and the target variable, denoted as Y .
The mutual information between these two variables can be defined as I(Z;Y)=H(Z)-H(Z|Y) . A high value of I(Z;Y) is indicative of a high marginal entropy H(Z) .
Achieving this dual objective is accomplished in classification [Boudiat et al., 2020 ] .

However, Zhang et al. [ 2023 ] have shown that regression primarily focuses on minimizing H(Z|Y) while disregarding H(Z) .
This results in a relatively lower marginal entropy for the learned representation Z and ultimately leads to performance deficits in comparison to classification.

To experimentally show that this theoretical result also applies to ConFrag, we replace classification-based expert feature extractor learning with regression-based one, where each expert feature extractor is trained with regression loss on its respective fragment pair dataset.
We name this variant ConFrag-R.
In ConFrag-R, self-agreement is defined using distances to the mean of each fragment in the contrasting pair (f,f^{+}) :

\alpha^{self}_{f} =\left[|\bar{Y}_{f}-h(x;\theta_{f_{,}f^{+}})|<|\bar{Y}_{f^{+}}-h(x;\theta_{f_{,}f^{+}})|\right], (6)

where \bar{Y}_{f} is the average of fragment f ’s labels, and h_{R}(\cdot) is the regression function output.
As in ConFrag, ConFrag-R also utilizes K -nearest neighbor-based classification for computing another variant of self-agreement. The results in Table 4 show that using classification for feature learning outperforms using regression (ConFrag-R) in all datasets.
D.2 Fragmentation and Neighborhood Jittering
ConFrag operates by partitioning data samples into fragments and leveraging trained feature extractors for sample selection through collective modeling.
We conceptualize this as a Mixture-of-Experts (MoE) model, wherein individual experts specialize in specific problem subspaces through data partitioning [Yuksel et al., 2012 , Masoudnia and Ebrahimpour, 2014 ] .
MoEs possess theoretically advantageous properties with respect to computational scalability and reduction of output variance [Yuksel et al., 2012 ] , contributing to the enhancements observed in ConFrag.
It is noteworthy that since each network is trained on a distinct training set, MoE effectively mitigates concurrent failures, thereby preventing error propagation among networks and ultimately improving the generalization performance of ConFrag as well [Sharkey and Sharkey, 1997 ] .

Additionally, our Neighborhood Jittering leads to a Partially Overlapping Mixture Model [Heller and Ghahramani, 2007a ] , theoretically enabling the modeling of significantly richer and more intricate hidden representations by accommodating multi-cluster membership, ultimately enhancing the selection and overall performance of ConFrag.
D.3 Prediction Depth Analysis
Prediction depth [Baldock et al., 2021 ] of an example refers to the earliest layer where the layer-wise K -nearest neighbor probes of the layer and all the subsequent layers are the same as the model prediction.
In other words, low prediction depth means that the example is easily distinguishable in early layers.
For example, a prediction depth of zero means that data can be predicted at the input level only based on its distances to other data.
Low prediction depth is positively correlated with better prediction consistency, lower learning difficulty, and larger margin.
Due to these traits, some previous works aim at reducing the prediction depth during training for better generalization performance [Zhou et al., 2022 , Sarfi et al., 2023 ] .

While prediction depth is initially designed as a measure of example difficulty, the mean prediction depth of the dataset can also be used as a measure of dataset difficulty [Baldock et al., 2021 ] .
Also, since early layers generalize while later layers memorize in deep learning [Stephenson et al., 2021 ] , the low mean prediction depth of a dataset means it is more generalizable since fewer examples require memorization.

Tab. 5 – 6 show the mean prediction depth of all samples for each feature extractor learning task, with and without noise.
The prediction depth for each task is measured using ResNet-18 trained for 20 epochs, at which the model achieves more than 99% training accuracy.
Following Baldock et al. [ 2021 ] , we use K=30 for K -nearest neighbor probe and did not use data augmentation during training.
The tables show that the binary classification tasks from contrastive pairing achieve much lower prediction depths than the multi-class classification tasks using all fragments (All-frag).
Also, the mean and maximum prediction depths of contrastive pairing are lower than those of alternative pairings, explaining why contrastive pairing outperforms alternative pairings as shown in § 4.3 .
Note that the mean prediction depths of the binary classification tasks correlate with the distance between fragments: the larger the distance, the lower the prediction depth tends to be.

Fig. 7 – 8 show the distribution of prediction depth for each task when F=4 and F=6 .
Without noise, about 40% of data in each contrastive fragment pair has a prediction depth of zero, indicating that two fragments are already much separated at input level.
Even with 40% symmetric noise, more than 25% of data in each contrastive fragment pair has a prediction depth of zero.
Meanwhile, the most frequent prediction depth when using all fragments is nine, which means that most data can only be predicted at the last hidden feature level.

While the prediction depths of alternative pairings are lower than those of using all fragments, we observe that they tend to perform worse as shown in Appendix G.4 .
We suspect that this is due to the limitation of mixture models that the individual expert feature extractor may not fully benefit from the full dataset as they model their own disjoint subset.
E.1 Continuously Ordered Correlation of Labels and Features
One distinctive characteristic of regression problems is their continuous label space, implying a high likelihood of correlation between regions within the feature and label spaces [Yang et al., 2022b , Gong et al., 2022 , Zha et al., 2022 ] .

Recent research has extensively explored these characteristics, encompassing issues such as label imbalance [Yang et al., 2022b , Gong et al., 2022 ] , age estimation [Li et al., 2019 ] , contrastive learning [Zha et al., 2022 ] , and mixup regularization [Yao et al., 2022 ] .

Yang et al. [ 2022b ] propose label and feature distribution smoothing based on their similarity, while Gong et al. [ 2022 ] introduce a regularization term aimed at aligning the rankings of feature-space and label-space neighbors. Zha et al. [ 2022 ] employ supervised contrastive learning with a pairing technique based on label distances in mini-batches.
To adapt MixUp [Zhang et al., 2018 ] for regression tasks, Yao et al. [ 2022 ] recommend interpolating proximal samples within the label space with a higher probability.

Ordinal regression, also known as ranking learning, pertains to predicting ordinal labels based on input data.
It is noteworthy that ordinal regression methods are adaptable for regression tasks due to the inherent numerical ordering within scalar label spaces.
Past studies in ordinal regression have successfully addressed various regression challenges, including facial age estimation [Niu et al., 2016 , Shin et al., 2022 ] , monocular depth estimation [Fu et al., 2018 ] , and credit rating [Hirk et al., 2019 ] .
Some of these methods share common characteristics with our approach, as they discretize continuous labels, effectively converting regression tasks into classification problems [Niu et al., 2016 , Fu et al., 2018 , Shah et al., 2022 ] .
Within the framework of ordinal regression, Garg and Manwani [ 2020 ] propose a loss correction method by estimating the noise transition matrix.

It is important to note that among the previously mentioned methods, only Yao et al. [ 2022 ] and Garg and Manwani [ 2020 ] can effectively address noisy label regression problems without the need for additional techniques.
Additionally, Wang et al. [ 2022 ] enhance the scalability of their approach by grouping dissimilar classes within the feature space.
Our work considers the continuity of labels and features and their correlation in fragmenting and grouping data.
This approach allows each component to learn distinguishable features and improve sample selection capabilities.
E.2 Noisy Label in Object Detection
Due to the abundance of research on object detection tasks, with bounding box localization being a prominent example of regression tasks,
we have explored the issue of noisy regression within the context of object detection.
In particular, obtaining accurate annotations for object detection is a resource-intensive task, often constrained by limited time, a small number of annotators, or reliance on machine-generated annotations.
These constraints frequently result in label noise, represented as incorrect class assignments or inaccurate bounding box locations.

Various strategies have been developed to address the issue of noisy labels in object detection.
To correct inaccurate bounding box locations, Li et al. [ 2020b ] leverage the discrepancy between two classification heads by emphasizing the objectness of the region. Liu et al. [ 2022 ] generates object bags using the classifier as guidance, Mao et al. [ 2021 ] employs center-matching correction, and Schubert et al. [ 2023 ] drop instances with high region proposal loss on an instance-wise basis.
In scenarios where image-level annotations are available, Gao et al. [ 2019 ] employs ensemble learning with two classification heads and a distillation head,
while Shen et al. [ 2020 ] decomposes the problem into foreground and background noise, employing residual learning and bagging-mixup learning.

We also explored the possibility of applying object detection techniques to noisy labeled regression.
However, our analysis revealed that these methods are not well-suited for the broader regression task.
Specifically, Liu et al. [ 2022 ], Schubert et al. [ 2023 ], Mao et al. [ 2021 ] utilize region proposal networks to generate bounding box proposals.
They leverage these proposals to selectively choose clean labels or re-weight the training samples.
However, because this approach necessitates an auxiliary model in the proposal generation process, it cannot be directly applied in the context of regression tasks.

Additionally, Li et al. [ 2020b ], Liu et al. [ 2022 ], Schubert et al. [ 2023 ], Gao et al. [ 2019 ] employ the object detector’s classifier to update or assess the quality of bounding boxes.
By evaluating the confidence or consistency of the bounding box through the classification output, this approach helps mitigate the impact of noisy labels.
However, implementing a similar approach in the context of regression tasks would require the inclusion of an auxiliary co-trained task.