Contrastive Fragment Pairing
In order to sample a clean data subset \mathcal{S}\subset\mathcal{D} ,
we need to learn a robust feature that can distinguish clean samples from noisy ones.
As one theoretical result in (Zhang et al., 2023 ) , the cross-entropy loss used in classification is better for learning high-entropy feature representation than the mean squared loss in regression (see Appendix D.1 for details).
Based on this, we start by discretizing the label space into F continuous fragments, transforming the original regression problem into the multi-class classification one.
This transformation harnesses an inherent property of regression: data points with similar labels are also represented with closely related features, as acknowledged in prior studies (Gong et al., 2022 ; Yang et al., 2022b ; Yao et al., 2022 ) .

However, instead of training a single feature extractor on the multi-class classification with F classes, we construct F/2 maximally contrasting fragment pairs and train a smaller expert feature extractor for each pair.
The procedure of contrastive fragment pairing is detailed below with an illustration in Fig. 2 :

Divide the range of continuous labels Y into F even number of equal-length fragments.
This allows to divide the dataset \mathcal{D} into F disjoint subsets: \mathcal{D}=\{\mathcal{D}_{1},...,\mathcal{D}_{F}\} , where
each \mathcal{D}_{i} contains the data samples whose y values are in the i -th fragment label range.

Construct a complete graph g=\{\mathcal{D},E\} , where each vertex is a fragment \mathcal{D}_{i} , and each edge weight e_{ij} is the distance in the label space between the closest samples of the fragments (\mathcal{D}_{i},\mathcal{D}_{j}) .

Compute all possible perfect matchings (Monfared and Mallik, 2016 ; Gibbons, 1985 ) , where every vertex of a graph is incident to exactly one edge in the graph.

Find the perfect matching with the largest minimal edge weight: \mathcal{P}=\operatorname*{arg\,max}_{\bar{g}\in\mathcal{G}}\Bigl{(}\min\nu(\bar{g})\Bigr{)} ,
where each \bar{g} is a perfect matching (graph), and \nu(\bar{g}) is the set of edge weights in \bar{g} .
Finally, \mathcal{P}=\{(\mathcal{D}_{i},\mathcal{D}_{j}),\ldots,(\mathcal{D}_{k},\mathcal{D}_{l})\} constitutes the maximally contrasting pairs of fragments.

Motivation behind contrastive fragment pairing. Formulating the multi-class classification problem into F/2 binary classification problems via contrastive fragment pairing has the following advantages.
Firstly, since the distance between fragments in each contrastive fragment pair is large, the feature extractor trained on each contrastive pair can generalize better (Shawe-Taylor and Cristianini, 1998 ; Grønlund et al., 2019 , 2020 ) .
Fig. 1 (c) shows the generalization abilities of the expert feature extractors trained on contrastive fragment pairs compared to the single feature extractor trained on all fragments.
When using a single feature extractor on all fragments (all-frags), the samples selected by the feature extractor tend to become more noisy as the feature extractor overfits, causing the regressor to perform worse over time.
On the other hand, when using multiple feature extractors trained on contrastive pairs, the performance of the regression model consistently improves, indicating that the learned features are more robust and the selected samples are cleaner.
The large distance between fragments also explains why contrastive fragment pairing is superior to other fragment pairings, as shown in § 4.3 .
The analysis of the prediction depth (Baldock et al., 2021 ) in Appendix D.3 supports the claim, as it shows that the binary classification on contrastive fragment pairs results in lower prediction depth, leading to better generalization.

Secondly, the contrastive fragment pairing transforms some of closed-set label noise (whose ground truth is within the label set) into open-set label noise (whose ground truth is not within the label set), as shown in Fig. 1 (a).
Previous works (Wei et al., 2021 ; Wan et al., 2024 ) observe that the open-set noise is less harmful than the closed-set noise and may even benefit generalization and robustness against inherent noisy labels.
Indeed, in our experiments, we found similar observations where injecting open-set label noise is less harmful than closed-set one, as shown in Fig. 1 (b).

The t-SNE visualization in Fig. 1 (a) also supports this observation.
Let f and f^{\text{gt}} be fragments that the observed label y and the groundtruth label y^{\text{gt}} respectively belong to.
Prior to contrastive fragment pairing, all of the noisy labeled data ( f\neq f^{\text{gt}} ) are closed-set noise as their ground truth fragment ids are within the label set ( f^{\text{gt}}\in[1\text{-}6] ) and their features are located in the feature spaces of incorrect classes within the group.
After contrastive fragment pairing, much of these noisy labeled data is transformed to open-set noise ( f^{\text{gt}}\notin[1,4] while f\in[1,4] in case of fragment pair [1,4] ),
and their learned features tend to reside outside the feature clusters of the clean samples, thus mitigating the adverse effects of the noise.