RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
G.12 Standard Mean Absolute Error
In Tables 14 – 15 , we report the standard mean absolute error (along with standard deviation) within the respective label ranges for each dataset.

[tb] Contrastive Fragmentation {algorithmic} \STATE Input: Train data \mathcal{D}=\{\mathcal{X},Y\} , Fragment number F , KNN parameter K , Jitter J , Total epochs N \STATE \STATE \Theta=\{\theta_{0,0}\ldots\theta_{i,j}\} \COMMENT feature extractors \STATE \Phi=RandomInit() \COMMENT regression model \STATE \STATE \mathcal{D}_{1\ldots F}=Fragmentation(\mathcal{D}) \COMMENT § 2.1 . 1 \STATE \mathcal{P}=ContrastivePairing(\mathcal{D}_{1\ldots F}) \COMMENT § 2.1 . 2 \sim 4 \FOR n to N \STATE # train feature extractors \STATE \mathcal{P}^{jitter}=NeighborhoodJittering(\mathcal{D},F,J) \COMMENT neighborhood jittering (§ 2.4 ) \FOR (\mathcal{D}_{i}^{jitter},\mathcal{D}_{j}^{jitter}) in \mathcal{P}^{jitter} \STATE \mathcal{D}_{i,j}^{jitter}=\mathcal{D}_{i}^{jitter}\cup\mathcal{D}_{j}^{jitter} \STATE \text{train }p(f;\theta_{i,j},\mathcal{D}_{i,j}^{jitter}) \ENDFOR \STATE \STATE \color[rgb]{0,0,1}{\text{\# initialize }\mathcal{S},\mathcal{S}^{p},\mathcal{S}^{r}} \STATE \mathcal{S},\mathcal{S}^{p},\mathcal{S}^{r}=\{\},\{\},\{\} \COMMENT selected samples \STATE \STATE \color[rgb]{0,0,1}{\text{\# obtain }\mathcal{S}^{p},\mathcal{S}^{r}} \FOR (x,y) in \mathcal{D} \FOR f=1 to F \STATE \text{calculate }\rho^{\text{}}_{f}(y) \COMMENT fragment prior (Eq. 2 ) \STATE # use two types of classification for neighborhood agreement \STATE \text{calculate }\alpha_{f}^{p}(x;\mathcal{D}_{1\ldots F},\Theta) \COMMENT predictive neighborhood agreement (Eq. 4 ) \STATE \text{calculate }\alpha_{f}^{r}(x;\mathcal{D}_{1\ldots F},\Theta) \COMMENT representational neighborhood agreement (Eq. 4 ) \ENDFOR \STATE p^{p}(s|x,y,\mathcal{D}_{1\ldots F};\Theta)=\sum_{f}^{F}\rho^{\text{}}_{f}(y)\alpha_{f}^{p}(x;\mathcal{D}_{1\ldots F},\Theta) \COMMENT pred. sample probability (Eq. 1 ) \STATE p^{r}(s|x,y,\mathcal{D}_{1\ldots F};\Theta)=\sum_{f}^{F}\rho^{\text{}}_{f}(y)\alpha_{f}^{r}(x;\mathcal{D}_{1\ldots F},\Theta) \COMMENT repr. sample probability (Eq. 1 )

\text{sample }\{u^{p},u^{r}\}\sim uniform(0,1) \IF p^{p}(s|x,y,\mathcal{D}_{1\ldots F};\Theta)>u^{p} \STATE \mathcal{S}^{p}=\mathcal{S}^{p}\cup(x,y) \ENDIF \IF p^{r}(s|x,y,\mathcal{D}_{1\ldots F};\Theta)>u^{r} \STATE \mathcal{S}^{r}=\mathcal{S}^{r}\cup(x,y) \ENDIF \ENDFOR \STATE \STATE \color[rgb]{0,0,1}{\text{\# union filtered samples }(\mathcal{S}^{p},\mathcal{S}^{r})} \STATE \mathcal{S}=\mathcal{S}^{p}\cup\mathcal{S}^{r} \STATE \STATE # train regression model \STATE \Phi=TrainOneEpoch(S;\Phi) \STATE \ENDFOR
Figure 1: (a) An example of t-SNE illustration of contrastive fragment pairing .
The data with label noise are grouped into six fragments ( f\in[1\text{-}6] ) and formed into three contrastive pairs ( f\in[1,4],[2,5],[3,6] ).
Contrastive fragment pairing transforms some of closed-set noise (whose ground truth is within the target label set) into open-set noise (whose ground truth is not within the label set).
For example, in the [1,4] figure, label noise whose ground truth fragment is either 1 or 4 is closed-set noise, and the others are open-set noise.
The t-SNE illustration shows that learned features of open-set noises tend to reside outside the feature clusters of the clean samples.
(b) The open-set noise is less harmful with much lower errors (MRAE) in the downstream regression.
(c) The contrastive pairing ( [1,4],[2,5],[3,6] ) is more effective than using all-fragments together ( [1\text{-}6] ), resulting in much lower MRAE scores.
All experiments are based on IMDB-Clean-B with more details in Appendix G.4 – G.5 .; (a) An example of t-SNE illustration of contrastive fragment pairing .
The data with label noise are grouped into six fragments ( f\in[1\text{-}6] ) and formed into three contrastive pairs ( f\in[1,4],[2,5],[3,6] ).
Contrastive fragment pairing transforms some of closed-set noise (whose ground truth is within the target label set) into open-set noise (whose ground truth is not within the label set).
For example, in the [1,4] figure, label noise whose ground truth fragment is either 1 or 4 is closed-set noise, and the others are open-set noise.
The t-SNE illustration shows that learned features of open-set noises tend to reside outside the feature clusters of the clean samples.
(b) The open-set noise is less harmful with much lower errors (MRAE) in the downstream regression.
(c) The contrastive pairing ( [1,4],[2,5],[3,6] ) is more effective than using all-fragments together ( [1\text{-}6] ), resulting in much lower MRAE scores.
All experiments are based on IMDB-Clean-B with more details in Appendix G.4 – G.5 .
Figure 2: The contrastive fragment pairing algorithm.; The contrastive fragment pairing algorithm.
Figure 3: Contrastive Fragmentation framework. (a) The overall sequential process of our framework.
(b) Shows the fragmentation of the continuous label space to obtain contrasting fragment pairs (§ 2.1 ) and train feature extractors on them.
(c) Sample Selection by Mixture of Neighboring Fragments obtains the selection probability in both prediction and representation perspectives (§ 2.3 ).
(d) Illustration of Neighborhood Jittering (§ 2.4 ).; Contrastive Fragmentation framework. (a) The overall sequential process of our framework.
(b) Shows the fragmentation of the continuous label space to obtain contrasting fragment pairs (§ 2.1 ) and train feature extractors on them.
(c) Sample Selection by Mixture of Neighboring Fragments obtains the selection probability in both prediction and representation perspectives (§ 2.3 ).
(d) Illustration of Neighborhood Jittering (§ 2.4 ).
Figure 4: Jittering analysis. (a) When trained without jittering, feature extractors easily overfit the noisy training data (yellow-shaded region), while jittering-regularized feature extractors robustly learn from the noisy training data. (b) Overfitted feature extractors (yellow-shaded region) on noisy samples increase their likelihood, leading to a higher selection rate and ERR.
It exhibits nearly twice higher ERR (a lower value is better).
(c) Most importantly, jittering regularization improves performance in regression.; Jittering analysis. (a) When trained without jittering, feature extractors easily overfit the noisy training data (yellow-shaded region), while jittering-regularized feature extractors robustly learn from the noisy training data. (b) Overfitted feature extractors (yellow-shaded region) on noisy samples increase their likelihood, leading to a higher selection rate and ERR.
It exhibits nearly twice higher ERR (a lower value is better).
(c) Most importantly, jittering regularization improves performance in regression.
Figure 5: Selection/ERR/MRAE comparison between ConFrag and strong baselines of CNLCU-H, BMM, DY-S, AUX and Selfie on IMDB-Clean-B. We exclude the performance during the warm-up.; Selection/ERR/MRAE comparison between ConFrag and strong baselines of CNLCU-H, BMM, DY-S, AUX and Selfie on IMDB-Clean-B. We exclude the performance during the warm-up.
Figure 6: Analysis with 40% symmetric noise. (a) Comparison between the proposed contrastive pairing and other pairings on IMDB-Clean-B.
(b) Comparison between fragment numbers on SHIFT15M-B and IMDB-Clean-B.; Analysis with 40% symmetric noise. (a) Comparison between the proposed contrastive pairing and other pairings on IMDB-Clean-B.
(b) Comparison between fragment numbers on SHIFT15M-B and IMDB-Clean-B.
Figure 7: Probability of prediction depth for examples in contrastive pairing and all-frag ( F=4 ).; Probability of prediction depth for examples in contrastive pairing and all-frag ( F=4 ).
Figure 8: Probability of prediction depth for examples in contrastive pairing and all-frag ( F=6 ).; Probability of prediction depth for examples in contrastive pairing and all-frag ( F=6 ).
Figure 9: Random Gaussian Noise. (a) Gaussian noise injected from the uniformly sampled random standard deviation between [1,30] .
(b) Gaussian noise injected from uniformly sampled random standard deviation between [1,50] .; Random Gaussian Noise. (a) Gaussian noise injected from the uniformly sampled random standard deviation between [1,30] .
(b) Gaussian noise injected from uniformly sampled random standard deviation between [1,50] .
Figure 10: Fragment number analysis compares the Selection rate, ERR and MRAE on IMDB-Clean-B with symmetric 40% noise.; Fragment number analysis compares the Selection rate, ERR and MRAE on IMDB-Clean-B with symmetric 40% noise.
Figure 11: Fragment number analysis compares the Selection rate, ERR and MRAE on SHIFT15M-B with symmetric 40% noise.; Fragment number analysis compares the Selection rate, ERR and MRAE on SHIFT15M-B with symmetric 40% noise.
Figure 12: Hyperparameter K analysis compares the Selection rate, ERR and MRAE on IMDB-Clean-B with symmetric 40% noise.; Hyperparameter K analysis compares the Selection rate, ERR and MRAE on IMDB-Clean-B with symmetric 40% noise.
Figure 13: Hyperparameter K analysis compares the Selection rate, ERR and MRAE on SHIFT15M-B with symmetric 40% noise.; Hyperparameter K analysis compares the Selection rate, ERR and MRAE on SHIFT15M-B with symmetric 40% noise.
Figure 14: Hyperparameter J analysis compares the average accuracy of feature extractors, the Selection rate, ERR and MRAE on IMDB-Clean-B with symmetric 40%, 60% noise.; Hyperparameter J analysis compares the average accuracy of feature extractors, the Selection rate, ERR and MRAE on IMDB-Clean-B with symmetric 40%, 60% noise.
Figure 15: Hyperparameter J analysis compares the average accuracy of feature extractors, the Selection rate, ERR and MRAE on SHIFT15M-B with symmetric 40%, 60% noise.; Hyperparameter J analysis compares the average accuracy of feature extractors, the Selection rate, ERR and MRAE on SHIFT15M-B with symmetric 40%, 60% noise.
Figure 16: Fragment pairing analysis compares contrastive pairings ( [1,4],[2,5],[3,6] ), all-fragments ( [1,2,3,4,5,6] ),
and alternative pairing methods ( [1,2],[3,4],[5,6] and [1,6],[2,5],[3,4] ) on IMDB-Clean-B with 40% symmetric noise when F=6 .
For feature extractor, all-fragments use a ResNet-34, while other pairing methods use ResNet-18 backbones.; Fragment pairing analysis compares contrastive pairings ( [1,4],[2,5],[3,6] ), all-fragments ( [1,2,3,4,5,6] ),
and alternative pairing methods ( [1,2],[3,4],[5,6] and [1,6],[2,5],[3,4] ) on IMDB-Clean-B with 40% symmetric noise when F=6 .
For feature extractor, all-fragments use a ResNet-34, while other pairing methods use ResNet-18 backbones.
Figure 17: Fragment pairing analysis compares contrastive pairings ( [1,3],[2,4] ), all-fragments ( [1,2,3,4] ),
and alternative pairing methods ( [1,2],[3,4] and [1,4],[2,3] ) on IMDB-Clean-B with 40% symmetric noise when F=4 .
For feature extractor, all-fragments use a ResNet-34, while other pairing methods use ResNet-18 backbones.; Fragment pairing analysis compares contrastive pairings ( [1,3],[2,4] ), all-fragments ( [1,2,3,4] ),
and alternative pairing methods ( [1,2],[3,4] and [1,4],[2,3] ) on IMDB-Clean-B with 40% symmetric noise when F=4 .
For feature extractor, all-fragments use a ResNet-34, while other pairing methods use ResNet-18 backbones.
Figure 18: Detailed Representation Depiction .
A detailed comparison of the effect of fragment pairings via t-SNE visualization of the penultimate features from the feature extractors.
The experiments are based on IMDB-Clean-B.; Detailed Representation Depiction .
A detailed comparison of the effect of fragment pairings via t-SNE visualization of the penultimate features from the feature extractors.
The experiments are based on IMDB-Clean-B.
Figure 19: Closed-set/open-set noise analysis displays the selection, ERR and MRAE
when closed-set or open-set noisy samples are injected into the clean dataset.
The experiments are based on IMDB-Clean-B.; Closed-set/open-set noise analysis displays the selection, ERR and MRAE
when closed-set or open-set noisy samples are injected into the clean dataset.
The experiments are based on IMDB-Clean-B.
Figure 20: Comparison of regularization methods .
Compared to other regularization methods, neighborhood jittering demonstrates superior performance in
(a) feature extractor test accuracy, (b) ERR, and (c) performance in regression.
The analysis is conducted on IMDB-Clean-B with symmetric 40% noise.; Comparison of regularization methods .
Compared to other regularization methods, neighborhood jittering demonstrates superior performance in
(a) feature extractor test accuracy, (b) ERR, and (c) performance in regression.
The analysis is conducted on IMDB-Clean-B with symmetric 40% noise.
Figure 21: Selection, ERR and MRAE comparison of ConFrag, Co-ConFrag and filtering/refurbishment baselines on IMDB-Clean-B
with symmetric 20%(a), 40%(b), 60%(c) and 80%(d) noise, repectively.; Selection, ERR and MRAE comparison of ConFrag, Co-ConFrag and filtering/refurbishment baselines on IMDB-Clean-B
with symmetric 20%(a), 40%(b), 60%(c) and 80%(d) noise, repectively.
Figure 22: Selection, ERR and MRAE comparison of ConFrag, Co-ConFrag and filtering/refurbishment baselines on IMDB-Clean-B
with Gaussian 30(a) and Gaussian 50(b) noise, repectively.; Selection, ERR and MRAE comparison of ConFrag, Co-ConFrag and filtering/refurbishment baselines on IMDB-Clean-B
with Gaussian 30(a) and Gaussian 50(b) noise, repectively.
Figure 23: Variance Analysis of three unique random seed experiments on IMDB-Clean-B.
The top two best-performing baselines under each noise type are reported.; Variance Analysis of three unique random seed experiments on IMDB-Clean-B.
The top two best-performing baselines under each noise type are reported.
Comparison of Mean Relative Absolute Error (%) over the noise-free trained model on the AFAD-B, IMDB-Clean-B, IMDB-WIKI-B, SHIFT15M-B, and MSD-B datasets.
Lower is better. A negative value indicates it performs even better than the noise-free model.
The results are the mean of three random seed experiments.
The best and the second best methods are respectively marked in red and blue .
CNLCU-S/H, Co-Selfie, and Co-ConFrag use dual networks to teach each other as done in Han et al. ( 2018 ) .
SPR (Wang et al., 2022 ) fails to run for SHIFT15M-B due to excessive memory usage.
Ablation of Mixture of Neighboring Fragments. MRAE on the IMDB-Clean-B dataset (lower is better).
Comparison between ConFrag and ConFrag-R: Mean Relative Absolute Error (%) to the noise-free trained model on the AFAD-B, IMDB-Clean-B, SHIFT15M-B, and MSD-B dataset.
Lower is better. The results are the mean of three random seed experiments.
Comparison of mean prediction depths of feature extractor learning tasks for all-frag, contrastive fragment pairing, and alternative fragmentation pairings when F=4 .
Comparison of mean prediction depths of feature extractor learning tasks for all-frag, contrastive fragment pairing, and alternative fragmentation pairings when F=6 .
Dataset Statistics on the six newly curated balanced datasets for regression: AFAD-B [Niu et al., 2016 ] , IMDB-Clean-B [Yiming et al., 2021 ] , IMDB-WIKI-B [Rothe et al., 2018 ] , UTKFace-B [Zhifei et al., 2017 ] , SHIFT15M-B [Kimura et al., 2021 ] , MSD-B [Bertin-Mahieux et al., 2011 ] .
Comparison of MRAE (%) on UTKFace-B datasets with symmetric noise.
Difference of selection rate and ERR between the samples at the boundary and center of fragments
Ablation and Combination Analysis. The values are mean relative absolute error to the noise-free trained model on the IMDB-Clean-B [Yiming et al., 2021 ] dataset, and lower values indicate better performances.
The results are the mean of three random seed experiments.
Discretized Baseline Analysis. Mean Relative Absolute Error to the noise-free model of discretized versions of strongly performing models on the IMDB-Clean-B [Yiming et al., 2021 ] dataset.
Lower is better.
Mean Relative Absolute Error (%) and its standard deviation to the noise-free trained model on the AFAD-B, IMDB-Clean-B and IMDB-WIKI-B datasets.
Lower is better.
A negative value indicates it performs even better than the noise-free model.
The results are the mean of three random seed experiments.
Number in parenthesis indicates standard deviation.
The best and the second best methods are respectively marked in red and blue .
CNLCU-S/H, Co-Selfie, and Co-ConFrag use dual networks to teach each other as done in Han et al. [ 2018 ] .
Mean Relative Absolute Error (%) and its standard deviation to the noise-free trained model on the SHIFT15M-B and MSD-B datasets.
Lower is better. A negative value indicates it performs even better than the noise-free model.
The results are the mean of three random seed experiments.
Number in parenthesis indicates standard deviation.
The best and the second best methods are respectively marked in red and blue .
CNLCU-S/H, Co-Selfie, and Co-ConFrag use dual networks to teach each other as done in Han et al. [ 2018 ] .
SPR [Wang et al., 2022 ] fails to run for SHIFT15M-B due to excessive memory consumption.
Standard Mean Absolute Error and its standard deviation to the noise-free trained model on the AFAD-B, IMDB-Clean-B and IMDB-WIKI-B datasets.
Lower is better.
The results are the mean of three random seed experiments.
Number in parenthesis indicates standard deviation.
The best and the second best methods are respectively marked in red and blue .
CNLCU-S/H, Co-Selfie, and Co-ConFrag use dual networks to teach each other as done in Han et al. [ 2018 ] .
Standard Mean Absolute Error and its standard deviation to the noise-free trained model on the SHIFT15M-B and MSD-B datasets.
Lower is better.
The results are the mean of three random seed experiments.
Number in parenthesis indicates standard deviation.
The best and the second best methods are respectively marked in red and blue .
CNLCU-S/H, Co-Selfie, and Co-ConFrag use dual networks to teach each other as done in Han et al. [ 2018 ] .
SPR [Wang et al., 2022 ] fails to run for SHIFT15M-B due to excessive memory consumption.