Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks
Authors: Hao Chen
Jindong Wang
Ankit Shah
Ran Tao
Hongxin Wei
Xing Xie
Masashi Sugiyama
Bhiksha Raj
Venue: ICLR 2024 spotlight (2024)
Keywords: Pre training
Noisy model learning
Label noise
Noise mitigation
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Pre-training on large-scale datasets and then fine-tuning on downstream tasks have become a standard practice in deep learning. However, pre-training data often contain label noise that may adversely affect the generalization of the model. This paper aims to understand the nature of noise in pre-training datasets and to mitigate its impact on downstream tasks. More specifically, through extensive experiments of supervised pre-training models on synthetic noisy ImageNet-1K and YFCC15M datasets, we demonstrate that while slight noise in pre-training can benefit in-domain (ID) transfer performance, where the training and testing data share the same distribution, it always deteriorates out-of-domain (OOD) performance, where training and testing data distribution are different. We empirically verify that the reason behind is noise in pre-training shapes the feature space differently. We then propose a light-weight black-box tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization on both ID and OOD tasks, considering one may not be able to fully fine-tune or even access the pre-trained models. We conduct practical experiments on popular vision and language models that are pre-trained on noisy data for evaluation of our approach. Our analysis and results show the importance of this interesting and novel research direction, which we term Noisy Model Learning.

Published as a conference paper at ICLR 2024
UNDERSTANDING AND MITIGATING THE LABEL NOISE IN PRE- TRAITRGNN ON DOWNSTREAM TASKS
Hao Chen 1 Jindong Wang"! Ankit Shah 1 Ran Tao', Hongxin Wei Xing Xie Masashi Sugiyama 1.5 Bhiksha Rajz.6
'Carnegie Mellon University. "Microsoft Research Asia. SusTech, 'RIKEN AIP, The University of Tokyo, "Mohamed bin Zayed University of A1
ABSTRACT
Pre-training on large-scale datasets and then fine-tuning on downstream tasks have become standard practice in deep learning. However, pre-training datasets, while inaccessible or too expensive to handle, often contain label noise that may ad- versely affect the generalization of the model and pose unexpected risks. This paper aims to understand the nature of noise in pre-training datasets and then mitigate its impact on downstream tasks. Specifically, through extensive exper- iments of supervised pre-training models on synthetic noisy ImageNet- IK and YFCC15M datasets, we demonstrate that while slight noise in pre-training benefit in-domain (ID) performance, where the training and testing data share the same distribution, it always deteriorates out-of-domain (OOD) performance, where training and testing distributions are different. We empirically ascertain that the reason behind is noise in pre-training shapes the feature space differently, We then propose a light-weight black-box tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization on both ID and OOD tasks, considering that one may not be able to access or fully fine-tune the pre-trained models. We conduct extensive experiments on popular vi- sion and language models including APIs that are supervised and self-superviser pre-trained on real data for evaluation. Our results show the importance of this novel and fundamental research direction, which we term Noisy Model Learning.
INTRODUCTION
The transfer learning paradigm of pre-training and fine-tuning (PT-FT) (Kornblith et al.. 2019) has become the de facto standard in today's 's deep learning research and application. Instead of training a neural network from scratch for each individual task, which can be time-consuming, resource- intensive, and less adaptable, the PT-FT paradigm first pre-trains a relatively larger and more general model with huge volumes of datasets, and then transfers this pre-trained model (or the foundation model (Bommasani et al. 202 to various downstream tasks (He et al., 2019; Radford al.. [021]] He et al.] 2022) Brown et al. 2020). For instance, ResNet (He et al. 2016a) and Vision Transform- (Dosovitskiy al. 2020) pre-trained on ImageNet (Russakovsky et al. 2015) and larger but potentially noisy datasets (Kolesnikov et al.) 2020; Xie et al.] 2020) Ridnik et al.) (022) have been widely adopted in computer vision. The PT-FT paradigm has also become predominant in natural language processing (Devlin et al.] 2018 g Liu et al. 2011 Radford et al., 2018 2019 Brown et al.)
2020 OpenAl 2023 Touvron et al.) 2023a) and multi-modality (Radford et al. 2021 Schuhmann
et al.
2022). where the pre-traiing is usually on large datasets scraped from the web.
The generalization and transferability of the pre-trained models are usually not guaranteed to be satisfying on downstream tasks, and the reason can lie in either the pre-training or the fine-tuning. Over the years, there have been tremendous efforts in improving the performance of fine-tuning in various practical downstream scenarios: out-of-distributin
generalization (Chen et al. [202] Kumar
cet al., 2022), semi-supervised learning (Sohn et al. 2020 Wang et al. 20222, imbalanced learning
(Zhang et al. 2023b Wang et al., 2023b), noisy label learning (Song et al.] 2022 Li et al. 2022
"haoc3 @ (ewndeww.cmu....... eww.edu, work done during a research intern at MSRA. Cooresponneece to: jindong. wang microsoft.com

---

Published as a conference paper at ICLR 2024
to name a few. While it is common belief that scaling up the size of the pre-training data can benefit the downstream performance (Kaplan et al. 2020), its distribution also plays an essential role (Entezari et al. 2023 Zhang et al. (023a). Recently, Nguyen et al. (2022) and Lee et al. (2022)
found that the quality of the pre-training data is more important for robust generalization compared to the quantity. The bias in pre-training data created by the collection (and annotation) process, e.g.-
corrupted, poisoned, and false information (Blodgett & O'Connor 2017] Chang et al. 2020). can
also impose malicious and unexpected influence downstream tasks (Bommasani et al.
2021
Take label noise as an example. Training CLIP (Radford et al. 2021) on LAION-2B (Schuhmann al.] 2022), which is a billion-scale uncurated image-tex text pair dataset, can just match the perfor- mance of training it on WIT-400M (Radford et al. (2021), which is heavily cleaned and processed by OpenAI. The label noise in large-scale datasets inevitably exists owing to the data collection process by human annotators and web crawlers. It thus can be difficult to avoid or eliminate in pre-training (Ridnik et al. 2021] Vasudevan et al. 2022 Schhhmaan et al.) 2022). fact, there are already numerous models pre-trained on large-scale noisy data and have been transferred on down- stream tasks, such as Noisy Student (Xie et al. (020), BiT (Kolesnikov et al. 2020), and Open CLIP (Cherti et al. 2023). Not to mention the enormous but noisy raw text (Yang al. 2019 Lee al., 2022) that has been utilized to pre-train pre-train language models such as BERT (Devlin et al., 2018) and GPT Radford et al. 2019 Brown al. 2020). As the pre-trained models and datasets have been growing significantly, it has become increasingly important and challenging to understand how the noise in pre-training data affects the performance of pre-trained models on downstream tasks.
This paper presents the first study on this un- explored problem, demystifying the label noise in pre-training data, understanding its effects on downstream tasks, and then mitigating such (malignant) effects. Notably, there are existing efforts under the name of "noisy label learn- ing" that train a robust model given noisy train- ing data (Ghosh et al., 2017: Li et al.) 2020) Northcutt et al. 2021). Our problem is inher- ently different since the noisy labels exist in the (usually black-box) pre-training data, and do not make noise assumptions on the down- stream data (while they be used together as in Section 4.3 more discussion is in Section (5). Due to the increasing size of pre-trained mod- els and datasets, it becomes notoriously diffi- cult to alter the pre-training process or fine-tune the entire models (black-box or cannot be up- dated due to the large parameter size and the constrained computation) Therefore, given pre-trained model, we should take special care of the fine-runing to overcome the influence of noise in pre-training on downstream tasks.
74
IS-K UP IS-K
ACCEECY
YBCC
STOCI-CC
64
28
378
375
0/
305
207
Note Ratio in Pastraining
Noise Malio Pro- Pre-triiting
(a) ID
(b) OOD
Figure 1: In-domain (ID) and out-of-domain (OOD) downstream performance when super- vised pre-training the model on synthetic noisy ImageNet-IK (IN-1K) and YFCC15M of various noise ratios. We compare linear probing (LP) and the proposed method on 14 ID and 4 OOD tasks. On ID, 5% noise in pre-training benefits the LP performance. Our method not only boosts the general performance but also rectifies the model pre-trained on clean data to be comparable to 5% noise. On OOD, noise in pre-training is detri- mental to robustness performance when conduct- ing LP. Our method improves the transferability on OOD tasks significantly compared to LP.
Our study aims to answer the following key questions: 1) Influence: Does the noise in pre- training data have an influence on downstream performance? 2) Analysis: Why does such influence happen? and 3) Miligation: How to mitigate such influence in light-weight and black-box fine-tuning process? We present an in-depth analysis to answer the above questions, based on the popular supervised pre-training paradigm ?
Influence: The label noise in pre-training data has both benevolent and malignant in-
fluence downstream tasks. In Sections 2.1 and 2.2
we conduct realistic experiments
'Llama TTuvron et al. 2023a model requires multiple V100 GPUs to fine-tune, which is not affordable most ordinary researchers; and proprictary models like ChatGPT cannot be locally fine-tuned.
"Supervised and self supervisdd learning are the most popular pre-training schemes. The former learns the mapping from inputs to labels (He al. 2016a Radford et al. 202T). while the latter does not rely on labels,
but predicts parts of the data itself (Devlin et al. 2018 He 2022
2

---

Published as a conference paper at ICLR 2024
with ResNet-50 models (He et al. 2016a) fully-supervised and contrastive pre- pre-trainied on syn- thetic noisy ImageNet-1K IK and YFCCI5M (Thomee et al. 2016) with various noisy ratios (0, 5%, 10%, 20%, 30%) and then study the generalization performance domain (ID) and out-of-domain (OOD) tasks. We observe that, on II) tasks, slight noise (up to 5% 10%) can benefit generalization performance. In contrast, even 5% noise can drastically deteriorate robustness and transfcrability on OOD tasks, as shown in Figure [ and Figure
the downstream in-
Analysis: The label noise in pre-training shapes the feature space significantly of the pre- trained model. In Section 2.3 we conduct empirical analysis from the singular value spectrum on the feature space of the pre-trained models. Noise in pre- pre-training results in the decreasing largest singular value and flatter singular value distribution with a higher dimension span in the feature space. An initial increase in the spanning dimension of the feature space is beneficial to the discriminability on ID tasks. Still, it then becomes detrimental with the further increase, indicating more feature capacities are learned fit to noise structure. The decrease in the dominant singular values leads to less transferability for OOD tasks (Chen et al., 2019), as shown in Figure S
Mitigation: We design a simple black-box fine-tuning algorithm to reshape the pre-trained feature space, reducing the influence of noisy pre-training data and boost the performance of downstream tasks. In Section 3 based on the analysis, we propose three regularization objectives on the singular value spectrum that help affine the feature space. We demonstrate the effectiveness of the proposed method on noisy ResNeet-50 models with extensive analysis, as shown in Figure In Section 4 we further validate our method on popular noisy pre-trained models (and APIs) and present superior generalization performance for both vision and language tasks.
Beyond our analysis, we view this research as novel and complementary topic to the classic noisy label learning setting, termed as Noisy Model Learning (NML). We think the value of this direction is even more significant in the of large foundation models (Bommasani al., (02TT) where the downstream users only have access to the model weights or APIs. It would be of particular interest to explore how eliminate the malignant influence of noise in pre-training on downstream tasks when adapting these models without full fine-tuning, since it may exist in broader applications such as the detection and segmentation in medical and autonomous driving. We hope that future research on this topic can facilitate a better understanding and application of large foundation models.
2 UNDERSTANDING THE LABEL NOISE IN PRE-TRAINED MODELS
In this section, we empirically and systemically investigate the effect of noisy labels in the supervised pre-training on the learned representations. We build our evaluation and analysis on the realistic motivating experiments of training ResNer-50 (He et al. (016a) on synthetic noisy ImageNet- (Russakovsky et al. 2015) and YFCC15M (a subset of YFCCTOOM (Thomee et al., 2016)).
2.1 EXPERIMENTS DESIGN
Noisy pre-training datasets. We assume that the supervised pre-training dataset consists of inputs x ~ X' and supervisions y ~ V. We define a clean dataset D = {(x;, y:) he of size N with accurate supervisions, where [N]] := {1...., .N.. We assume that y can exist in different formats in pre-training, e.g., an actual label for the input as in fully-supervised learning (Russakovsky al.) 2015 He et al. 2016al Ridnik et al. 2021) or a text description for an input image as in contrastive learning of CLIP (Thomee et al.] 20166] Radford al., 2021] Jia al.. 202T Changpinyo et al., 2021 Desai et al. Schuhmaan et al. 2021 2022). Due to the scale of data collection and the cost of data annotation, the pre-training dataset can usually contain noisy supervision ğ that does not
accurately match the corresponding x (Recht et al. 2019 Beyer et al. 2020. Northcutt et al. 2021 al. (022). We define such noisy pre-training
Yun al. 2021 Vasudevan et al.) [2022] Schuhmann
dataset as D {(x;; (:) [e]]) and the noise ratio y as the percentage of noisy supervision in D.
Pre-trained models. The pre-trained models serve a foundation for downstream tasks and usually can be abstracted as the stack of a feature extractor and a projection head. We define the feature extractor with learned parameters o mapping function f from the input space to feature space of dimension D: fo : X + F. The projection head go F + y is jointly pre-trained with the feature extractor, but not used when adapting fo on downstream tasks. We consider two types of supervised pre-training on images for this motivating example: fully supervised mrettrining where y is the actual class label and the projection head is linear classifier (He et al. 2016a)), and contrastive pre-
3

---

Published as a conference paper at ICLR 2024
31.0
01.0
360
61.
2011
62.0
260
A000000
67.0
10.0
62.0
53.0
250
90.1
10%
165
010
25%
25.0
261
200
385
270
30
300
2015
10775
175
3000
7555
107%
51A
Paronsiage of Samplo
Persisaage Sumples
Peoonlage of Samplor
Purassiag of Sumppis
(a) INIK. ID
(b) INIK, OOD
(c) YFCCI5M, ID
(d) YFCCI5M, OOD
Figure 2: Average ID and OOD evaluation results of ImageNet-IK (IN-IK) fully supervised pre- training ((a) and (b)) and YFCCISM CLIP pre-training ((c) and (d)) on downstream tasks with various percentages of data using ResNet-50. On ID evaluation, the transfer performance first in- creases as noise increases (to 5% or 10%) and then decreases with more noise. On OOD evaluation, the robustness performance constantly decreases once noise introduced in pre-training.
training with text supervision (CLIP) where y is the text and the projection is a non- non-linear function maps the image and text to common feature space (Radford et al. 2021 Cherti et al. 2023
In-domain (ID) and out-of-domain (OOD) evaluation. To investigate the effect of noisy super- vision comprehensively, we leverage both in-domain (ID) and out-of-domain (OOD) evaluation to assess the generalization capability of the pre-trained feature extractor (Djolonga et al. 2021 f that are obtained from the pre-training data of different noise ratios. To evaluate the pre-trained models on downstream dataset D' = {22, y: heeM and measure the quality of the learned representa- tion, we conduct linear probing (I.P) 0 where only a C-way linear classification head is re-trained on the downstream dataset and the feature extractor is frozen. The linear probing can be viewed as a simple black-box tuning method for pre-trained models that are typically large and difficult or unable to fully fine-tune. For ID evaluation, we assume the same marginal distribution over X' for both training and testing. In contrast, for OOD evaluation, we train the linear classifier on source distribution and evaluate it on (multiple) different target distributions (Kumar et al. 2022).
Experiment setup. We use ImageNet-11 KK (IN-1K) (Russakovsky et al. 2015) in fully supervised pre-training and YFCC15M (Thomee et al., 2016) in CLIP pre-training, with ResNet-50 (He et al.) 2016a). To introduce noisy supervision in the datasets, we uniformly flip the ground truth class label into the other classes in IN-IK and randomly swap the text description from another image-text pair in YFCC15M. We set the noise ratio 7 to (0%, 5%, 10%, 20%, 30%}. where 0% represents the clean dataset. For ID evaluation, we use 14 downstream datasets including CIFAR- 10/100000000 (Krizhevsky jet al., 2009).00).... Flowers102 (Nilsback & Zisserman, 2008), ) Food101 (Bossard et al., (2014). Oxford-
Pet (Parkhi al., 2012), StanfordCars (Krause al. 2013). FGVCAireraft (Maji al. 2013),
SVHN (Netzer et al.] 2011).
I), DTD (Cimpoi et al.] 2014). Caltech (Fei-Fei et al.] 20044). EuroSAT
(Helber al. 2018 2019). PatchCamelyon (Veeling et al., 2018 RESISC45 (Cheng et al. 2017)
and Rendered SST2 (Socher et al.) 2013), which cover various visual domains. For OOD evalu- ation, we use the "real", "sketch", "inpainting", and "clippart" of DomainNet (Peng et al. where we train on either "real" or "sketch" and evaluate on the others. For CLIP pre-trained mod-
2019
els, we additionally use 6 ImageNet variants (Recht et al.) 2019 Henddryck et al., [202aa] Wang let al. 2019a| Hendrycks et al. 2021b Shankar et al. |202 Barbu et al. 2019) for OOD evaluation while train on ImageNet- We report the LP performance for both ID and OOD evaluation using {10%, 25%, 50%, 75%, 100%} percentage of downstream datasets. The setup can be extended to other architectures and pre-training proxy objectives, as shown in Section [ Our pre-training pri- marily follows Wightman et al. (2021) and Cherti et al. (2023), with similar performance achieved as shown in Appendix|A. A. More details of the setup are included in Appendix A.2
2.2 RESULTS
In Figure 2 we plot the average accuracy for ID and OOD tasks of adapting the IN-1K fully super- vised and YFCC15M CLIP pre-trained ResNet-50 models. With the extensive motivating experi- ments, we empirically find two important and counter-intuitive observations from the results:
We always treat E [C] as an actual class label on downstream datascts. 'I Lincar probing is evaluation protocol accessing feature quality (He 2019 Liu
2021a
4

---

Published as a conference paper at ICLR 2024
15
00
10
000 Scerray
275 305
Ausurayy
207
50
1200
-8
65 56 Gs
25
501
20 3.0 3.1 322 3. 34 35
Singiiar Value Entroyy
Largnet Singrle: Velue Natoo
Singilar Value Entropy
Langeet Singilar Vilue Taato
(a) INIK, ID
(b) INIK, OOD
(c) YFCCI5M, ID
(d) YFCC15M, OOD
Figure 3: Feature SVD analysis. We compute the singular value entropy (SVE) for in-domain (ID) tasks and the largest singular value ratio (LSVR) for out-of-domain (OOD) tasks. Both metrics computed for ImageNet- 1K fully supervised pre-trained ((a) and (b)) and YFCC15M CLIP pre- trained ((c) and (d)) models. The SVE first slightly improves as the noise ratio increases to 5% or 10%, indicating better generalization. As the noise ratio increases, the SVE further improves, and the LSVR drops significantly, corresponding to worse generalization on ID and OOD tasks, more noise structure is learned. The dominant singular components become less transferable.
Proper noisy labels in pre-training (e.g., 5% or 10%) can benefit the performance on ID down- stream tasks, while more noise results in inferior results;
The robustness of transferability on OOD downstream tasks constantly deteriorates as the noise increases, even with the improvement in II) tasks on 5% noise.
While prior arts in noisy label learning mainly aim to correct/elimiatt the noise or perform robust learning against noise (Ghosh al.) 2017 Li show that the noise in pre-training can have both benevolent and malignant effects on downstream tasks. These observations raise a natural and fundamental question: where does the superior trans- ferability (with slight noise) and the inferior robustness stem from? We further analyze the feature space to understand the change the pre-trained feature extractor caused by noise.
2020 Liu et [022a Xue al. 2022).
2.3 FEATURE SPACE ANALYSIS
To understand the noise in pre-training data, we empirically analyze the singular value spectrum of the pre-trained feature space on downstream datasets, which is widely considered to be related to the generalization performance (Oymak et al. 2019 Chen et 2019 Xue
2022). More specifically, we perform singular value decomposition (SVD) on the features F E R D of pre- trained feature extractors on downstream dataset: F UΣV Appendix A.4 based on which we define two metries that can help understand the observations:
We plot the singular values in
Definition 2.1 (Singular Value Entropy). The singular value entropy (SVE) is defined as the entropy of normalized singular values. SVE measures the flatness the singular value distribution.
D
SVE = Σ O,
0, log
(I)
21=10 Σ/=1 oj
Larger SVE values indicate that the feature space captures more structure in the data and thus spans more dimensions either due more discriminated features are learned or memorization of the noise.
Definition 2.2 (Largest Singular Value Ratio). The largest singular value ratio (LSVR) defined as the logarithm of the ratio of the largest singular value d1 to the summation of all singular values:
LSVR - log 01
(2)
Σb
L.VR measures the variations in data captured by the singular vector corresponding to the largest singular value Ø1, which relates to the transferability of model (Chen et al. 2019).
denote as number samples downstream datasets and as feature dimension. 'We assume D < M (Kumar et al. 20022.. U and V denotes the left and right singular vector matrices, respectively, and E denoting the diagonal singular value matrix (a1
5

---

Published as a conference paper at ICLR 2024
Inaccessible Noisy
Dataset
Noisy Downstream Data
Downstream Data
Pre-truining Duta
Chihuahua
Flamingo
Cat
Bulldog
Model
(Pre-trained)
Pre-trained
No access
fe
fe
Transfer
fe
Learnable
Frozen
Lincar
Projection
Linear MLP
Black-box
Train
CE Truin from Scratch or Transfer from Pre-training Noisy Label Learning
CE
CCE
LcB ALNMTune
Lineur Probing
NMTune
Pre-training
or MLP Tuning
(Ours)
Noisy Model Learning
Figure 4: Illustration of noisy label learning (left) and the proposed Noisy Model Learning (right).
Noisy label learning mainly focuses on robustly training model from scratch fine-tuning a model from pre-training on noisy dataset. Noisy model learning focuses on robustly adapting the black- box noisy pre-trained models to downstream datasets with no assumption on the downstream dataset.
Analysis. We plot the SVE for ID tasks and LSVR for OOD tasks, as shown in Figure For ID tasks, as the noise ratio slightly increases, the learned representation usually presents slightly higher SVE, which indicates the pre-trained feature extractor captures more structure in data. Specifically, more capabilities of the feature space assigned to fit the noise in data, resulting in feature space spanning more dimensions, which provides better-initialized features on downstream tasks and facilitates generalization. Similar observations have also been found and explored in Wu et al. (202)). However, as the noise ratio further increases, the increased SVE indicates that more noisy data structure is captured and memorized, thus leading deteriorated generalization performance. When the labels in pre-training are random, the SVE of the feature extractor would further increase by memorizing all the noise but generalize on downstream tasks, similar to Zhang al. (20216). For OOD tasks, the robustness performance is negatively correlated with the LSVR. As the noise ratio increases, the LSVR consistently increases with the decreasing largest singular value. less transferable component is learned, thus resulting in worse generalization on unseen OOD tasks.
3 MITIGATING THE NOISE WITH REGULARIZATION ON SINGULAR VALUES
In this section, propose black-box fine-tuning method, which we call "Noisy Model Tuning" (NMTumee Figure in response to the noisy model learning setting. We demonstrate that NMTune boost the generalization downstream tasks and provide the analysis for the reasons behind.
3.1 METHOD
Per analysis above, noise in pre- pre-training shape the feature space differently from pre-training clean data, reducing the top dominant singular values with dampened transferability while increas- ing the spanning dimensions of the feature space fit noise structure. Since the large pre-trained models are usually difficult to fully fine-tune due to the enormous parameter size and limited compu- tation resources, we propose to alter the pre-trained feature space JF in a light- and black-box fashion. More specifically, we introduce a multi-layer perceptron (MLP) fw transforming the pre- trained features into new feature space Z. We propose three regularization terms on Z, to encourage the pre-trained knowledge to be maintained and improving SVE and LSVR of the new feature space.
Consistency regularization. To encourage the consistency of the pre-trained knowledge, we adopt a mean- mean-square square-error (MSE) loss between the normalized features F and Z:
F
Z
LMSE
(3)
|
|Z|2
This objective facilitates inheriting the pre-trained knowledge in the transformed features Z.

---

Published as a conference paper at ICLR 2024
1:00
0.30
Segnle Fitteg
6.50
0.22
SELF
B-ll SILP
1S-X MLL
B-lle Cran
TOLI Ceo
0.39
6.00
Larzed
0.22
630
10%
37A
Neti: Barko Pro training
Nose Patio Pro truniig
Nobe: Rarto in Fre traiiing
Noxe Ritlo in Pre trainin
(a) FI
(b) ID SVE
(c) OOD FI
(d) OOD LSVR
Figure 5: Evaluation of our method on ID and OOD downstream tasks, compared to MLP tuning and LP on ResNet-50 models pre-trained on ImageNet-IK (IN-1K) and YFCCI5M. (a) Average Fl score on ID tasks; (b) SVE on ID tasks; (c) Average FI score on OOD tasks; (d) LSVR on OOD tasks. Our method presents better SVE and LSVR on both ID and OOD tasks with better gener- alization performance. Our method also rectifies the malignant noise effect: the feature extractor pre-trained on clean data exhibits better performance than others noisy data ID tasks; and the performance gap between the clean one and the one with 5% noise gets smaller on OOD tasks.
Covariance regularization. We define the covariance loss to encourage the off-diagonal elements in the covariance matrix of the transformed feature CZZ) to be close 0:
Loov - = EiCZZ) where C(Z) M
(z (z+ 2)²
Zz-
i
Inspired by Zbontar et al. (2021) and Bardes et al. (2022). we use the covariance regularization term improve the SVE of feature space by preventing the different coordinates the features from encoding similar information. It also encourages more discriminative features to be learned.
Dominant singular value regularization. To help transferability, we use more specific regular- ization to improve the LSVR by directly maximizing the ratio of the largest singular value:
LSVD = 01
(5)
In summary, the total objective on a downstream task becomes:
L LCF ALNNTTmm LCF + X (LMSF Leov LSvD),
where Lee is the cross-entropy loss for downstream classification. We set À = 0.01 and use 2 layers MLP for all our experiments. Ablation study on MLP architecture and Appendix B.7
3.2 EVALUATION ON NOISY IMAGENET-IK. AND YFCCI5M
Here, evaluate the proposed NMTune the noisy models and analyze the reason for its effec- tiveness. We compare against solely training the ML.P without the regularization, termed as MLP tuning, show the effectiveness stems from the regularization rather than the extra parameters.
For ID tasks, we plot the average Fl score and SVE in Figures 5(a) and 5(b) respectively. The F1 score of linear probing (LP) on different pre-training noise ratios follows the same trend as the accuracy: it first increases as the noise ratio goes up to 5% and then decreases. While adding an MLP can improve the Fl score general, we find that it cannot mitigate the effect of noise, i.c., the clean pre-trained model underperforms the 5% noisy pre-trained models. Further introducing our method can rectify the effect of noise on tasks, leading the clean pre-trained feature extractor to achieve the best results. More interestingly, only adding a MLP to LP can result in a smaller SVE, especially on ImageNet-I corresponding to a much sparser feature structure. In contrast, our method provides larger and flatter SVE. It indicates the transformed feature space not only maintains the pre-trained knowledge but also spans more dimensions. For OOD tasks, the F1 score and LSVR are shown in Figure ((c))ana and 5(d) respectively. Similarly, better generalization performance deploying NMTune, compared to the MLP and LP. We also notice smaller performance gap for the clean pre-trained feature extractor and 5% noisy pre-trained,
observe significantly
7

---

Published as a conference paper at ICLR 2024
Table 1: Results on popular vision models that are pre-trained on noisy datasets. We use 14 in- domain (ID) and out-of-domain (OOD) tasks.
Table 2: Evaluation of our method on language models practice that are pre-trained on noisy datasets. We use GLUE for in-domain (ID) tasks and GLUE-X for out oft-oi-domain (OOD) tasks.
Pro-trained Model
Tuning
In-Domain
Damoan
Method
Acc.
Acc.
JFT30MM
LP
76.72
0,3815
44.13
0.3594
Pre-trained Model
Tuning Method
In-Domain
Out-of-Domain
Somi-Supervised
MLP
76.87
0.3833
45.95
0.3624
FaficientNat-33
Ours
77.63
0.3874
46.84
0.3684
LP
69.44
50.65
IetgeNet-21K
LP
77.51
0.3718
40.82
0.3062
BERT-L
MLP
69.78
50.62
Fully Supervised
MLP
77.58
0,3726
41.73
0.3053
Ours
70.26
51.63
Res-ett2-152x22
0.3862
42.42
I.P
69.75
44.55
ImageNet-21I
LP
81.91
0,4092
50.88
0.3838
RoBERTa-L
MLP
70.27
45.22
Fully Supervised
MLP
82.51
0.4128
51.21
0.3811
Ours
70.97
47.01
Swin-L
Ours
84.16
0.4177
52.35
0,3901
LP
58.67
36.68
Laion-2B
LP
88.86
0,4432
66.86
0.4253
GPT-2
MLP
58.44
37.24
CLIP
MLP
88.53
0,4417
68,43
0.4304
Ours
59.34
39.07
CowNNext-L
Ours
89.48
0.4457
70.30
0.4367
LP
56.96
44.06
Laion 2B
LP
86.85
0,4328
66.89
0,4208
CLIP
MLP
87.23
0.4375
69.50
0.4221
text-ada 002
MLP
63.89
$1.30
VITLL
Ours
88.57
0,4414
70.47
0,4246
Ours
65,99
53.48
especially on YFCCI5M. On LSVR, MLP tuning usually imposes larger LSVR compared to LP, presenting smaller dominant singular values. Considering MIP tuning also presents smaller SVE, its resulting feature space is expected to present a more long-tailed spectrum than the original feature space. Maximizing the dominant singular values results in better transferability for OOD tasks.
4 EXPERIMENTS
We further validate NMTune on practical large-scale vision and language models that are pre- trained on noisy data, and discuss the noisy label learning and running time analysis in this section.
4.1 VISION MODELS AND DATASETS
Setup. For vision models, we use ResNet152 (He et al. 2016a) with dimensions widened by factor of two (eesNet152x2)) fully supervised pre-trained on ImageNet-21K (Kolesnikov al.) 20200. Swin-L. (Liu et al.) (021c) fully supervised pre-trained on ImageNet-21K,, HffhcientNet-333 semi-
supervised pre-trained on noisy JFT-300M (Hinton et al.] [015 Chollet (017) and ImageNet-
and VIT-L (Dosovitskiy et al.
2020) and ConvNext-L (Liu ct al. (022cc contrastive pre-trained on
noisy Laion-2B (Cherti et al. 2023). All pre-trained models are adapted from TIMM (Wightman)
2019). We evaluate the models on the 14 downstream ID and 4 OOD vision datasets as in Section The details of
are shown in Appendix B.T due to space limit.
Results. We present the average accuracy and F1 score across different datasets with three runs on vision models in Table [ Our method improves the quality of the noisy pre-trained features with better accuracy and F1 score both IID and OOD vision tasks. A large margin downstream task across different pre-training architectures and datasets is present by NMTuuee, demonstrating better feature learned. Noteworthy is that, although the MLP tuning also improves the performance in general, its performance gain is much smaller compared to our method, showing the effective- ness of the proposed regularization terms on mitigating the malicious effect of noise and improving generalization. More detailed results with error bars for each dataset are shown in Appendix B.2
4.2 LANGUAGE MODELS AND DATASETS
Setup. We evaluate BERT-L DDellin et 2018 RoBERTa-L Liu al. 2019). and GPT-2 (Rad- Ford et al. 2019) on the GLUE (Wang et al. 2018) and GLUE-X ( (Yang et al. 2023) benchmarks for ID and OOD performance.. BERTH L and RoBERTa-L are pre- trained on the combination of the Books Corpus data (Zhu et al. 2015) and English Wikipedia with uncompressed raw text. It is found that the raw pre-training data of BERT can be reduced from 16GB to 12GB with data cleaning Yang et al. 2019). GPT-2 is pre-trained on WebText (Radford et al. 2019), scraped web dataset from Common Crawl that contains low-quality raw texts. We also leverage OpenAl's I's API service "text-ada-002" Details of the parrr--rrameterss and evaluation metrics are in Appendix B.3
We cannot use larger and more recent language models such as I.IaMA (Touvron al. 2023a), since they unable fit in single V100 GPU and we unsure whether GLUE is in their training data.
8

---

Published as a conference paper at ICLR 2024
Results. In Table 2 NMTune consistently achieves the best generalization performance. It presents superior performance gain, especially on OOD tasks of GLUE-X. On the "text-ada-002" model with only API access, it also outperforms LP significantly, demonstrating the necessity of mitigating the effect of noise for better generalization. Interestingly, on the ID tasks of GLUE, we also observe a smaller gap of MLP tuning method to L.P even with more parameters, showing that the MLP alone may not mitigate the influence of noisy data in pre-training Full results are in Appendix B.4
4.3 DISCUSSION
Noisy model learning with noisy label learning. We explore another setting, where these two paradigms occur together with both the pre-training and fine-tuning containing label noise, as shown in Appendix B.5 Our exploration in synthetic noisy CIFAR- 10/100 presents similar observations of LP and NMtune as in clean downstream datasets, and they can work closely to achieve better performance downstream datasets with slight noise. Running time analysis. We present the average GPU hours of NMTune, MLP tuning, and LP in Appendix [B6] showing that it introduces negligible computation. The ablation study and architecture of MLP are shown in Appendix [B.7] Finally, our results may not be comparable to white-box full fine-tuning results, which is acceptable pursue
since we perform black-box tuning and the feature extractors are frozen. Our goal is the best but to offer insights and discuss new research possibilities in the era of foundation models.
RELATED WORK
Noisy label learning. Prior arts on noisy label learning mainly focus how train robust models or how to adapt clean pre-trained models on noisy (downstream) datasets from scratch, including robust loss functions (Ghoss et al.
2017
Zhang & Sabuncu 2018 Wang et al. 2019b Ma al.,
2020), noise estimation (Xiao et al.
2015
Goldberger & ReuneReuven 2016 Liu et al. 2020 North
cull et al. 2021 Li et al. 20211. ) and noise correction (Han et al. 2018 Li Li al. 2020 Zhang al.
c02 Liu et al. 2022a Kim et al. 2021
Chen et al. 2023p. Perhaps more close to our work is the
line of understanding noisy label learning. Ghosh et al. (2017) looked at theoretical conditions for
a loss function to be noise-tolerant. CIFAR-N (Wei et al. (0226) was built to understand the real- world instance-dependent label noise. [Cheng et al. (2023) proposed to mitigate the memorization of noise labels by analyzing the regularization between representations. Wen et al. (2022) provably verified the failure of benign overfitting with label noise. Xue et al. (2022) investigated the robust- ness of contrastive pre-training with noisy labels downstream tasks. Our work differs from the noisy label learning paradigm by focusing on the effect of pre-training noise on downstream.
Pre-training and fine-tuning, Pre-training and fine-tuning is the dominant transfer learning paradigm that allows a pre-trained model to adapt to a new, but similar, dataset. Many techniques proposed for better transfer performance (Cheng et af.) 2023), unlabeled data (Sohn et al. imbalanced data (Kang et al., 2019 Wang et al. 2023c). and noisy data (Wei et al. [202a Xue iet al. 2022). There are also much relevant work studying and processing the pre-training data for better transfer performance by diversity trade-off (Kaplan al. 2020 Zhang et al. (2023)). data
the new dataset when it contains distribution shift
2020 Zhang et al., 2021a Wang et al. 2023a)
selection (Entezari et al. 2023). quality-guantity trade-off (Magar & Schwartz 2022 Nguyen et al., methods Tsai et al.] 2020 Kumar et al. [2022; Wortsman et al.] 2022 Goyal et al. 2023] Xu et al.) 2003)) Parameter-efficient transfer learning (He et al. 2021: Oh et al.] (023) is lightweight paradigms by adding adapters (Houlsby et al. 2019), low rank approximation (Hu et al.) (02T), or prompt tuning (0226)]021bb, 2021b). However, they all assume the availability of pre-trained models while we deal with bla models. They also do not consider the noise in pre-training data.
2022 Lee et al., 2022] Carlini et al. [022] Gadre et al. (023). and specified
(Liu et al. (022b)
6 CONCLUSION
We presented Noisy Model Learning, a new research direction for understanding and mitigating the effect of label noise in pre-training on downstream tasks. Extensive experiments demonstrate that proper noise in pre-training can benefit in-domain tasks and hurt out-of-domain tasks. We then proposed NMTune to mitigate the malignant effect of noise and improve the generalization performance of various noisy pre-trained models and APIs. While being the first study in this area, the explored models are still relatively small-scale in terms of pre- training, and we only use ResNet- 50 for analytical experiments, due to the limited computing resources. We hope our work can inspire more researchers this important and challenging topic in more practical settings.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENT AND DISCLAIMER
Masashi Sugiyama was supported by the Institute for Al and Beyond, UTokyo. In this paper, we gen- erated some noisy pre-training images using ImageNet- IK to thoroughly study the noisy pre-training data. Such noisy data indeed could have malignant influence downstream tasks, according to our findings. The only purpose conducting this research to study the noisy pre-training data, but not to claim their instability in real applications. Additionally, all the generated noisy images and our pre-tre vre-trained models based on these data are for research purpose only, and will be released per request.
REFERENCES
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: large-scale bias-controlled dataset for pushing the lim- its of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, 2019.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variaace-invariacce-ooovaaac regularization for self-suppe supe--uue learning. In International Conference Learning Representations (ICLR), 2022.
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.077 0022000067799, 2020.
Su Lin Blodgett and Brendan O'Connor. Racial disparity natural language processing: case study of social media african-amerrcan english. arXiv preprint orXiv:1707.00060 2017.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportu- nities and risks of foundation models. arXiv preprint 10X.::200..07558, 2021.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative com- ponents with random forests. In European Conference Computer Vision, 2014.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher Manning. A large anno- tated corpus for learning natural language inference. arXiv preprint /500 053266.0036,,.. 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 202..
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2022.
Hongyan Chang, Ta Duy Nguyen, Sasi Kumar Murakonda, Ehsan Kazemi, and Reza Shokri. On adversarial bias and the robustness of fair machine learning. arXiv preprint arXiv:2006.0888 086669 2020.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Hao Chen, Ankit Shah, Jindong Wang, Ran Tao, Yidong Wang, Xing Xie, Masashi Sugiyama, Rita Singh, and Bhiksha Raj. Imprecise label learning: A unified framework for learning with various imprecise label configurations. arXiv preprint arXiv:2305. 12715, 2023.
Mayee Chen, Karan Goel, Nimit S Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher Ré. Mandoline: Model evaluation under distribution shift. In International conference on machine learning, pp. 1617-1629. PMLR, 2021.
10

---

Published as a conference paper at ICLR 2024
Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability discrim- inability: Batch spectral penalization for adversarial domain adaptation. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1081-1090. PMLR, 09-15 Jun 2019.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the art. Proceedings of the IEEE, 105(10):1865 -18188666--88--,----888 Oct 2017. ISSN 1558- 2256.
Hao Cheng, Zhaowei Zhu, Xing Sun, and Yang Liu. Mitigating memorization of noisy labels via regularization between representations. In International Conference on Learning Representations (ICLR), 2023.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-22882 2818-2829, 2023.
François Chollet. Xception:
Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Patlern Recognition (CVPR), 2014.
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. RedCaps: Web-curated image-text
data created by the people, for the people. In NeurIPS Datasets and Benchmarks, 2021.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' Amour, Dan Moldovan, Sylvain Gelly, Neil Houlsby, Xiaohua Zhai, and Mario Lucie. On robustness and transferability of convo- lutional neural networks. 2021 IEEE/CVF Conference on Computer Vision and Puttern Recogni- tion (CVPR), Jun 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929, 2020.
Rahim Entezari, Mitchell Wortsman, Olga Saukh, M Moein Shariatnia, Hanie Sedghi, and Ludwig Schmidt. The role of pre-training data in transfer learning. arXiv preprint a3Xiv2230.11300. 2023.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Pattern Recognition Workshop, 2004.
Samir Yitzhak Gadre. Gabriel Ilharco, Alex Fang. Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang. Eyal Orgad, Rahim En- tezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexan- der J. Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alexandros G. Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Dat- acomp: In search of the next generation of multimodal datasets. ArXiv, 2023.
Aritra Ghosh, Himanshu Kumar, and P. Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2017.
Jacob Goldberger and Ehud Ben-Reuven. Training deep ceural-networke using a noise adaptation layer. In International Conference Learning Representations (ICI.), 2016.

---

Published as a conference paper at ICLR 2024
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19338-19343 2023.
Bo Han. Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Wai-Hung Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in Neural Information Processing Systems (NeurIPS), 2018.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. Towards unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.0436 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog-
nition. CVPR, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual net- works. Computer Vision-ECCY ECCV 2016: 14th European Conference, Amsterdam, The Nether- Springer. 2016b.
lands, October 11-14, 2016, Proceedings. Part IV 14, pp.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint
2019.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked autoencoders are scalable vision learners. 2022 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2022.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. In IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, pp. 204-207. IEEE, 2018.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces robustness: A critical analysis of out-of-distribution generalization. 2021 IEEE/CVF International Conference Computer Vision (ICCV), Oct 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), Jun 2021b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253: 2015.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. farameter-efficien transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.06685, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- languaee representation learning with noisy text supervision. In International conference machine learning, pp. 4904 4916. PMLR, 2021.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference Learning Representations, 2019.
12

---

Published as a conference paper at ICLR 2024
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 2020.
Tushar Khot, Ashish Sabharwal, and Peter Clark. Scitail: A textual entailment dataset from science question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Tachyeon Kim, Jongwoo Ko, JinHwan Choi, Se-Young Yun, al. Fine samples for learning with noisy labels. Advances in Neural Information Processing Systems, 34:24137-24149, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: w4tp. 69800, 2014.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Computer Vision- FCCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pp. 491-507. Springer, 2020.
Simon Kornblith, Jonathon Shlens, and Quoc V. Le. Do better imagenet models transfer better? In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Jun 2019.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine- the-ggai categorization. In 2013 IEEE International Conference Computer Vision Workshops, pp. 554- 561, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ananya Kumar, Aditi Raghunathan, Robbie Jones. Tengyu Ma. and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202. 10054, 2022.
Katherine Lee. Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL., 2022.
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In International Conference on Learning Representations (ICLR), 2020.
Shikun Li, Xiaobo Xia, Shiming Ge, and Tongliang Liu. Selective-superviste contrastive learning with noisy labels. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 316-325, 2022.
X. Li, T. Liu, B. Han, G. Niu, and M. Sugiyama. Provably end-to-end label-noise learning without anchor points. In Proceedings of 38th International Conference on Machine Learning (ICML2021), 1), pp. 6403-6413, 2021.
Hong Liu, Jeff Z HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In International Conference on Learning Representations, 2021a.
Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Simandez--rrande Early-leariin regularization prevents memorization of noisy labels. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by over- parameterization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the International Conference on Machine Learning (ICML), volume 162, pp. 14153-14172. PMLR, 17-23 Jul 2022a.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du. Zhilin Yang, and Jie Tang. P- tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602 2021b.
13

---

Published as a conference paper at ICLR 2024
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 61-68, 2022b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprini arXiv: 1692, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. 2021 IEEE/CVF In- lernational Conference on Computer Vision (ICCV), Oct 2021e.
Zhuang Liu, Hanzi Mao, Chao Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. 2022 IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), Jun 2022c.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Monazam Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language lechnologies, pp. 142-150, 2011.
Inbal Magar and Roy Schwartz. Data contamination: From memorization to exploitation. In Pro- ceedings of the 60th Anmual Meeting of the Association,fo for Computational Linguistics ( (Volume 2: Short Papers), pp. 157-165, 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 2013.
Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 3428-3448, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 2.118653/v1/P99 1334. URL httts::/accaateee
orq/P19-1334
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Unsupervised Feature Learning 2011, 2011.
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. Advances in Neural Information Processing Systems, 35:21455-2144 2022.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification large number of classes. In 2008 Sixth Indian conference on computer vision, graphics image processing, pp- 722-729. IEEE, 2008
Curtis G. Northeutt, Lu Jiang, and Isaac L. Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research (JAIR), 70:1373-1411, 2021.
Changdae Oh, Hyeji Hwang, Hee-young Lee, YongTack Lim, Geunyoung Jung, Jiyoung Jung, Hosik Choi, and Kyungwoo Song. Blackvip: Black-hox visual prompting for robust transfer learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pp. 24224-24235, 2023.
OpenAl. Gpt-4 technical report, 2023.
Samet Oymak, Zalan Fabian, Mingehen Li, and Mahdi Soltanolkotabi. Generalization guaran- tees for neural networks via hamnessing the low-rank structure of the jacobian. arXiv preprint arXiv:/906.00 05392, 2019.
14

---

Published as a conference paper at ICLR 2024
O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference Computer Vision and Pattern Recognition, 2012.
Xingchao Peng. Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1406-1415, 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, lly Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML. 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International conference on machine learning, pp. 5389-5400. PMLR, 2019.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zannik-Manor. Imagenet-21K pretraining for the masses, In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 1152))211-225, 2015.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open dataset of 1.02/ clip-filtered 400 million image-text pairs. arXiv preprint a 1.00:///1/001//11 14, 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Vaishaal Shankar, Achal Dave, Rebecca Roelofs, Deva Ramanan, Benjamin Recht, and Ludwig Schmidt. Do image classifiers generalize across time? 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2021.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics.
sentiment
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances Neural Information Processing Systems (NeurIPS), 33, 2020.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-19, 2022. ISSN 2162-2388.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105-6114 14. PMLR, 2019.
15

---

Published as a conference paper at ICLR 2024
Bart Thomee, David A. Shamma, Gerald Friedland. Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcm100m. Communications of the ACM, 59(2):64-73, Jan 2016. ISSN 1557-7317.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023b.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. Newsqa: A machine comprehension dataset. arXiv preprint arXiv: 1.100980, 2016.
Yun- Yun Tsai, Pin- Yu Chen, and Tsung- Yi Ho. Transfer learning without knowing: Reprogramming black-box machine learning models with scarce data and limited resources. In International Conference on Machine Learning, pp. 9614-9624. PMLR, 2020.
Vijay Vasudevan, Benjamin Caine, Ruphael Gontijo-Lopes, Sara Fridovich- Keil, and Rebecca Roelofs. When does dough become a bagel? analyzing the remaining mistakes on imagenet. arXiv preprint arXiv:2205.045555 2022.
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling, Rotation equiv- ariant CNNs for digital pathology. June 2018.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2018
Haohan Wang, Songwei Ge, Eric P. Xing, and Zachary C. Lipton. Learning robust global repre- sentations by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), 2019a.
Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang. Zhi Zhou, Lan-Zhe Guo, Heli Qi, Zhen Wu, Yu-Feng Li, Satoshi Nakamura, Wei Ye, Marios Savvides, Bhiksha Raj, Takahiro Shinozaki, Bernt Schiele, Jindong Wang, Xing Xie, and Yue Zhang. Usb: A unified semi-supervised learning benchmark. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Sav- vides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie. Freematch: Self-adaptive thresholding for semi-supervised learning. In International Conference on Learning Representa- tions (ICLR), 2023a.
Yidong Wang, Zhuohao Yu, Jindong Wang, Qiang Heng, Hao Chen, Wei Ye, Rui Xie, Xing Xie, and Shikun Zhang. Exploring vision-language models for imbalanced learning. International Journal of Computer Vision (JJC),, 2023b.
Yidong Wang, Bowen Zhang, Wenxin Hou, Zhen Wu, Jindong Wang, and Takahiro Shinozaki. Margin calibration for long-tailed visual recognition. In Asian Conference on Machine Learning, pp. 1101-1116. PMLR, 2023c.
Yisen Wang. Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV), pp. 322-330, 2019b.
Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu. To smooth or not? when label smoothing meets noisy labels. In International Conference on Machine Learn ing (ICMI), 2022a.
16

---

Published as a conference paper at ICLR 2024
Jiaheng Wei, Zhaowei Zhu, Hao Cheng. Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations (ICLR), 2022b.
Kaiyue Wen, Jiaye Teng, and Jingzhao Zhang. Benign overfitting in classification: Provably counter label noise with larger models. In The Eleventh International Conference on Learning Represen- fations, 2022.
Ross Wightman. Pytorch image models. pytorch- www-iimaaaeem-e 2019.
htttps//ggthubb.com/riiiiat
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure timm. arXiv preprint arXiv:2110.00476 2021.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for 2017.
sentence understanding through inference. arXiv preprint
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings ofthe the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 7959-7971, 2022.
Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. Noisytune: little noise can help you finetune pretrained language models better. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), 2022.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2691-2699, 2015.
Qizhe Xie, Minh-Tham Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. 2020 IEÉE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2020.
Shoukai Xu, Jiangchao Yao, Ran Luo, Shuhai Zhang, Zihao Lian, Mingkui Tan, and Yaowei Wang. Towards efficient task-driven model reprogramming with foundation models. arXiv preprint arXiv:2304.02263, 2023.
Yihao Xue, Kyle Whitecross, and Baharan Mirzasoleiman. Investigating why contrastive learning benefits robustness against label noise. In International Conference Machine Learning, pp- 24851-24871. PMLR, 2022.
Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang. Glue-x: Evaluating natural language understanding models from an out-of- distribution generalization perspective. In Findings of ACI., 2023.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. XInet: Generalized autoregressive pretraining for language understanding. Advances neural information processing systems, 32, 2019.
Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun. Re-labeling imagenet: from single labti,, from global localized labels. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2021.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang. Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi- sumi-iueervised learning with curriculum pseudo la- beling. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, 2021a.
17

---

Published as a conference paper at ICLR 2024
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 2021b.
Jieyu Zhang, Bohan Wang, Zhengyu Hu, Pang Wei Koh, and Alexander Ratner. On the trade-off of intra-/inter-clas diversity for supervised pre-training. arXiv preprint 1xXiv:2305.12222 2023a.
Li Zhang, Steven R Wilson, and Rada Mihalcea. Multi-label transfer learning for multi-relational semantic similarity. arXiv preprint 1005. 2018.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023b.
Yivan Zhang, Gang Niu, and Masashi Sugiyama. Learning noise transition matrix from only noisy labels via total variation regularization. In International Conference on Machine Learning, pp- 12501-12512. PMLR, 2021c.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. 2015 IEEE International Conference Computer Vision (ICCV), Dec 2015.
18

---

Published as a conference paper at ICLR 2024
Appendix
CONTENTS
Understanding the Noisy Labels in Pre-training Data
19
A.I
Pre-training Datasets and Hyper-parametees
19
A.2 Downstream Vision Datasets and Hyper-parameters
19
A.3 Detailed ID and OOD Linear Probing Results
20
A.4
Detailed ID and OOD Singular Value Spectrum
Experiments
22
B.1
Detailed Setup for Vision Models Experiments
22
B.2
Detailed Results for Vision Models Experiments
24
B.3
Detailed Setup for Language Models Experiments
24
B.4
Detailed Results for Language Models Experiments
................................
27
B.5
Transferring on Noisy Downstream Datasets
B.6
Runtime Analysis
29
7
Ablation Study
30
More Discussions
30
Limitations
31
Potential Failure
31
A UNDERSTANDING THE NOISY LABELS IN PRE- TRE-TRAINING DATA
We provide additional experiment details for the motivating example of ResNet-50 in this section. We also present the detailed results on each downstream dataset for noisy pre-trained models on both ImageNet- 1K and YFCC15M. The SVD plots on each dataset are also shown here.
A.1 PRE-TRAINIGG DATASETS AND
For analysis in Section conduct pre-training of ResNet-50 on ImageNet- IK and YFCCI5M.
For ImageNet- pre-training, we follow the training recipe in Wightman et al. (202T). To introduce noise in ImageNet-IK, use function cleanlab (Northcutt et al. 2021) 1 to introduce symmetric noise in each class. For YFCCISM CLIP pre-training, we follow the training recipe in [Cherti et al. 2023). To introduce noise in YFCC15M, sampled image-text pairs until the noise ratio is achieved. We show the validation accuracy on ImageNet-IK IK of the noisy ResNet-50 models pre-trained on ImageNet-IK and zero-shot accuracy on ImageNet-IK IK of the noisy ResNet-50 models pre-trained on YFCC15M in Table 3 The results
swap the text description between
randomly
show that our pre- pre-training achieves the state-of- of-to---thheeee art results (Wightman al. 202
Cherti al.
2023). as a basis for our further analysis.
A.2 DOWNSTREAM VISION DATASETS AND HYPER-PARAMETERS
We present the details of the in-domain (ID) vision datasets in Table and out-of-domain vision datasets Table 5 For ID, we conduct training the training and test the validation set of the downstream dataset. For OOD on DomainNet (Peng et al. 2019), we conduct training on the training set of DomainNet Real DomainNet Sketch, and test all the other three NetainNet
19

---

Published as a conference paper at ICLR 2024
Table 3: ImageNet-IK validation and zero-shot accuracy of ImageNet-1K pre-trained and YFCC15M CLIP pre-trained noisy ResNet-50 models.
ImageNett- Pre-train Validation Accuracy
YFCCI5M CLIP Pre-train Zero-shot Accuracy
Noise Ratio
0%
79.96
32.64
5%
79.18
30.86
10%
78.61
29,54
20%
76.27
27.72
30%
73.11
26.53
datasets not used in training. For OOD on ImageNet [Russakovsky et al. 2015 we conduct training on ImageNet training split and test on its variants.
To transfer a pre-trained model, we use linear probing (LP) for analysis as shown in Section 2 We train the linear classifier for 30 epochs on each downstream dataset, using AdamW (Kingma & Ba
2014) optimizer with cosine scheduler. We do not use weight decay for linear probing and set the learning rate to 0.1 for all tasks.
Table Details of the in-domain (ID) vision datasets used to evaluate ID transfer performance of vision models.
Dataset
Classes
Train Size
Test Size
Evaluation Metric
CIFAR-10 Krizhevsky et al.
2009
10
50,000
10,000
accuracy
CIFAR 100 (Krizhevsky et 2009
100
50,000
10,000
accuracy
Flowers102 (Nilsback & Zisserman
2008
102
2,040
6,149
mean class
Food101 Fei-Fee et al. 2004
101
75,750
25,250
accuracy
OxfordPet (Prrkhi et al. 2012
37
3,680
3,669
mean per class
StanfordCars (Kauss
2013
196
8,144
8,041
accuracy
FGVCAircraft (Maji
2013
102
6,667
3,333
mean per class
SVHN (Netzer ei al.
73.257
26.032
accuracy
DTD (Cimpor el al. 2014
47
1,880
1,880
accuracy
Caltech (Fer- Fei ct al. 2004
102
3,060
6,084
mean per class accuracy
EuroSAT (Helber et al. 2019
10
21,600
5,400
PatchCamelyon (Veeling ct al.
2018
10
73,257
26,032
accuracy
RESISC45 (Cheng al. 2017
45
25,200
6,300
accuracy
Rendered SST2 (Schher et al.
2013
2
6,920
1,821
accuracy
Table 5: Details of the 4 out-of-domain (OOD) DomainNet datasets and 6 out-of-domain (OOD)
ImageNet variants used evaluate OOD transfer performance of vision models.
Dataset
Classes
Train Size
Test Size
Evaluation Metric
DomainNet Sketch (Peng al. 2019]
345
48,212
20,916
accuracy
DomainNet Real (Peng al. 2019
345
120,906
52,041
accuracy
DomainNet Painting [Peng ct al. [2019
345
21,850
accuracy
DomainNet Clipart (Peng ct al 2019
345
14,604
accuracy
ImageNet- V2 (Recht |l. 2019
1,000
10,000
accuracy
ImageNet-R [Hendrycks et
2021
200
30,000
accuracy
ImageNet-Skeich [Wang ct al.
2019a
1,000
50,889
accuracy
ImageNet-A A (Hendrycks et al
2021a
200
7,500
accuracy
ImageNet- (Shnnar ct al.
2021
1,000
accuracy
ObjectNet (Barbu et al. 2019
112
18,574
accuracy
A.3 DETAILED ID AND OOD LINEAR PROBING RESULTS
We present the detailed ID and OOD linear probing results we analyzed in Section here.
The ImageNet- IK and YFCC15M pre-trained ID results in Figure X and Figure respectively. On all the datasets, we can observe that the 5% or 10% noise pre-traned models outperform the clean pre-trained models, matter which pre-training dataset and method is used.
20

---

Published as a conference paper at ICLR 2024
The OOD results are in Figure 7 and Figure respectively. On the validation split of the training dataset (ID), the trend follows the ID observations, where 5% noisy pre-trained model is better. However, on the OOD datasets, the model performance deteriorates noise increases.
2713
Figure 6: ImageNet- IK pre-trained ResNet-50 in-domain (ID) evaluation results
Figure 7: ImageNet-IK pre-trained ResNet-50 out-of-domain (OOD) evaluation results
ISPARTS
ITTRII
Eeinttaer Sanike
Figure 8: YFCCI5M pre-trained ResNet-50 in-domain (ID) evaluation results
A.4 DETAILED ID AND OOD SINGULAR VALUE SPECTRUM
We plot the singular value spectrum for ID datasets and OOD datasets of the noisy ResNet-50 models. To better visualize the spectrum, we split the singular values into three groups: the top 20, 20-500, and the remaining.
The singular value spectrum of the ID datasets is shown in Figure II and Figure 1 respectively. From 20-500 singular values visualization, general have larger singular values in this range, corresponding to a feature space that spans more of its coordinates. We summarize this visualization the SVE introduced Section Here, we provide
observe that the noisy pre-trained models in
21

---

69 7.0 7.1 Published as a conference paper at ICLR 2024
Figure 9: YFCCI5M pre-trained ResNet-50 out-of-domain (OOD) evaluation results
056
80
10%
Accurayy
Accuroryy
20%
30%
60
30
20
30
5.0
5.2
5.4
5.6
0.5
6.6 6.7
688
Singular Value Entropy
Singular Value Entropy
(a) INIK, ID
(b) YFCCI5M, ID
Figure 10: Zoom-in visualization of feature SVE analysis for in-domain (ID) tasks.
more explanation how to make Figure First, each color and marker represents different pre- training noise ratio. We plot the average accuracy of different percentage of downstream datasets and the SVD (or LSVR) of the downstream test data for each downstream task. Thus each points corresponds to a downstream task. The results of different pre-training noise ratio for each task are thus clustered together.
We also provide zoom-in version for Figure ((a) and Figure 3(b) for better visualization, as shown Figure I0
The singular value spectrum of the OOD datasets is shown in Figure 12 and Figure 14 respectively. From the top singular values visualization, we can observe that the clean pre-trained model tends to present larger singular values in this range, especially the largest singular value. We connect this observation with the transfcrability performance on OOD tasks (Chen et al. 2019),, and summarize it LSVR introduced in Section
B EXPERIMENTS
More details of experiments in Section 4 are shown here.
B. DETAILED SETUP FOR VISION MODELS EXPERIMENTS
We provide a more detailed setup of evaluation on practical vision models. First, we summarize the noisy pre-trained models with their pre-trained dataset, parameter size, and validation accuracy
22

---

Published as a conference paper at ICLR 2024
Nik- Pock
Cie Pock
Pork
Vile Pock
Stak 122
Dalee
Toap Vile Por
Vile Park
2012
Lake Boak
Sugide Like Book
EmcolT
2000
Fock Lake
Volle Rock
Stcak:
TEEESC 1000)
FICEE 2030
TETEOO 20.313
Aea 200
Tanken Th 2000
Distro 20.2
Figure 11: ImageNet-1K K R50 in-domain (ID) feature SVD spectrum analysis
23

---

Published as a conference paper at ICLR 2024
Figure 12: ImageNet-IK R50 out-of-domain (OOD) feature SVD spectrum analysis
ImageNet- 1K used in Table 6 We the same ID vision and OOD vision datasets in Table and Table for evaluation. Each experiment is run with three random seeds.
Table Noisy vision models we evaluated.
Model
Pre Pre-trained Data
Pre-trained Method
Param. Size (M)
EfficientNet-B (Tan
2019
JFT-300M (Hinton et al.
2015
Noisy Student (Xie et
2020
12.23
ResNetv2-152x7 (He al. 20Thh
ImageNet-211 Kidni al.
2021
BiT Kolesnikiy al
2220
236.34
Swin-L. (Lu èl al. 2011
ImageNet-21K Kidmk el
Supervised [1 2021C
196. 74
VIT-L (Dosovitskiy et al. 2020
Laion-2B (Schuhmann et al.
2022
CLIP Radford et al.
2021
304.20
ConvNext (Lau el al.
47224
Laion-28 (Schhhman ot al.
2022
CL.IP Radlordd et al.
202T
200.13
We mainly compare method with MLP tuning and LP, where fine-tuning the modules using
AdamW (Kingma & Ba 2014) for 30 epochs with a cosine learning rate scheduler. We set the learning rate 0.1 and weight decay of 0 for LP, and set the learning rate as 0.001 and weight decay of 1e-4 for MLP tuming and our method.
B.2 DETAILED RESULTS FOR VISION MODELS EXPERIMENTS
More results on each evaluated dataset are provided here. The ID results with standard deviation in accuracy each ID datasets are shown in Table 7 and the OOD results with standard deviation in accuracy on the evaluated OOD datasets are shown in Table g
Table Evaluation our method on vision models in practice that are pre- pre-trained on noisy datasets. We compare different methods on 14 vision datasets for in-domain (ID) evaluation
CIPARTE
Preesin
Redti
FOVE
SOVEV
PEAM
Readow
Akg
MLP
24
SEBLLE
4140110
15.09 LACE
2508 2015
NLP
M1:2-411
0113-441
15.0
17P
11
01300111
Wat-8a
1640-117
140-144
15346631
Tee
LP
00 1.11-:00
032-411
01 1355
mL8
01340
1118-111
OL
MLP
17500814
34
010011
14.-000
01-1-111
02354045
7413-0-35
65244015
1276403)
11.0
B.3 DETAILED SETUP FOR LANGUAGE MODELS EXPERIMENTS
The model details for natural language processing are shown in Table 9 We did not leverage larger language models mainly due to the limited computational resources. The recent open-sourced lan- guage models, such as Llama, have been trained very large-scale corpus of the web, evaluating them on GLUE and GLUE-X has the possibility to impose the problem of performing testing on the training samples.
24

---

Published as a conference paper at ICLR 2024
Vale Pock
Vile Puck
Boak Luiee
Stater Via
Vilte Pork
Rock Dalee Cnatenti www.erittti....
Suprler
Jake
20d
Pook
Duchit
Tode fork Inkes
Loae Fodk
Snile Fondh
Secck: S23
EEEEE 2000
Tes Soater 2000
Taahen 2200
Diotion C. 21
Figure 13: YFCCI5M R50 in-domain (ID) feature SVD spectrum analysis
25

---

Published as a conference paper at ICLR 2024
is 13:20
$10.0
Spale
Bock
Viae Pork Take"
Frse Se Px20
Teaee Cad 20.
For Seree
Taur Mal2 1200
Teamc 21..2
Skey 5i0e
Mae &
2112
Lme wwoollett
Figure 14: YFCC15M R50 out-of-domain (OOD) feature SVD spectrum analysis
26

---

Published as a conference paper at ICLR 2024
Table Evaluation of our method on vision models in practice that are pre-trained on noisy datasets. We compare different methods on 4 DomainNet datasets for out-of-domain (OOD) evaluation. We perform training on either DomeenNetSketch or DomainNetReal, and evaluate on DomainNetS- ketch, DomainNetReal, DomainNetPaining DomainNetClipart without the training set.
Pre-triined Model
Toning Mathod
DomainNet Shetch
DomainNet Real
Asg
46.22.16.17
Somi- Supervised
MLP
18.29115.78
13.61-27.53
15.95
EliciemiNet-33
Ouns
48.8415.63
44.83-7745
46.84
ImagoNat-21K
LP
41 11.09114881
40.55-77.89
40.82
Supereised
MLP
41,47=745
41.73
BeNNev2-152..
Ouns
42.68:14.85
42.15
42.42
21K
LP
$4.57.1.700
17.19
50.88
Fully Supervised
MLP
53.8111.001
6.:88.122
51,21
55.6N-15.48
49.01
52.35
Laion-28
LP
66:66-27
47.05::455
66.86
CLIP
MLP
66.78.17.00
70.07
68.43
CoutN
Ours
69.74116.22
7.785::4.77
70.30
Laion-2B
7.
CLIP
MLP
68.99:1:.599
70 70:00:44.66
69.50
VIT-L
Ous
70.48.16.91
40.45_
70.47
Table 9: Noisy language models we evaluated.
Model
Pre-trained Data
Pre-trained Method
BERT-L Devlin et al. 2018
BooksCorpus and d English Wikipedia BooksCorpus and d English Wikipedia
Masked Modeling (Devlin et al 201 Masked Modeling (Lu et al. 2019
RoBERTa-L L Luu et al
2019
GPT-2
Radford et al.
2019
WebText
Autoregression
iexl-ada-102
Now, we present the dataset details here used in our analysis. For ID evaluation, we use CoLA,, SST-
2, MRPC, STS-B, QQP, MNLI, QNLII and RTE tasks of GL.EE benchmark (Wang et al. 2018). shown in Table 10 For OOD evaluation, following GLUE-X,
use Grammar Test (Yang cct al. (023) for CoLA, IMDB (Maas et al.] 201T) for SST-2, QQP for MRPC, MNLI mismatched WWilhams al., 2017). SNLI (Bowman et al. 2015), SICK (Zhang al. 2018) for MNLI, Recon- structed NewsQA (Trischler et 20166) for QNLI, SciTail (Khot et al. 2018) and HANS (McCoy
al. 2019) for RTE, as shown in Table 11
Table 10: Details of the 8 in-domain (ID) tasks of GLUE used to evaluate ID transfer performance.
Dataset
Classes
Train Size
Test Size
Evaluation Metric
CoLA
8,500
1,000
matthews correlation
SST-2
2
67,000
1,800
accurncy
MRPC
3,700
1,700
accuracy
STS-B
7,000
1,400
pearson correlation
QQP
364,000
391,000
accuracy
MNL.I
3
393,000
20,000
accurncy
QNLI
105,000
5,400
accuracy
RTF
2
2,500
3,000
accurncy
We use the AdamW optimizer and set the learning rate for LP as 0.01 and for others as 0,001 for all the experiments of language models. For LP, we do not weight decay, and for others a weight decay of 0.0001. All tuning methods are trained for 10 epochs with a linear learning rate scheduler.
B.4 DETAILED RESULTS FOR LANGUAGE MODELS EXPERIMENTS
The detailed ID and OOD results of language models evaluation are shown in Table 12 and Table [3 respectively. NMTune outperforms LP and MLP tuning across all the tasks, whereas MLP tuning sometimes fall short than LP, demonstrating the necessitate of using the proposed regularization terms to help mitigate the effect of noise in pre-training and improve generalization performance.
27

---

Published as a conference paper at ICLR 2024
Table 11: Details of the out-of-domain (OOD) tasks of GLUE-X used evaluate OOD transfer
performance.
Dataset
Classes
Test Size
Evaluation Metric
Grammar Test
2
304,777
matthews correlation
IMDB
2
50,000
aecuracy
MILI mismatched
2
9,832
accuracy
SNLI
570,152
accuracy
SICK
2
9,840
accuracy
NewsQA
2
119,525
accurncy
Seifail
2
26,527
accuracy
HANS
2
60,000
accuracy
Table 12: Evaluation our method on language models practice that are pre-trained on noisy datasets. We compare different methods GLUE dev set for in-domain (ID) evaluation.
CoLA MCC
MNLI Ace
MRPC Acc
QNLI Acc
QQP Acc
RTE Acc
SST2 Acc
STS PCC
Model
Tuning
Avg
LP
46.18
62.75
72.05
74.10
84.83
50.90
88.19
76.55
69.44
BERT-
MI.P
46.99
63.97
72.30
73.80
84.88
51.62
88.30
76.36
69.78
Ours
46.12
64.37
73.04
74.25
85.11
52.54
88.84
76.64
70.26
LP
41.09
59.27
76.47
76.07
83.19
55.96
90.02
75.93
69.76
RoBERTa-L
MLP
42.93
60.38
76.22
76.11
82,94
57.40
90.25
75.92
70.27
Ours
43.91
60.59
77.69
76.33
83.30
58.96
90.71
76.24
70.97
LP
4.86
53.11
72.55
67.65
78.82
56.68
84.75
40.93
$7.42
GPT-2
MLP
4.46
53.38
73.53
67.17
78.72
56.68
84.75
40.93
57.19
Ours
6.41
54.21
73.77
68.09
78.95
57.07
85.14
41.09
58.09
LP
22.92
69.85
64.74
62.14
74.96
51.98
91.05
18.04
56.96
text-ada-002
MLP
33.90
70.49
66.42
69.54
84.28
52.34
91.85
42.28
63.89
Ours
36.96
72.05
70.34
70.25
85.54
54.51
92.89
45.39
65.99
Table 13: Evaluation of method language models in practice that pre-trained noisy datasets. We compare different methods on GLUE-X for ouf-of-domain (OOD) evaluation.
CoLA MCC
MNLI Acc
MRPC Acc
QNLI Acc
QQP Acc
RTE Acc
SST2 Acc
STS PCC
Model
Tuning
Avg
LP
13.71
36.82
55.85
63.60
53.14
50.67
73.66
57.81
50.65
BERT-L
MLP
14.11
36.49
56.15
63.61
52.34
50.02
73.32
58.91
50.62
Ours
14.57
37.62
56.68
63,88
55.17
50.98
74.52
59.61
51.63
L.P
24.07
37.05
48.51
57.83
24.93
51.02
71.24
41.74
44.55
RoBERTa-L
MLP
24.50
37.32
48.96
58.56
24.12
52.51
73.86
41.95
45.22
Ours
25.29
37.84
49.52
59.72
26.78
53.19
81.42
42.29
47.01
LP
4.12
24.93
45.24
$4,00
16.22
49.50
53.75
45.68
36.68
GPT-2
MLP
4.89
25,20
47.99
54.11
17.21
49.71
53.60
45.21
37.24
Ours
5.31
32.56
49.78
$5.97
17.34
49.96
54.28
47.39
39.07
8.06
65.14
38.66
58.12
32.60
51.53
84.07
14.30
44.06
text-ada-002
MLP
15.47
67.23
44.91
63.59
50.24
51.44
83.11
34.46
51.31
Ours
17.50
68.97
46.41
63.81
55.34
52.81
85.16
37.80
53.48
B.5 TRANSFERRING ON NOISY DOWNSTREAM DATASETS
We additionally study the setting where both pre-training and downstream datasets contain noisy labels. For pre-training noise, we use the ResNet-50 models pre-trained on noisy ImageNet- and YFCCI5M with different noise ratios 7 (0%, 5%, 10%, 20%, 30%), ), as Section 2 For downstream noise, adopt synthetic noise CIFAR-10 10 and CIFAR-100 which are usually used in noisy label learning (Liu et al. 2022a Chen et al. 2023). We generate symmetric label noise by uniformly flipping labels for percentage of the training set for all classes. We denote the noise ratio of downstream datasets as 7, and set it to (0%, 10% 20%. 30%, 40%, 50%). We compare LP and NMTune in this setting, shown in Figure 15 and Figure T6 respectively.
28

---

Published as a conference paper at ICLR 2024
2
15
105
E
103
Amsrayy
Ausurtty
150
E
81
70
70
004
200 30X
30%
off
105
3014
30%
072
3nt
505
075
1222
301
Divmucccam Soier Ralio
Downcrcam Noiee Halio
Dowrrtramm Noise Salio
Downstrram Noice Aulio
(a) IN-IK IK CIFARIO
(b) IN-IK CIFARI00
(c) YFCCI5M CIFARIO (d) YFCCI5M CIFARI00
Figure 15: Linear Probing of noisy ResNet-50 models on noisy CIFAR- 10 and CIFAR-
TU
8O
21
Arsinryy
Auturaai
G0
21
231
102
115
00
35
00
075
3000
300
30%
505
075
50%
30X
35%
M65
175
100%
300
30%
M55
175
100%
300
30%
10%
Donane seam Nuter Ralin
Douiee roam Nuiar Rolin
Dowiteroom Nator Rolin
Dowsred Naioe Rolin
(a) IN-IK CIFARIO
(b) IN-IK CIFARI00
(c) YFCCI5M CIFARIO (d) YFCCI5M CIFARI00
Figure 16: NMTune noisy ResNet-50 models on noisy CIFAR-10 and CIFAR-100.
On the LP results in Figure 15 we find similar observations as our analysis in Section where the 5% and 10% noisy pre-trained models usually outperforms the clean pre-trained model down- stream tasks, even the downstream tasks contain different level of noise. It indicates that the same conclusion from main paper may extend and generalize to noisy downstream tasks, which high- lights the importance of the proposed new topic Noisy Model Learning as the complementary to noisy label learning.
More importantly, we find that the proposed NMTune method has similar mitigation effect on noisy downstream tasks the clean ones. On the NMTune results in Figure [6 show that the clean pre-trained models now produce superior performance compared to noisy pre-trained models by utilizing the proposed regularization terms. It also improves the general performance when the noise ratio in downstream tasks is light, e.g., smaller than 40%. When the noise ratio in downstream NMTune fall shorts LP, which acceptable because
tasks further increases, the performance the regularization terms are not designed to be noise-tolcrant. Noteworthy is that, even with slightly worse performance than LP, the performance of clean pre-trained mode still stays the best with NMTune. Devising NMTune to be more noise-tolerant on downstream tasks and experiments on practical asymmetric and instance-dependent noise (Wei et al. 2022b) would be very interesting and leave for the future exploration.
B.6 RUNTIME ANALYSIS
The runtime analysis for NMTune, comparison to LP and MLP tuning is shown in Table I4 All of our experiments on downstream conducted single NVIDIA V100 GPU. Thus report the average GPU hours for rumning the ID and OOD evaluation vision and language tasks. From the results, the proposed NMTune introduces minimal computation, compared MLP with the exactly the same parameters. The additional computation burden may involve in the SVD calculation and the covariance matrix calculation the features.
29

---

Published as a conference paper at ICLR 2024
Table 14: Average run time for LP, MLP, and NMTune (Ours) in terms of GPU hours across in- domain and out-of-domain vision and language datasets.
Datasets
LP
MLP
Ours
Vision In- In-Domain (14)
5.01
7.96
8.23
Vision Out-of of Domain
Langunge In-Domain (8)
2,87
3.76
3.82
Language Out-of-Domain (14)
121
1.34
1.45
MI.P
ReLU
Layer 2, 2,40 x
Figure 17: Architecture of the 2-layer MLP with ReLU activation.
B.7 ABLATION STUDY
The ablation study of NMTune is present here, where we run evaluation on the ID vision datasets. We use three ResNet-50 models from ImageNet-IK pre-training and YFCC15M pre-training for ablation, including the clean pre-trained, 5% noise pretrained, and 10% noise pretrained.
We study the MLP architecture, more specifically, the non-linearity and the number of layers in MLP in Table 15 From the results, one observe that removing the non-linearity reduces the per- formance significantly. Adding more layers only improves the performance slightly but introduces much more parameters. Thus adopt the 2-layer MLP architecture with ReLU activation. The overall structure is shown Figure I7
Table 15: Ablation study of MLP architecture on ID vision datasets.
IN IK Pre trained
YFCC15M Pre trained
Layers
Activation
Models
TD Accuracy
Accuracy
67.34
2
ReLU
5
74.87
67.44
0
73.83
65.62
2
None
5
74.17
66.14
10
73.39
65.51
0
75.16
67.56
ReLU
5
67.49
10
74.48
67.13
75.13
67.51
4
ReLU
5
74.92
67.48
10
74.14
67.02
We also conduct ablation on the loss weight of different regularization terms we proposed in Ta- ble [6] From the results, we find that the proposed covariance regularization Lcov in general rectifies the effect of noise, improving the performance of clean pre-trained models to achieve better results than noisy pre-trained models. We can also observe that the dominant singular value regu- larization LsvD helps improve generalization. Solely adding LMSE or LsvD does not produces this behavior and yields slight worse results.
C MORE DISCUSSIONS
More discussions about work are provided here.
30

---

Published as a conference paper at ICLR 2024
Table 16: Ablation study of different loss weights on ID vision datasets
IN-IK Pre-trained
YFCCISM Pre-trained
LMSR
Leow
Cavn
Models
ID Accuracy
ID Accuracy
0
75.06
67.34
0.01
0.01
74.87
67.44
10
74.27
66.95
0
74,34
66.59
0.00
0.01
5
74.18
66.54
10
74.09
66.17
73.65
65.41
0.01
0.00
0.01
5
74.23
66.02
10
73.56
65.39
74.
66.27
0.01
0.00
0.00
5
74.24
66.31
10
73.17
66.08
74.74
67.03
0.01
0.01
0.00
5
74,32
66.83
73.92
66.70
0
74.41
66.51
0.00
0.01
0.00
5
74,20
66.47
73.98
66.12
0
72.21
65,08
0.00
0.00
0.01
5
73.49
65.24
72.87
64.76
0
73.96
66.11
0.00
0.00
73.97
66.19
10
73.13
65.93
C.I LIMITATIONS
The limitation mainly lies in our empirical study the noise in pre-training. Due to the limited computing resources, could only conduct experiments reltively small scale backbone and datasets, while most of the SOTA foundation models are of much more parameters and are trained much larger datasets. Also, the empirical experiments is limited actual supervised pre-training. Other pre-training objectives will be explored our future work. That being said, we do believe the observation and conclusions from practical experiments could scale and extend larger datasets, stronger backbones, and other training objectives.
C.2 POTENTIAL FAILURE
We do observe some failure cases of the proposed methods. For example, from the results in Ta- ble.7, the proposed method falles short to LP Caltech101 almost all backbones studied, while improving over MLP. Our hypothesis for the failure that the SVD regularization term the proposed method might need to optimize the top-K singular values instead of just the largest The optimal value K might be different dataset. However, setting can already achieves reasonable performance for most of the tasks.
31


Reviews:
[{"review_id": "m9LVgU5ZVq", "reviewer": "Reviewer_jd9i", "summary": "The paper addresses the challenges posed by label noise in pre-training datasets and its impact on downstream tasks. The authors focus on supervised pre-training models using synthetic noisy ImageNet-1K and YFCC15M datasets. They observe that while slight noise in pre-training can enhance in-domain (ID) transfer performance, it consistently harms out-of-domain (OOD) performance. The reason behind is noise in pre-training shapes the feature space differently. They introduce a lightweight black-box tuning method, NMTune, to mitigate the adverse effects of noise and improve generalization on both ID and OOD tasks.", "questions": "- Self-supervised pre-train does require external supervision. Does it mean those models will not suffer from the noise issue? \n- Does it proposed method generalize to self-supervised pre-trained models?\n- When you trained CLIP, did you train the text encoder together or you use a frozen text encoder?\n- \"For OOD evaluation, we use DomainNet (Peng et al., 2019) where we train on either “real” or “sketch” images and test on “real”, “sketch”, “inpainting”, and “clippart” images\". If you trained on either “real” or “sketch”, you should only test on domains that the model did not seen during training right？ This sentence is a bit confusing.\n- \"we empirically analyze the singular value spectrum of the pre-trained **the** feature space on downstream datasets\" Typo: extra \"the\"\n- Section 2.3,  the authors should let or remind the readers what are M and D. They should be the number of samples and latent dimension size right?\n- Figure 3 is a bit confusing. For a specific noise level, there are many points (e.g. many blue stars). What does each point mean? One downstream task? A lot of points are overlapped and I don’t know which 5 points (0%, 5%, 10%, 20%, 30%) should be read together. \n- \"An initial increase in the spanning dimension of the feature space is beneficial to the discriminability on ID tasks. \" The reason behind that is \"the pre-trained feature extractor captures more structure in data\" due to noise. But why more structure does not help OOD?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper studies a problem that is both practical and significant, yet has not been sufficiently investigated in prior research.\n- This paper is well-motivated and easy to follow.\n- The analysis of features is useful to understand the noise's impact on ID and OOD data. \n- Experiments are comprehensive.", "weaknesses": "- Improvements are needed in the writing and presentation quality. Please check the Question section below. \n- Some claims in the paper are unclear and confusing. Please check the Question section below. \n- The authors did not mention the limitations of their method and potential future work. The paper does not explore or discuss potential failure cases of the proposed methods. Understanding when and why the methods might fail is crucial for practical applications", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ymMfVtI7k", "reviewer": "Reviewer_5WgV", "summary": "This paper endeavors to comprehend the underlying characteristics of noise within pre-training datasets and seeks to mitigate its influence on downstream tasks. The study reveals that the noise present in pre-training datasets exerts distinct effects on in-domain (ID) and out-of-domain (OOD) tasks. In the case of ID tasks, slight noise during pre-training can yield improvements in in-domain transfer performance. However, for OOD tasks, noise consistently degrades out-of-domain performance. To substantiate their findings, the authors employ Singular Value Entropy (SVE) and Largest Singular Value Ratio (LSVR) to capture the behavior of the trained features in both ID and OOD tasks. Subsequently, the authors devise a loss function that enhances the SVE and LSVR of these features, resulting in superior overall performance.", "questions": "See **Weaknesses**", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The impact of noise within pre-training datasets on subsequent tasks has not been thoroughly investigated in the existing literature. The insights presented in this paper, such as the distinct effects of noise on in-domain (ID) and out-of-domain (OOD) tasks, are novel and intriguing.\n\n- The design of the loss function is a direct consequence of the insights gained from observations, and experiments demonstrate that this designed loss outperforms the Cross-Entropy (CE) baseline, including LP and MLP structures.\n\n- Experiments encompass a wide range of tasks, including both image and image-language tasks. Furthermore, various popular base model structures are used in this paper.", "weaknesses": "-  I find Figure 3 challenging to interpret. It consists of numerous data points for each configuration, lacking connecting lines. Consequently, I struggle to draw the conclusions reached by the authors based on this figure.\n\n-  I've observed a potential contradiction between SVE and LSVR. For instance, in the case of two-dimensional features, [1.0, 0.0] exhibits the highest LSVR while having the lowest entropy. It would be beneficial if the authors could provide further clarification regarding this inconsistency.\n\n-  I would appreciate it if the authors could offer more detailed explanations as to why a slight amount of noise can benefit in-domain (ID) tasks. This is somewhat contradictory to the existing literature on learning with noisy labels, and additional insights would be valuable.\n\n-  In the paper, the authors claim that the proposed method enhances SVE and LSVR. However, Figure 5 (d) indicates that the proposed method does not yield superior LSVR compared to the LP and MLP models. Furthermore, it seems that as the noise ratio increases, LSVR does not drops significantly for all the settings.\n\n- It's worth noting that some related work, such as [R1], also employs Singular Value Decomposition (SVD) to address noisy label problems. It would be beneficial for the authors to discuss the distinctions between your approach and previous work.\n\n- I would like to point out that the improvements achieved by the proposed method appear to be relatively modest. According to the experimental results, the proposed method only demonstrates an approximately 1% improvement compared to the MLP model.\n\n- For ResNet-50, it might be worthwhile to explore the feasibility of fine-tuning all layers rather than constraining the encoder. It would be valuable if the authors could conduct experiments to determine if the proposed loss function is effective in cases where all layers are fine-tuned.\n\n[R1]: FINE Samples for Learning with Noisy Labels", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QYk3fqRpkB", "reviewer": "Reviewer_vftf", "summary": "This work aims to study the noise in pertaining data and its impact on downstream tasks. The authors exploit the Singular Value Entropy (SVE) and the Largest Singular Value Ratio (LSVR) to analyze the singular value spectrum of the pre-trained feature space, and discover that proper noise in pre-training data increases both SVE and LSVR, leading to better transferability and worse robustness. Based on the observations, the authors further introduce an MLP together with three regularizations to transform the pre-trained features into a better feature space. Experiments with different model architectures and datasets are conducted to demonstrate the effectiveness of the proposed method.", "questions": "- On what scale the SVE and LSVR  is computed? The entire dataset?\n- Do the conclusions (Fig1- 3) always hold for stronger backbone models other than resnet50?\n- In the loss function, are the regularizations computed per batch?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The analysis of feature space with the singular value spectrum is interesting and meaningful. Rich experiments and analyses are conducted to show how the noise in pertaining data can impact the learned feature embedding.\n- The proposed regularizations are intuitive and effective. Extensive comparisons are presented to show the improvements.", "weaknesses": "- This paper is featured with extensive empirical results. However, the core techniques in methodology (analysis and regularization of singular value spectrum ) have been studied in existing works[e.g. Chen et al., 2019, Bardes et al. 2022], which may undermine the theoretical contribution of this work.\n\n- Some figures are hard to understand by themselves. E.g. different types of marks are cluttered in Fig3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hCxBqf15aq", "reviewer": "Reviewer_JmqY", "summary": "This paper presents a novel topic of studying the effect of pre-training noise on various downstream datasets, termed noisy model learning.\nThe authors conduct the empirical study and analysis of noisy ImageNet and YFCC15M of supervised pre-trained and contrastive pre-trained ResNet50 models and illustrate that slight noise in pre-training improves performance on in-domain downstream tasks but always hurts the performance on out-of-domain tasks. From the singular value analysis of the pre-trained feature space, the authors designed two metrics that in general align with the downstream empirical observations.\nThe authors also propose several regularization terms based on the singular values of features that can mitigate the noise in pre-training in a block-box tuning manner. The authors provide comprehensive experiments to verify the effectiveness of the proposed method and offer interesting analyses and discussions.", "questions": "Since ImageNet or YFCC15M itself also originally contains noise, is there any optimal noise ratio that achieves the best ID downstream performance?\nSince NML assumes an inaccessible pre-trained model, how is other black-box tuning methods perform on the noisy model learning setting?", "limitations": "", "rating": 10, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper is generally well-written and organized.\nThe authors provide a first novel and interesting study on the effect of pre-training noise, demonstrating the importance of this research topic, especially in the context of large foundation models. \nThe empirical study for revealing the effect of pre-training noise is extensive and comprehensive, including both in-domain and out-of-domain datasets from various distributions. \nThe proposed method may not be very innovative, but it is simple and verified on both CV and NLP tasks with different large backbones. The method also works in the API case mentioned in the paper.\nThe authors also additionally study the combination of the proposed noisy model learning and traditional noise label learning, demonstrating the effect of noise in pre-training also exists when downstream data has noise. \nThe detailed results, experiments setup, and ablation study are presented in the Appendix.", "weaknesses": "How to introduce synthetic noise in ImageNet and YFCC15M needs more explanation.\nThe pattern SVE analysis of the ImageNet model and YFCC15M model are slightly different in Fig.3, and perhaps need more explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)