Denoising Pretrained Black-box Models via Amplitude-Guided Phase Realignment
Authors: Hongliang Ni
Tong Chen
Shazia Sadiq
Gianluca Demartini
Venue: Accepted by TMLR ()

Abstract: Pre-trained models tend to inherit noisy label information from their training datasets, internalising it as biased knowledge. While learning with label noise has been explored, existing approaches rarely address the mitigation of biased knowledge embedded in pre-trained representations introduced by noisy labels. Moreover, existing denoising methods invariably rely on modifying training datasets or models to improve downstream task performance. However, we observe a growing trend in which both pre-trained models and their training datasets are scaling up significantly and becoming increasingly inaccessible, making modifications ever more infeasible. In this paper, we propose a black-box biased knowledge mitigation method called ``Lorem'', which leverages feature frequency amplitudes to guide phase correction on pre-trained representations, without access to training data or model parameters. We first present empirical evidence that, across different noise levels, the phase components of pre-trained representations are more sensitive to noisy labels than the amplitude components, while discriminative information for classification is primarily encoded in the amplitude. Moreover, we find that the impact of noisy labels on amplitude is global, leading to a gradual loss of discriminative information. Therefore, corrective strategies must be adaptive across the entire frequency spectrum rather than limited to the high-frequency components. Inspired by this observation, we design a method that leverages the amplitude residual to realign phase, thereby removing biased knowledge from pre-trained representations. Experiments on a variety of popular pre-trained vision and language models suggest that, even with a simple linear classifier, our method can enhance downstream performance across a range of in-domain and out-of-domain tasks.

Published in Transactions on Machine Learning Research (12/2025)
Denoising Pretrained Black-box Models via Amplitude-Guided Phase Realignment
Hongliang Ni
ughniSug.edie edu.ou
School of Electrical Engineering and Computer Science University of Queensland ARC Training Centre for Information Resilience
Tong Chen
tong. chen tong.cher
Sugq. edua
School of Electrical Engineering and Computer Science University of Queenslond
ARC Training Centre for Information Resilience
Shazia Sadiq
shazia Maazia&deess.oo uq. edu..a
School of Electrical Enginecring and Computer Science University of Queensland
ARC Training Centre for Information Resilience
Gianluca Demartini
d.demartiniuug edu
School of Electrical Engineering and Computer Science
University of Queensland
ARC Training Centre for Information Resilience
Reviewed on OpenReview: https: // wpw.eeviee.........emem net/ forum? id- 526futtJik
Abstract
Pre-trained models tend inherit noisy label information from their training datasets, internalising it as biased knowledge. While learning with label noise has been explored, existing approaches rarely address the mitigation of biased knowledge embedded in pre- trained representations introduced by noisy labels. Moreover, existing denoising methods invariably rely on modifying training datasets or models to improve downstream task per- formance. However, we observe a growing trend which both pre-trained models and their training datasets are scaling up significantly and becoming increasingly inaccessible, making modifications ever more infeasible. In this paper, we propose black-box biased knowledge mitigation method called "Lorem", which leverages feature frequency amplitudes to guide phase correction pre-trained representations, without access to training data model parameters. We first present empirical evidence that, across different noise levels, the phase components of pre-trained representations are more sensitive to noisy labels than the am- plitude components, while discriminative information for classification is primarily encoded in the amplitude. Moreover, we find that the impact of noisy labels on amplitude is global, leading to a gradual loss of discriminative information. Therefore, corrective strategies must be adaptive across the entire frequency spectrum rather than limited to the high-frequency components. Inspired by this observation, we design method that leverages the amplitude residual to realign phase, thereby removing biased knowledge from pre-trained representa- tions. Experiments on variety of popular pre- treitrained vision and language models suggest that, even with a simple lincar classifier, our mothod can enhance downstream performance across range of in-domain and out-of-domain tasks.

---

Published in Transactions on Machine Learning Research (12/2025)
1 Introduction
As public expectations for the performance of pro-trained modcls continue to risc, to meet these expectations, pre-trained models increasingly rely on web-crawled data and large-scale annotations, which makes these large-scale pre-training datasets inevitably affected by noisy labels (Piktus et al. 2023 A growing concern is that noisy labels affect the learned weights of pre-trained models, leading to biased knowledge being encoded in their representations. For example, when benign tumours are mislabeled as malignant during pre-training, the model may overemphasise irrelevant pixel-level artefacts. This biased knowledge affects the gencralisation and transforability of pre-trained modcls in downstream tasks, and addressing the problem by simply fine-tuning the representations with downstream data is often suboptimal. Since noisy labels alter the feature space, these distortions constrain the optimisation capacity of fine-tuning, especially when the downstream dataset is small or domain-shifted.
Existing research in learning with noisy labels has mainly focused on the training phase, with mainstream
methods generally categorised into noise modelling (Van Rooyen et al. 2015 Han et al. 2018a Van Rooyen
& Williamssn, 2018 Yao et al. 2019) and model-based techniques (Reed et al. 2014 Liu & Tao 2015
Thulasidasan et al. 2019 Lyu & Tsang, 2019 Yu et al.) 2019 Han et al. 2020). These methods assume
access to the model's ss training data, architecture, and parameters, allowing them to mitigate the effects of label noise to some extent. However, such strategies are difficult to apply to improving the performance of large-scale pre-trained models on downstream tasks. Due to their size, re-training pre-trained models is often computationally infeasible- even when noisy labels are identified. Moreover, both the datasets and models are not always publicly available. Therefore, it remains an open problem to design effective strategies for mitigating the influence of noisy labels in pre-trained models in a black-box setting, where neither the original data nor model parameters are accessible.
In this paper, we explore frequency-based approach to mitigating biased knowledge in pro-trained repre- sentations, leveraging the Fourier transform global transformation that decomposes representations into frequency components. This decomposition allows us to disentangle amplitude (global energy distribution) and phase (structural layout), enabling us to identify and correct frequency-speciffe distortions introduced by noisy supervision, thereby mitigating biased knowledge. We begin by presenting an empirical analysis, where amplitude and phase signals obtained under different. levels of noisy supervision are used as predictive features, and the performance of classifiers built on these features is compared across downstream tasks, Our analysis shows that both amplitude and phase affected as the noise level increases. However, classi- fiers built on amplitude signals consistently outperform those using phase, suggesting that phase should be the primary target of correction, as restoring distorted phase components is key to recovering the seman- tie integrity of pre-trained representations and improving their quality for downstream tasks. To further investigate, we examined how noisy labels affect amplitude and found that as the noise level increases, the magnitude, structural patterns, and distribution of amplitude have significant global changes, rather than being limited to the high-frequency components emphasised in prior work (Chen et al., 2021). These fluctu- alions imply that amplitude encodes label-related global patterns that are progressively distorted by noisy supervision. Thus, while amplitude itself is affected, its variations provide reliable signal of where and how noise alters the feature space, making it a natural guide for phase realignment, where amplitude residuals are used to realign phase components, thereby mitigating biased knowledge in pre-trained representations.
Motivated by these findings, we propose "Lorem", a lightweight black-box framework designed to improve the generalisation ability of pre-trained representations. We first transform pre-trained features into a label- guided feature space, while preserving their original amplitude information. We then compute the amplitude residuals defined as the difference between the amplitudes in the original feature space and those in the new feature space and use them to guide the correction of the phase components. The aligned phase, along with the new amplitude, is combined via the inverse Fourier transform to generate enhanced representations, aiming to remove biased knowledge and support more reliable transfer to downstream tasks.
The main contributions of this paper are as follows:
We provide an empirical analysis of amplitude and phase components under varying levels of label noise, demonstrating their differences in sensitivity to noisy labels and their relevance to discrimi-
2

---

Published in Transactions on Machine Learning Research (12/2025)
native information. Furthermore, subsequent experiments reveal that label noise globally changes both the structural patterns and the magnitude of amplitude. This highlights the potential of using amplitude to guide the correction of phase for mitigating biased knowledge in pre-trained represen- tations.
We propose lightweight black-box method, named "Lorem", which uses amplitude residuals to guide the adjustment of phase. The enhanced features are then reconstructed via the inverse Fourier transform, allowing for improved generalisation and transferability of pre-trained models with min- imal computational cost.
We conduct extensive experiments on a variety of popular pre-trained vision and language models, evaluating both in-domain (ID) and out-of-domain (OOD) downstream tasks. Our mothod is com- pared against strong baselines and widely-used fine-tuning techniques, consistently demonstrating superior performance.
2 Problem Formulation
We assume the pre-training dataset is defined as D = {z,, yi Whe where it consists of inputs I; X and supervision y/ ~ y, and the dataset size is N. The inputs and corresponding supervision can take any form. For example, in an image classification task, the inputs are images and the supervision are class labels (Russakovsky et al. 2015 He et al., 2016); in a centenco-pairing task, both the inputs and outputs are sentences (Cer et al. 2017 Wang al. (2018). We further assume that the pre-training dataset contains noisy labels, such as data samples annotated with incorrect labels, which is common in large-scale pre-training datasets collected using web crawlers and similar techniques.
Thus, we define the noisy dataset as D = { j which we assume is inaccessible. The pre- tetirrained model can be abstractly viewed as a combination of a feature extractor and a projection head. The pre-trained representations are extracted from the feature extractor Se, and then mapped into the target domain using the projection head ge. Similarly, the parameters of the feature extractor fo are also assumed be inaccessible. Since the training dataset contains noise, the extracted pre-trained representations may not accurately reffect the semantic content of the samples. Therefore, representations to enhance the performance of pre-trained models on downstream lasks, without requiring re-training, To achieve this, we assume access to a downstream dataset D' = {æ,, 34 H which is used for optimization. Unless otherwisc stated, we consider D' to be clean (i.c., without additional label noisc).
goal is to repair the pre-trained
3 Analysis of frequency component robustness across noise levels
Given current amplitude recombination approaches for learning with noisy labels (Huang et al., 2023; Chen al.) 2021),, we are interested in investigating the downstream performance of the frequency compo- nents-ampline amplitude and phase-of pre-trained representations obtained from models under different levels of noisy supervision. In this analysis, we use five noisy pre-trained ResNet-50 (He et. al. 2016 models from Chen et al. (0024), each trained with different levels of label noise, each trained with different label noise ratio (0%, 5%, 10%, 20%, and 30%, where 0% denotes the clean dataset) using CNN backbones, to extract features from four image datasets: CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), Food 101 (Bossard et al. 2014) and ImageNet Val (Russakovsky et al. 2015) We then apply the Fourier transform obtain their amplitude and phase components, and compare their performance in linear probing.
As shown in Figure We have several observations:
(1) The classifiers using amplitude signals consistently outperform those using phase features in down- stream classification tasks. This can be attributed to the pre-trained model's inherent lendency to en- code label-related semantic information into the amplitude. The most visually obvious differences between categorics-sie such as colour intensity, odge thickness, and texture density-are directly associated with the amplitude. During training, the model tends to prioritise memorising such casily accessible information to converge more quickly, thereby embedding most of the discriminative information into the amplitude.
3

---

Published in Transactions on Machine Learning Research (12/2025)
OPAR12
DU CEPRLCO
tst Foed 201
Imagellet
01-2
0.55
74-
48-
00/3
85-
51.
7.4
21
15%
25%
37%
12%
325
27Ah
Figure 1: Classification accuracy on CIFAR-10 (a), CIFAR-100 (b), Food 101 (c), and ImageNet Val (d) using original pre-trained representations, their amplitude, and their phase under varying noise ratios pre-training.
VIT CIFA10
VT CIFA100
VIT on Food
07
62.5
80.5
$3.0
72.5
77.5
75.0
Orginal
Amrpiadd
12.5
32.5
Pheas
35.0
Swin DIAR1O
Swin CIFAL00
Swin on Food-103
Arts
Prase
A2200000
06
Buve
Lange
Small
Large
Smal
3am
Lage
Classification accuracy of Vision Transformer (ViT) and Swin Transformer models of different scales (Small, Base, and Large) on CIFAR-10, CIFAR-100, and Food-101. Each plot compares classifiers trained on original pre-trained representations, their amplitude, and their phase.
Figure 2:
Previous studies have also reported that CNNs favour amplitude over phase (Chen et al. 2021). In addi- tion, the amplitude typically follows a power-law decay (Ruderman & Bialek 1993 Simonnelli & Olshausen 2001), which means that most of the energy is concentrated in the low-frequency components, resulting in a globally redundant structure. For example, in images of dogs, the low-frequency amplitude mainly captures the overall impression that "there is large brown region with a light background." Because ad- jacent low-frequency components are highly correlated, amplitude patterns of images from the same class are almost repetitive. This redundancy makes the amplitude more robust to noisy labels and allows it to encode stronger generalisable information. In Figure 2 we further present results on two Transformer-based architectures Visioon Transformer (ViT) and Swin Transformer across multiple model scales (Small, Base, Large). With the exception of ViT models on the low-resolution CTFAR-10 dataset, we observe a consistent trend where amplitude- becomes more pronounced with larger model scales, higher-resolution datasets, and more fine-grained class distinctions. Given that CIFAR-10 is relatively low-resolution dataset with limited semantic variation across its ten categories, its practical value is somewhat limited. Therefore, we argue that the observed superiority of amplitude features is likely to generalise well to real-world applications.
-Visin
classifiers outperform their baaso counterparts. This performance gap
(2) In contrast, phase-based classifiers are more sensitive to noisy labels, resulting in poorer classification performance. The amount of classddisrrmminatiee information contained in the phase is relatively limited. Although positional and structural information of the image subject is indeed encoded in the phase, such geometric structures do not always directly reveal class distinctions, leading to relatively limited generaliza-
4

---

Published in Transactions on Machine Learning Research (12/2025)
tion performance. On the other hand, in the phase domain, this local-focus bias causes small-scale shifts, which make the phase distribution more dispersed. This distorts the geometric information in the phase and further degrades classification.
(3) Second, on the three datasets CIFAR-10, CIFAR-100, and Food-101, amplitude- based classifiers using 0%-10% noisy pre-trained representations show minimal performance differences, In fact, on CTFAR-100, there is even a slight improvement the 10% noise level. This performance trend may be attributed noisy labels inducing the pre-trained model to focus more local features. Under mild noise, the model may start associating a label with unique details of specific image (e.g., reflections or the edge of particular pixel), and memorising additional textures may accidentally increase discriminability, explaining why the 5% noisy model in Chen et al. (2024) slightly outperforms the clean model in downstream tasks. Finally, on the ImageNet Val dataset, the performance of amplitude- based classifiers is nearly identical to classifiers using the original pre-trained features. This suggests that the pre-trained model we used indeed encodes most of the discriminative information into the amplitude.
Overall, from the classification results the four image datasets, we find that the amplitude of pre- pre-trained features encodes more label-related discriminative information, whereas the phase is more sensitive to noisy labels. They severely disrupt positional and structural information, leading to a more pronounced perfor- mance degradation for phase-based classifiers. To further investigate the impact of noisy labels on amplitude, we present the variation the average amplitude magnitude of pre-trained representations from the CIFAR- 10 dataset across different noise levels.
As shown in Figure 3. we observe that, as the noise level increases, we clearly observe the gradual shift of magnitude in the feature amplitude from low frequencies toward mid- and high-frequence ranges, accompanied by progressively weakened structural patterns. Compared to clean pre-trained represen- tations, the energy ratio in the low-frequency range (frequency index 2000 decreases (from 46.98% to 45.50% to 44.37%) as noise increases (to 5% 10%). In contrast, the ratios in the mid-frequency (200-600) and high-frequency (600-1024) ranges in- crease (mid frequency: from 25.76% to 26.42% to 27.40%; high frequency: from 27.20% to 28.08% to 28.23% as noise increases. Under 5% noisy super- vision, the first amplitude peak shifts from the low- frequency range to the mid-frequency range. The second peak, which originally appears around fre- quency index 400, is also delayed and emerges near index 600, while the changes in the high-frequency range remain minor. The spectral centroid in- creases from 2.18 to 2.48. For amplitudes obtained from 10% noisy supervision, both peaks are further shifted, with the first. occurring around frequency index 400 and the second also around index 600. The spectral centroid continues to increase to 2.81. As the noise level increascs, the normalized spectral entropy of representations also shows an incrcasing trend, from 74.98% to 77.31%, indicating gradual weakening of structural regularity. At extreme noise ratios of 20% and 30%, the structure becomes disrupted, with no distinct peaks. At the 30% noise level in particular, the overall magnitude level is even reduced, indicating severe loss of discriminative information. Overall, we argue that the impact of noisy labels does not always manifest first as a simple high-frequency peak (Huang al. 2023). but rather as a global alteration of the amplitude magnitude structure. Therefore, our correction method docs not need to target any specific frequency band; instcad, by leveraging amplitude residuals, it can more effectively strengthen suppress particular frequencies as needed.
Smoothed Magnitude vs Level
350
Average Kagnituee
300
200
==
150
5%
10%
20%
y 30%
0
200
400
600
800
1000
Frequency Index
Figure 3: Smoothed amplitude magnitude across fre- quency indices under different noise ratios (7). We divide the spectrum into three ranges based on the fre- quency index: low frequency (0-200), mid frequency 2200-600), and high frequency (600-1000).
5

---

Published in Transactions on Machine Learning Research (12/2025)
4 Seplitude-Guided Phase Realignment
Motivation. Through experiments presented in Section 4, we observe that: (1) discriminative information is primarily encoded in the amplitude, while the phase highly sensitive to noisy labels; and (2) noisy labels in the pre-training dataset reshape both the distribution and the structural patterns of amplitude in pre- trained features. Since the distribution patterns of amplitude correspond to specific spatial structures, these structures in turn determine the distribution of phase. For example, repetitive textures (e.g., brick walls, grids) produce periodic peaks in the frequency domain, and the arrangement of phases at these frequencies decides the alignment of textures. Variations in amplitude also reflect changes in texture repetitiveness; if the periodic peaks are disrupted, it often indicates phase misalignment (Yu et al. 2022). Based on these observations, amplitude variations can serve as an effective indicator for guiding phase correction. The adjusted phase better aligns with the semantic content of the samples, eliminates the biased knowledge embedded in pro-trained features, and enhances generalisation on downstream tasks.
Algorithm 1: Amplitude Guided Phase Realigument Classifier (Lorem)
Input Pre-trained features E ER 3xd
Parameters: MLP weights (W,, W2) with BN+ReeLU; classifier We; learnable template & E Bxd (init π); step size E 0.01
Output Logits M BBC
Semantic branch: $ +- FF)
Fourier transforms (last dim):
X' + FFF), - Ax + JX|,
4x f L(X))
As + SS), $ - (S')
Amplitude residual:
+ tanh(Ax - As)
Phase update:
+ r क + Ds + t A4
Reconstruction (inverse Fourier transform):
2 + R (A )c
Classification:
M TWT
return M
4.1 Method
Our method aims to leverage amplitude residuals in the frequency domain to guide phase correction, thereby improving the generalisation of pre- pre-trained representations on downstream tasks. Specifically, given an input batch X, we extract the original pre-trained representation F. We then transfer this representation into a new semantic space through a two-layer multi-layer perceptron (MLP) hus obtaining a new pre-trainod representation batch S. To enable comparison the frequency domain, we apply the Fourier transform both and S:
='=F(F) Ax Oe exx ='= = As ② "s
(1)
where Ax, As E Bxd denote the amplitudes, and Φx, ts E [- Bxd denote the phases. Next, we compute the residual between the original and semantic amplitudes and regularise it with tanh(-)) function to ensure a stable training:
tanh(Ax As). r (-1,1)1)
(2)
Since the phase an angular variable with values in [-,, T), its update involves not only the magnitude of correction but also the sign (direction). However, the amplitude residual only reflects the strength of

---

Published in Transactions on Machine Learning Research (12/2025)
the discrepancy between the two representations, lacking guidance the direction of phase adjustment. To address this issue, we introduce learnable phase template Φ E [- x Bxdd with 6, the model can adaptively learn which components require stronger correction and in which direction the correction should be applicd. In other words, & provides the phase correction with frequency selectivity, avoiding the under- or over-correction caused by uniform updates across all frequencies. Moreover, its Icarnability provides task adaptivity, enabling the model to automatically learn correction patterns that best fit a specific downstream task data distribution, rather than relying solely on residual-driven updates. The phase correction term is given by the elementwise product of the amplitude residual and Ø, and be represented as follows:
x By combining the residual r
A @
(3)
DomainNet
40
original Amplitude
The updated phase is then computed as:
35.
Phase
30
① = ds t AΦ, $ e Bxd
(4)
25
Accuracy
20
where & 0.01 a global scaling factor that en- sures gradual adjustments and stabilises training. We present the hyperparameter analysis in Ap- pendix A.2 Here, we use &g instead of 4x because 2x comes directly from pre-training under noisy su- pervision, and therefore inherently contains biased knowledge. If used directly, it would easily bring noise into the correction process.
15.
10
Real Real--Sketch
Sketch RReal
Figure 4: Classification accuracy DomainNet using original pre-trained representations, their amplitude, and their phase when training on the "real" subset and lesting on the "sketch" subset, and vice versa.
Finally, the updated phase is combined with the mantic amplitude, and the inverse Fourier transform is applied to reconstruct the corrected representa- tion:
2 = RRR (As)))))) Ż E Bxd.
(5)
The operator R(-) denotes taking the real part to en- sure that the reconstructed representation remains in the real space Rdd The corrected representation Z is then fed into linear classifier We, and the entire model is optimised under the supervision of down- stream labels. Ablation study on amplitude residual guidance and model designs are in Appendix A.1
Accuracy under Different Types
GFAB-10
80
20
4.2 Discussion
Arreituce
#1350
One may argue that Lorem may fail if class-related texture patterns are not captured in the amplitude. To further investigate the effectiveness of our pro- posed method, follow the experimental setup of Section 3 and design two types of OOD tasks: (1) Using the "real" subset. of DomainNet as the train- ing set and the "sketch" subset as the testing set, and vice versa; (2) Using the original CIFAR-10/100 training sets while applying four types of blur per- turbations Gaussian blur, motion blur, glass blur, and defocus blur-u to the corresponding test sets.
Figure 5: Classification accuracy on CIFAR-10/100
using original pre-trained representations, their am- plitude, and their phase, when four types of blur per- turbations are applied to the test sets.

---

Published in Transactions on Machine Learning Research (12/2025)
As shown in Figure when the "real" subset of DomainNet is used for training, the amplitude- classifier slightly outperforms the phase-based classifier. However, when the roles are reversed, the phase-based classifier performs better than the amplitude-based one. This difference stemss from the fact that the decision boundarics learned from the "rcal" subset rely heavily on texture information encoded in the amplitude. Although such information is removed in the "sketch" subset through decolorisation and texture removal, the overall shapes remain, thereby preserving transferability. When the training set becomes the "sketch" subset, which only contains positional and structural information, the amplitude-bes colour and texture patterns, causing it struggle when applied to the more complex "real" subset. Figure D shows the OOD results under the four blur types. The amplitude-based classifier outperforms the phase- based classifier in all but the glass blur setting, which swaps local pixel pairs, reassigning class-specific texture pallerns, but leaves object edges and geometry unaffected; as a result, the phase retains some discriminative information. The other three types of blur mainly reduce fine-grained texture details while preserving the basic texture type, enabling amplitude-based classifier to remain effective.
classifier merely learns simple
In summary, our proposed method is able to achieve superior performance in most downstream tasks because the amplitude captures class-related texture patterns with relatively high generalisability As long as such patterns are not completely destroyed in the target domain, the model can still extract relevant class- discriminative information.
5 Experiments
Pre-trained models. For vision pre-trained models, we select the fully supervised ResNet-50 (e et al. 2016) and Swin-L (Liu et al.) 2021b), as well as the semi-supervised EfficieetNNetB33 (Tan & Le 2019). In addition, we include two noisy ResNet-50 pre-trained models trained on ImageNet-1K (IN-1K) (Russakovsky jet al. 2015) with 5% and 10% label noise, provided by Chen et al. (2024). In their setup, noisy supervision is simulated by uniformly flipping ground-truth class labels into other classes. For text pre-trained models, we select BERT-Large (Devlin et al. 2019). LoBERTa-Large (Liu et al. 2019), and GPT-2 (Radford et al., 2019h. Both BERT-Large and RoBERTa- Large are Transformer encoders, while GPT-2 is Transformer decoder pre-trained on WebText using large-scale autoregressive language modelling.
Datasets. We validate our model on seven in-domain (ID) vision tasks and two out-of-domain (OOD) vision tasks. For the ID tasks, we use seven downstream datasets, including CIFAR-10/100 (Krizhevsky et al. 2009). Caltech101 (Fei-Fei Fei et al. 2004). Food101 (Bossard et al.) 2014), EuroSAT (Helber et al. 2019), RESISC45 (Cheng et al.] (017). and StanfordCars (Krause et al.] 2013). For the OOD tasks, we use the "real" subset of Domain (Peng al. (019) as the training set and the "sketch" subset the testing set, and vice versa. For text tasks, we validate our model on GLUE (Wang et al. 2018 and GLUE-X (Yang
ct al. 2022). for both ID and OOD evaluation.
Baselines. We compare our method with three related approaches: (1) Linear Probing: a single fully connected layer (Linear-Probe (FC)) and a two-layer MLP classifier (Linear- (MLP)); (2) NMTune (Chnn et al. (004): a black-box method that applics singular value docomposition (SVD) to the original pre-trained features and introduces three regularisation strategies to adjust the Singular Value Entropy (SVE) and Largest Singular Value Ratio (LSVR), thereby transferring pre-trained representations into the new feature space and improving the generalisation of pre-trained models on downstream tasks.
Experimental Setting. We train each downstream classifier for 30 epochs using the Adam optimizer. The learning rate is set to 0.001 for the other basclines and 0.0001 for our proposed method. For evaluation, we report the average performance results over five runs.
5.1 Results
Vision tasks. Tables 1 2 show the results comparing our models to other baselines. We use the same datasets and identical train validation-teds test splits, reporting the average accuracy across five runs on each datasct. Overall, our method consistently achieves higher accuracy in in-domain (ID) vision tasks, improving the quality of noisy pre-trained features. When using a single-layer linear network as the classifier, our ap- proach significantly enhances the generalisation ability of pre-trained models on downstream tasks compared
8

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: Performance comparison on ID tasks. Evaluation metries: (1) " "Acc" denotes accuracy; (2) "MPC" denotes mean per class. indicates that Lorem statistically better/wond than the compared method with the best performance by pairwised (-lest at 95% confidence level.
CIFAR-10
CIFAR-100
Caltech101
Food101
EuroSAT
RESISC45
StanfordCars
Model
Tuning
Ace
Ace
MPC
Ace
Aec
Ace
Acc
$1.00
0.42
79.37 1,00
Resnet-50
ML.P
82.36
0.00
50.07 + 0.44
87.08 + 0.90
56.96
0.24
94.71 0.63
$4.28
0.26
46.65
0.33
72.87
NMTune
81.37
0.27
65.73
0.21
$4.8 0.51
52.45 0.39
94.74
0:21
82.56 0.45
12.78
0.20
70.06
82.46
0.13
59.95
0.40
68,23
0.14
0.20 56.75
95.03 0.30
85,18 0.12
48.46
0.22
73.72
LP
89.88 4: 0.32
00.55 0.31
80.68 + 0.36
67.52 0.18
96.50 0.34
86.77 0.20
48.02
0.28
78.13
MLP
NelicienNee-Bee B3
NMTune
91.97 1: 0.15
70.22 0.49
91.71 1: 0.43
69.57 1: 0.28
96.27 0.15
87.43 0.55
58.98
0.26
80.88
onrs
92.69
61.38
0.18
82.45
LP
95.68 1: 0.24
82.31 0.26
93.83 0.32
85.05 1: 0.22
96.53 0.39
87.79 0.52
50.78 1.99
84.57
Swin-T.
MLP
98.42
0.12
84,28 0.19
95.72
0.17
81.23 0.16
97.23 0.39
90.38 1.15
64.19 1.99
87.92
NMTune
82.67 0.23
96.42 0.44
85.00
80.39 0.27
64.27
87.28
ouns
96.48 1. 0.08
84.45 1. 0.17
96.18 1: 0.32
0.11 87.26
97.33 0.17
90.67 1. 0.46
69.72 J. 0.21
88.87
85.03
05.76 0.28
83.84 0.41
73.11
Romeet50-5%%
MLP
91.35
0.09
72.17
0,08
90.29 0.22
63.19 0.19
95.96 0.34
86.65
0.62
49.11 0.52
78.39
NMTune
91.28 0.18
80.87 0:34
87.75 0.89
60.00 0.11
95.88 0.35
85.68 0.27
49.28 0.50
77.12
L.P
84.28
0.26
60.12 0.31
83.38 +: 1.13
56.14 0.18
94.59 0.14
84.02 0.78
34.34 1.24
72.84
Res0et-50-10%
ML.P
91.14
0.00
72.16 0.26
00 0.35
61.98 0.19
0.20
$7.04
49.00
71.80
NMTune
90.96
0.22
69.70 0.22
85,51 + 0.81
59.17 0.21
95.88 0.31
85.89 0.58
19.35 0.61
76.67
0.14 91.30
72.37 0.22
67.99 + 0.27
61.83 0.21
95.68 0.14
88.01 0.41
52.39 +: 0.13
78.51
Table 2: Performance comparison on OOD tasks. Evaluation metric: (1) "Ace" denotes accuracy. indicates that Lorem statistically better/worse than the compared method with the best performance by pairwised t-test 95% confidence level.
DomainNet Real DomainNet Sketch
Avg
Model
Tuning
Acc
Acc
LP
16.67 = 0.60
31.08 0.37
23.88
MLP
19.69 + 0.21
32.03 0.17
25.86
Resnet-50
NMTune
17.61 = 0.31
28.00 0.32
22.81
ours
20.02 + 0.14
32.19 + 0.36
26.10
L.P
23.57 = 0.15
37.40 0.22
30.49
MLP
25.46 + 0.12
40.54 + 0.40
33.00
EfficientNNe-B33
NMTune
23.03 = 0.20
35.10 + 0.33
29.07
ours
25.46 + 0.09
37.70 + 0.26
31.58
LP
38.67 = 0.29
60.50 + 0.48
49.59
MLP
40.38 + 0.41
59.65 0.87
50.02
Swin-L
NMTune
37.48 = 0.31
58.64 0.27
48.06
ours
39.65 0.27
60.34 0.73
50.00
LP
18.02 0.33
33.02 0.77
25.52
MLP
19.91 0.24
36.68 + 0.29
28.30
Resnet-50-5%
NMTune
18.20 = 0.13
34.02 0.43
26.11
ours
19.94 0.15
36.22 + 0.08
28.08
L.P
17.64 = 0.12
32.33 + 0.48
24.99
MLP
19.90 = 0.28
36.05 + 0.22
27.98
Resnet-50-10%
NMTune
17.87 = 0.21
33.30 0.34
25.59
ours
19.92 + 0.33
35.40 0.12
27.66
standard linear probing. out-of-domain (OOD) vision tasks, the performance of our method mixed. Nevertheless, the gap between our method and the best-performing approach small, and it still outper- forms other baselines. The reason for this discrepancy has already been analysed in the discussion section

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3: Performance comparison on GLUE tasks. Evaluation metries: (1) "Acc" denotes accuracy; (2)
"MCC" denotes matthews correlation; (3) "PCC" denotes pearson correlation. indicates that Lorem is statistically better /worse than the compared method with the best performance by pairwised 1-lest at 95%
confidence level.
Model
Tuning
CoLA
MINII
MRPC
QNLI
QQP
RTE
SST
STS
Avg
MOC
Ace
Apc
Ace
LP
38.97 # 1.33
82.50
2: 2.33
70.06
0.65
06.50
3.31
77.07 # 1.32
66.76 #: 1.32
K3.53
3.29
76.27 d: 0.94
62.72
BERT-L
MLP
41.90 + 1.03
32.57
1.80
1.50
69:10
67.83
+ 2.80
60.01 = 0.57
84.18
0.12
0:13
53.93 1.22
55.07 2.01
65.64 0.40
74.57 :: 1.23
0.43
63.91
42.63
de 0.92
W2.50
1.28
70.47
1.221
08.12 0.70
83.22 0.14
56.67 = 1.02
86.93 + 0.9G
70.43 0.80
64.48
LF
06.84
1.19
31.35
3.20
12.07
1:36
70:31 1 2.18
78.37 1: 1.15
$8.05
5.19
87.84 0.26
06.78 2:08
63.95
RoBERT:
NMTone
MLP
7:27 30:15
45.46
2.38
35.85
6.98 35.45
3.42
71:23 0.44
69.26 1.92
72.05 + 0.70
84-48 0.00
0.50
55.07
64.37 2.38
1.56
86.96 + 1.40
85.07 4.02
68.96
68.07 +: 3.52
4.19
63.88
0.41
72.66
84.64
1.77
1.41
LP
17.19:3: 7.65
37.55
22:25
70.41
2.44
67:24 2.32
74.30 0.57
50.04 1.16
83.30 3: 0.30
44.99
55.63
GPT-2
MLP
20.16 + 1.64
37.06
22.13
68.63 1.92
66.68 + 1.55
82.43 + 0.60
51.26
1.84
76.17 + 3.71
58.43 2.15
07.73
NMTTme
2,87
17.02
66.72
2.32
OGIS
22.22 d: 2.60
40.01 d: 17.46
71.59 1.37
08.09 1.37
30.70 2: 1.48
52.42 d: 1.15
81.36 d: 2,00
60.50 d: 1.59
59.44
Table 4: Performance comparison on GLUE-X tasks. Evaluation metrics: (1) * "Acc" denotes accuracy; (2) "MCC" denotes matthews correlation. /o indicates that Lorem is statistically better/ better/worse /woree than the compared method with the best performance by pairwised 1-test at 95% confidence level.
Model
Tuning
GT MCC
IMDB
MNLI-mis Acc
SNLI
SICK
NewaQQA
SciTail Ane
HANs
Avg
Ace
Ace
Ace
Are
Ane
52.77
70.39
53.36
BERT-L
NMTume
MI.P
42:25 2.03
43.87-
72.63 1.59
72
22.12-22.66
58.77
68.76 0.44
0.669
74.39 + 0.25
72.99
0.71
1..81
72.57 +.5++3... 3.54
37.27
57.41 1.24
0.7.6
59.85
00.78
1.46
1.48
48.60 1.01
48.59
1.26
59.56
58.55
OUR
45.45
72.09
1.26
59.29
0.59
74.38
0.55
2.97 77.09
38.33
60.94
50.07
LP
+..41+27733
71.77 2.23
74.00 2.57
70.86 2.29
43.58 +-4.6 4.61
+0.17+3.45
56:47 = 8:25
49.99 4: 0.01
16.47
RoBERTOaL
45.66
NMTTuee
09.81=7.55
59:23
5.59
76.35 a: 2.00
72.67
2.85
50.53 10.84
38.07 1: 1.84
00.43 3.32
48.82 0.855
55.85
47.54
70.65 1.75
77.40 0.78
74.30
1.13
S7.59 4.13
39.71 +: 1.87
62.61 +: 5.81
49,86 1.101
$9.98
41.022- 9.56
50.94
68.43 6.58
58.50 2.50
02.45 0.57
34.84 0.50
55.75 4.19
51.74
GPT-2
MLP
40.51 16.00
$1.05
= 0.25
64.79 1 0.84
63.32 1.78
$7.49 1.:19.1.5 3.10
57.72 1 2.51
55.31 1.45
52.92 1. 2.41
52.89
42.48
57.88
65.13
57.30 3.26
2.64
1.36
43.86
12.88
51.04 0.16
63.70 2,88
65.50
1.10
62.87 2.86
38.64 1: 3.61
58.11 1: 4.50
53.46 1: 1.09
54.63
Ablation study on amplitude residual guidance and model designs are in Appendix A.1 We also present the hyperparameter analysis in Appendix A.2
(Section 4.2
NLP tasks. Tables 3 present the comparison of our method against other baselines on GLUE and GLUE-X tasks. GLUE and GLUE-X can be regarded as the ID and OOD tasks in NLP, respectively. Overall, our proposed method achieves the best average performance across both types tasks, demonstrating its ability to adapt to diverse downstream tasks and dataset distributions, thereby improving the generalisation of pre-trained models. Another finding is that, the MLP classifier also performs strongly in both ID and OOD tasks, ranking just below our method. In contrast, simple linear probing, while occasionally strong on specific tasks, shows unstable performance overall, consistently ranking last in average results. This conclusion differs from that of Chen's work, which we believe may be due to differences in learning rate selection.
6 Related Work
Noisy Supervision. Mainstream approaches can be broadly categorised into the following directions: (1)
Noise Modelling (Van Rooyen et al.
2015 Han et al.
2018a. Van Rooyen & Williamson, 2018 Yao et al.
2019): Directly modelling the label noise generation process to correct the loss during feature learning; (2)
Robust Loss Functions (Liu & Tao 2015 Zhang & Sabuncu 2018 Thulasidasan et al. 2019 Charoenphakdee
ct al. 2019 Lyu & Tsang] 2019 Menon et al. 2020): Modifying the optimisation objective to enable the
model to learn features even under noisy labels; (3) Regularisation (Reed et al. 2014 Azadi et al.) 2015
Zhang et al. 2018 Han et al. 2020): Introducing prior constraints that. the features are less sensitive to
10

---

Published in Transactions on Machine Learning Research (12/2025)
2018b Yu et al, 2019): Using multiple models to supervise each other, thereby reducing the risk of single model being mis- led by noise. In contrast to these methods, our work focuses on mitigating the influence of biased knowledge in pro-trained representations cansed by noisy labels. The most relevant to our work is Noisy Model Tuning (NMTune) proposed by Chen el al. (2024), a black-box fine-tuning method that employs multiple regulari- sation strategies. In Chen's work, the authors compare Singular Value Entropy (SVE) and Largest Singular Value Ratio (LSVR) computed from pre-trained features obtained under different levels of noisy supervision, analysing how noise affects representation learning and generalisation. As the noise ratio increases, LSVR rises and SVE becomes excessively high, indicating that the pre-trained features become more constrained to specific directions and less diverse on new data distributions ultimately reducing transferability. Based on these observations, Chen proposes adjusting the pre-trained feature space to reduce the effect of noise and improve generalisation. We note that there has been relatively little work on understanding the influence of biased knowledge in pre-trained features. Our work approaches this problem from a froquency-domain perspective, examining how different frequency components change as noise levels increase. In contrast to Chen's findings, conclusion is that higher noise levels cause pre-trained features to learn high- high-frequency details and unique texture patterns, rather than concentrating a few specific dominant directions, Thus, our work offers a distinct interpretation of the problem from the trequency-domale viewpoint, providing new insights into this research area.
noisy labels; and (4) Multi-model Learning (Veit et al. 2017 Li et al.
2017 Han et al
Applications of Fourier Transform. Frequency-domain analysis helps neural networks identify and pre- serve key features more effectively. Several studies have provided new insights into explaining the behaviour of neural networks from frequency-domain perspective (Wang et al. 2020 Guo et al. 2020 Chen et al. 2021 Liu ct al. 2021a Yu et al., 2022; Zhou al. 2024). In the visual domain, some rescarch has found that high-frequency componnents play important role in improving the accuracy of CNNs (Wang et al. 2020 Chen et al., 2021). In Chen et al. (2021), the authors observed that CNNs tend to converge to local optima closely related to the high-frequency components of training images. However, the high-frequency components are susceptible to noise and perturbations. Inspired by the phenomenon in the hunan visual system- sestem-where robust recognition relies more heavily on phase information the authors proposed a novel data augmentation method that recombines the phase spectrum of the current image with the magnitude spectrum of distractor image to generate new training samples. This approach encourages CNNs to focus more on structural information (derived from phase) and become more robust to variations in amplitude (such as noise, brightness, and colour distortions). In Huang et al. (2023), the authors investigated the differential impacts of phase and amplitude on CNN robustness, proposing a method to decouple phase and amplitude in certain layers via the Discrete Fourier Transform (DFT) during training, and applying distinct early-stopping strategies to each component, thereby enhancing the network's robustness. In natu- ral language processing (NLP), research has shown that pre-trained large language models implicitly utilise Fourier features when performing arithmetic tasks (Zhou et al.) 2024). MLP and attention layers leverage low-frequeccy and high- frequeeyy Fourier components, respectively, to accomplish tasks, and different pre- training strategics directly influence the model's effective utilisation of these Fourier features. Our work also investigates the impact of noisy labels on pre-trained features from a frequency-domaii perspective. The key distinction from prior work is that we focus on leveraging the observed characteristics to improve gener- alisation in downstream tasks, rather than modifying the pre-trained model itself or improving its training process.
7 Conclusion
In this paper, we propose a novel black-box method to mitigate the impact of noisy labels on the down- stream performance of pre-trained models. The method leverages amplitude residuals to realign the original phase of pre-trained representations, thereby mitigating biased knowledge and improving generalisation. By analysing the performance of amplitude and phase extracted from pre-trained representations trained under varying levels of noisy supervision, as well as the changes in amplitude distribution induced by noisy labels, we gain deeper insights into how noisy labels distort phase, drive models to overfit on irrelevant texture pat- terns, and consequently preserve biased knowledge in pre-trained representations. Our algorithm employs amplitude residuals guidance for phase correction, making the representations more robust. and generalis-
11

---

Published in Transactions on Machine Learning Research (12/2025)
able. Experimental results demonstrate that method outperforms state-of-the-art baseline methods and widely used fine-tuning approaches. We conduct experiments on variety of vision and language pre-trained models, and our method achieves competitive results on both in-domain (ID) and out-of-domain (OOD) tasks. It is important to note that our approach operates in the froquency domain of pre-trained representa- tions. Whether can be exlended to large language models and other foundation models remains an open challenge, which warrants further exploration in future research.
Acknowledgments
This work is partially supported by the Australian Research Council (ARC) Training Centre for Information Resilience (Grant No. IC200100022) and by an ARC Future Fellowship Project (Grant No. FT240100022). We acknowledge the authors of the paper "Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks" for making their pretrained models available, which greally assisted our experimental analysis.
References
Samaneh Azadi, Jiashi Feng, Stefanie Jegelka, and Trevor Darrell. Auxiliary image regularization for deep cnns with noisy labels. arXiv preprint arXi::1111.07060 2015.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101-mint discriminative components with random forests. In European conference on computer vision, pp. 446 461. Springer, 2014.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Llpez-Gazpll and Lucia Specia. Semeval-2017 task 1: Semantic textual mimilarity-muliilngualll and cross-lingual focused evaluation. arXiv preprint 17X8::1708.000 2017.
Nontawat Charoenphakdee, Jongyeong Lee, and Masashi Sugiyama. On symmetric losses for learning from corrupted labels. In International Conference on Machine Learning, pp. 961-970. PMLR, 2019.
Guangyao Chen, Peixi Peng, Li Ma, Jia Li, Lin Du, and Yonghong Tian. Amplitudc-phase recombina- tion: Rethinking robustness of convolutional neural networks in frequency domain. In Proceedings of the IEEE/CVF international conference computer vision, pp. 458 467, 2021.
Hao Chen, Jindong Wang, Ankit Shah, Ran Tao, Hongxin Wei, Xing Xie, Masashi Sugiyama, and Bhiksha Raj. Understanding and mitigating the label noise in pre-training on downstream tasks. Tn The Tweefth International Conference on Learning Representations, 2024.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art Proceedings of the IEFE, 108(10):1865-1888, 2017.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for compulational linguislics: human language lechnologies, volume 1 (long and short popers), pp. 4171-4186, 2019.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference computer vision and pottern recognition workshop, pp. 178 178. IEEE, 2004.
Chuan Guo, Jared S Frank, and Kilian Q Weinberger. Low frequency adversarial perturbation. In Uncertainty in Artificial Intelligence, pp. 1127-1137. PMLR, 2020.
Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. Advances in neural informalion processing systems, 31, 2018a.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advonces in neural information processing systems, 31, 2018b.
12

---

Published in Transactions on Machine Learning Research (12/2025)
Bo Han, Gang Niu, Xingrui Yu, Quanming Yao, Miao Xu, Ivor Tsang, and Masashi Sugiyama. Sigua: Forgetting may make learning with noisy labels more robust. Tn International Conference on Machine Learning, pp. 4006 4016. PMLR, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Tn Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 778, 2016.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep
learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2211-25) 2226, 2019.
Huaxi Huang, Hui Kang, Sheng Liu, Olivier Salvado, Thierry Rakotoarivelo, Dadong Wang, and Tongliang Liu. Paddles: Phase-amplitude spectrum disentangled early stopping for learning with noisy labels. In Proceedings of the IEFE/CVE International Conference on Computer Vision, pp. 16719-16730, 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained catego- rization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of fcatures from tiny images. 2009.
Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In Proceedings of the IEEE international conference on computer vision, pp. 1910-1918, 2017.
Honggu Liu, Xiaodan Li, Wenbo Zhou, Yuefeng Chen, Yuan He, Hui Xue, Weiming Zhang, and Nenghai Yu. Spatial-phase shallow learning: rethinking face forgery detection in frequency domain. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 772 781, 2021a.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEFE Trans- actions on pattern analysis and machine intelligence, 38(3):447 461, 2015.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint crXtc:197..1169. 2019.
Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012 10022, 2021b.
Yueming Lyu and Ivor W Tsang. Curriculum loss: Robust learning and generalization against label corrup- tion. Xiv preprint Xix::1955.10045 2019.
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Can gradient clipping mitigate label noise? In International conference on learning representations, 2020.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer 2019.
vision, pp.
Aleksandra Piktus, Christopher Akiki, Paulo Villegas, Hugo Laurengon, Gérard Dupont, Sasha Luccioni, Yacine Jernite, and Anna Rogers. The roots search tool: Data transparency for llms. In Proceedings of the 61st Annuull Meeting of the Association for Computational Linguistics (Volume 3: System Demonstra- lions), pp. 304 314, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised issu multitask learners. OpenA7 7 blog, 1(8):9, 2019.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks noisy labels with bootstrapping. arXiv preprint orxiv: 141:11 12. 12.6596, 2014.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Daniel Ruderman and William Bialck. Statistics of natural images: Scaling in the woods. Advonces in neural information processing systems, 6, 1993.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenct large scale visual rocognition challenge. International journal of computer vision, 115(3):211 252, 2015.
Eero P Simoncelli and Bruno A Olshausen. Natural image statistics and neural representation. Annual review of neuroscience, 24(1):1193 1216, 2001.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105 6114. PMLR, 2019.
Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd- Combating label noise in deep learning using abstention. arXie preprint arXix:1905.10900 2019.
Brendan Van Rooyen and Robert C Williamson. A theory of learning with corrupted labels. Journal of Machine Learning Research, 18(22)):1 50, 2018.
Brendan Van Rooyen, Aditya Menon, and Robert C Williamson. Learning with symmetric label noise: The importance of being unhinged. Advances in neural information processing systems, 28, 2015.
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie. Learning from noisy large-scale datasets with minimal supervision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 839-847, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1xXiv:180.007401 2018
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric Xing. High-frequeney component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8684-8888 0894, 2020.
Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang, Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective. ar preprint arXiv:2211.08077 2022.
Jiangchao Yao, Hao WI, Ya Zhang, Tvor W Tsang, and Jun Sun. Safeguarded dynamic label regression for noisy supervision. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9103-9110, 2019.
Hu Yu, Naishan Zheng, Man Zhou, Jie Huang, Zeyu Xiao, and Feng Zhao. Frequency and spatial dual guidance for image dehazing. In European conference on computer vision, pp. 181-198. Springer, 2022.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International conference on machine learning, pp. 7164- 7173. PMLR, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
Tianyi Zhou, Deqing Fu, Vatsal Sharan, and Robin Jia. Pre-trained large language models use fourier features to compute addition. Advances in Neural Information Processing Systems, 11f:5120-25153 2024.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Table 5: Ablation study on amplitude residual guidance and learnable template across ID tasks. Evaluation metries: (1) "Ace" denotes accuracy; (2) "MPC" denotes mcan per class.
CIFAR-10
CIFAR-100
Caltech101
Food101
EuroSAT
RESISC45
StanfordCars
Model
Variant
Ace
Ace
MPC
Ace
Acc
Are
Ace
Amplitude guidance Random phase shift
50.26
0.51
0.39
56.51
0.40
94.20
48.31
0.39
72.80
Resuet-50
81.11 0.94
00.39
0.42
87.44 = 0:41
56.61
+ 0.24
94.76
= 0.43
84.64 + 0.72
48.15
0.25
73.16
Fixed phase shift
82.24 +: 0.35
50.13
+: 0.46
87.33
0.40
56.44 +: 0.30
94.39
0.38
84.24 + 0.57
48.26
0.37
73.15
Fixed template ouns
81.44 + 0.68
0.37
88.72
56.63
+ 0.24
94.33
0.48
84.19 0.40
48.14
0.53
82.46 1: 0.13
50.05
a: 0.40
88.23 + 0.14
56.75
a: 0.20
95.00
0.30
85.18 1: 0.12
48.48
0.22
73.72
Amplisude gusdinnce
91.68 1: 0.25
71:29
0.25
91.19
0.42
65.28 1: 0.17
91.88
0.06
56,38 1: 0.58
51.41
0.49
78.87
EfficientNet-B
Random phase shift
91.39 1: 0.22
71.11
1: 0.32
90.43
0.74
65.36
1: 0.20
94.96
0.21
86.43
1: 0.35
51.41
0.24
78.73
Fized plase skift
91.61 + 0.10
71.31
0.24
90:45
0.25
65.57 +: 0.18
91.99
= 0.10
86.39 2: 0.62
51.60
0.33
78.85
Fixed template
71:22
0.40
0.12
65.60
95.08 0.54
0.39
92.19 +: 0.08
72.85 0.14
92.63
0.33
T2.11 + 0.23
96.61 = 0.16
89 89.33 0.19
61.38
0.18
82,
Amplisude guidance
96.40 0.07
84.30 +: 0.38
0.18 94.84
87.09 0.28
97.28 = 0.17
90.1 0.23
09.37 0.49
88.48
Swin-L
Raudom shift
96.16
0.26
0.23
0.71
Fixed phase shift
96.48 0.11
84.38
+ 0.14
96.86 0.13
$7.08 + 0.10
98.18 + 0.19
90.34 + 0.78
00.11 0.66
88.63
Fixed template
96.39 0.03
84.36 0.20
91.68
0.12
87.15
0.24
97.12 0.38
90.
0.46
09.58 0.77
88.51
96.48 + 0.08
84.45 0.11
96.18
0.32
87,26 + 0.11
97.33
0.17
90.67
0.46
00.72 0.21
88.87
A Appendix
A.1
Ablation Study
The ablation study of Lorem is presented here, where we evaluate on the in-distribution (ID) vision datasets. We use three pre-trained vision models-RosVeet550,, EfficientNet-B3. and Swin-L-for this analysis. Wo investigate the effectiveness of amplit ude residual guidance and the learnable template. Specifically, we report results under different model configurations: (1) removing amplitude guidance; (2) applying a random phase shift; (3) using fixed phase shift; and (4) using fixed phase template, as shown in Table From the results,
observe that proposed full model framework indeed improves downstream performance and yields more stable outcomes. In particular, the proposed method significantly outperforms the model variant without amplitude residual guidance on downstream tasks. Moreover, our method achieves notably better results on downstream tasks using EfficientNet-Ba compared to other pre-trained models.
We also conduct an ablation study on model design. Specifically, we report results under different model configurations: (1) using the original phase; and (2) adjusting the semantic amplitude with the residuals, as shown in Table From the results, one can observe that replacing the semantic phase with the original phase significantly impairs downstream performance. The proposed model design which adopts the semantic phase and applics phase adjustment instcad of combining it with semantic amplitude- corissssnntsltll enhances both accuracy and stability across downstream tasks.
A.2 Hyperparameter sensitivity
The lyperparaaeter sensitivity analysis of global scaling factor e in Lorem is presented here, where we evalu- ate on three representative in-distribution (ID) vision datasets, which are CIFAR-10, CIFAR-100 (Krizhevsky) et al.] 2009), and Food 101 (Bossard et al. 2014). We use three pre-trained vision models ResNet-50, EfficientNet-B3, and Swin-L- for this analysis. From the results in the table, choosing E = 0.01 yields the best and most stable downstreem performance across all three pre-trained models.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Table 6: Ablation study on model designs across ID tasks. Evaluation metrics: (1) "Ace" denotes accuracy;
(2) "MPC" denotes mean per class.
Model
Variant
CIFAR-10
CIFAR-100
Caltech101
Food101
EuroSAT
RESISC45
StanfordCars Acc
Avg.
Acc
Acc
MPC
Acc
Acc
Ace
Original plase
82,03 = 0.22
58.21
0.58
86.18 0.32
55.22 0.52
94.75
1: 0.44
8151 0.65
37.23 2 0..3
71.17
Resnet-50
Semanaic amplitade
82.31
94.99
82.48 0.13
59.95
0.10
88.23 0.14
06.75
95.03
+ 0.30
85.18 0.12
18.16
0.22
73.72
Original plasse
91.77 0.166
68.54 0.28
80.0 + 0.74
13.33:00.00
94.62 0.19
84.06 0.44
37.92
= 0.21
75.09
Semantic amplitude
Effinient Net-R3
92.18 0.08
72.86
11366+000 0.14
02.63 + 0.33
72.11 0:223
96.64 + 0.16
89.33 0.19
61.38
0.118
82.46
Original phase
96.42 -: 0.15
83.25
0.18
94.37
0.25
86.37
0.24
07.29
0.12
90.55 0.61
52.91
0.54
85.88
Semantic amplitade
Swin-L
oams
96.48 1: 0.08
84.45 4 0.17
96.18 0.32
87.26 0.11
97.33 1. 0.17
90.67 0.46
69.72
0.21
88.87
Table 7: Hyperparameter sensitivity study on global scaling factor across TD) tasks. Evaluation metrics " "Ace" denotes accuracy.
CIFAR-10
CIFAR-100
Food101
Model
E
Acc
Acc
Acc
0.1
81.58 + 0.76
59.12 + 0.81
56.39 + 0.33
0.01
81.39 0.40
59.19 + 0.28
56.54 + 0.33
Resnet-50
0.001
82.46 + 0.13
59.95 + 0.40
56.75 = 0.20
0.0001
81.66 + 1.02
59.68 + 0.57
56.68 + 0.38
0.1
91.64 + 0.22
70.95 + 0.30
65.85 + 0.21
0.01
91.64 + 0.10
71.49 + 0.14
65.70 + 0.11
EfficientNet-B3
0.001
+.09
72.85 + 0.14
72.11 + 0.23
0.0001
91.56 0.25
+ + 0.14
65.71 + 0.32
0.1
96.32 + 0.12
87.25 + 0.12
0.01
96.31 + 0.20
84.39 - 0.19
87.21 + 0.35
Swin-L
0.001
96.48 + 0.08
84.45 + 0.17
87.26 + 0.11
0.0001
96.31 + 0.17
84.27 + 0.33
87.20 + 0.18
16


Reviews:
[{"review_id": "7MMNgeWyvK", "reviewer": "Reviewer_WkqP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces “Lorem”, a black-box method for mitigating biased knowledge in pre-trained models caused by noisy labels, without requiring access to model parameters or training data. It operates purely on pre-trained representations by manipulating their frequency components—specifically the amplitude and phase derived via Fourier transforms.", "claims_and_evidence_yn": "No", "claims_explanation": "- the method is based on the assumption that amplitude contains more discriminative information; however, it is not necessarily true since there is no theoretical justification; besides, some results in OOD tasks and noise types suggest the opposite and the validity in modern architecture such as ViT and larger dataset (ImageNet-21k) are not assessed.\n- only experiments regarding ResNet is per-trained with noise, while other experiments is not clearly related to the claim that the proposed method could relieve the noisy pre-trained model for downstream tasks.\n- the proposed Lorem is not convincing: basically, it tries recover the information loss caused by $g_{\\theta}$ using amplitude containing more discriminative information; however, it does not imply the necessity of Fourier transforms as the original feature should possess the most information; besides, whether the improvement is caused by the proposal or increasing parameters $\\phi$ is unclear; for example, what if we just simply connect $X$ and $S$ with a residual layer or concatenate them ?\n- this work is based on pure empirical evidence; however, the current results is clearly not enough to support the claim compared to the related work [1], which provide substantial details of different experiment settings; for example, how would a simple regularization such as label smooth in pre-training, the size of $D'$ and the architecture of $g_{\\theta}$ affect the final results is not fully investigated.\n\n***\n[1] UNDERSTANDING AND MITIGATING THE LABEL NOISE IN PRE-TRAINING ON DOWNSTREAM TASKS, ICLR 2024", "audience_yn": "Yes", "audience_explanation": "the task is important as the growing size of pre-trained data", "requested_changes": "see above", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IvQhoDICRf", "reviewer": "Reviewer_bL7T", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the problem of biased knowledge in pre-trained models caused by noisy labels in large-scale datasets. Existing noise-robust learning methods typically require modifying training data or model parameters, which is impractical for massive and inaccessible pre-trained models. \nTo overcome this, the authors propose a black-box denoising framework, named \"Lorem\", which operates directly on pre-trained representations without retraining. The method analyzes feature representations in the frequency domain, finding that amplitude components encode most discriminative information while phase components are more sensitive to label noise. \nBased on this insight, Lorem uses amplitude residuals to guide phase realignment, effectively removing biased knowledge. \nExperiments on multiple pre-trained vision and language models demonstrate that this approach consistently improves downstream performance across both in-domain and out-of-domain tasks.\n\nStrengths:\n\n1. The paper introduces an amplitude-guided phase realignment method (\"Lorem\") for mitigating biased knowledge in pre-trained models without access to training data or model parameters, addressing a critical challenge in large-scale model adaptation.\n\n2. The authors provide solid frequency-domain analysis that distinguishes between amplitude (discriminative) and phase (noise-sensitive) components, offering insightful theoretical grounding for their correction mechanism.\n\n3. The proposed method is extensively validated on multiple pre-trained models (ResNet, Swin, EfficientNet, BERT, RoBERTa, GPT-2) and diverse in-domain (ID) and out-of-domain (OOD) tasks across vision and NLP, showing consistent gains.\n\n4. The algorithm is computationally efficient and training-free for the backbone, making it practical for large-scale, frozen, or inaccessible models, with potential real-world usability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors have conducted extensive experiments on vision and language tasks to show the effectiveness of their method in improving the pre-trained models trained on biased data in downstream tasks.", "audience_yn": "Yes", "audience_explanation": "Biased knowledge in pre-trained models caused by noisy labels in large-scale datasets is an important problem. The author provided an alternative method to address it.", "requested_changes": "Though there are many strengths of the paper, there still remain some critical issues:\n\n1. While empirically effective, the method lacks rigorous theoretical justification for the stability and convergence of the amplitude–phase correction process. The authors fail to present any proof to show why their method can yield effective result as they expected.\n\n2. The improvement on OOD tasks (e.g., DomainNet in Table 2) is modest, and the analysis does not thoroughly dissect when and why performance gains diminish under certain domain shifts.\n\n3. The approach assumes that discriminative information is primarily encoded in amplitude and that phase realignment universally improves generalisation. This assumption may not hold across all model architectures or modalities. As shown in Figure 1, only one set of vision experiments on CIFAR are shown. Besides, there is a significant drop in \"amplitude\" feature as well, while the proposed method keeps the amplitude in A_S without fixing the bias encoded in amplitude term.\n\n4. Although the paper mentions potential extension to large language models, it provides no concrete experiments or adaptation strategy for such settings, leaving generalisation to larger-scale models speculative.\n\n5. There are some minor issues:\n\nA. Some typos exist, for example, empty parentheses in \"different levels () of noisy\", wrong equation in \"X′ = F(F)\" of Eqn. 1.\n\nB. The full name of the proposed method Lorem is not specified.", "broader_impact_concerns": "No Broader Impact Concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rglBgLnTk2", "reviewer": "Reviewer_cjqC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes **Lorem**, a **black-box denoising method** that mitigates **biased knowledge** in **pre-trained models** caused by **noisy labels**. It operates in the frequency domain, using **amplitude residuals** to guide **phase realignment** of representations  without modifying model parameters or accessing training data. The approach is tested on vision and language pre-trained models across in-domain (ID) and out-of-domain (OOD) tasks, outperforming linear probing, MLP, and NMTune baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "1. The claim that amplitude carries more discriminative information and phase is more noise-sensitive is partially validated by experiments on **CIFAR-10/100**. This lacks broader generalization to large scale datasets like **ImageNet**.\n\n2. The finding that noise alters amplitude **globally** rather than only in high frequencies is qualitatively shown, but lacks quantitative metrics to confirm the trend. \n\n3. The paper shows that its method improves accuracy slightly over baselines, suggesting the amplitude-guided phase correction might help. However, it provides no ablation studies to verify this. Without comparisons to variants that remove or change the guidance step, we cannot confirm that the observed gains actually come from the proposed mechanism. Thus, the causal role of amplitude-guided phase correction remains unproven.", "audience_yn": "Yes", "audience_explanation": "Members of the TMLR community, especially those focused on representation quality, frequency analysis, and robustness under noisy supervision, would be interested in the findings, even if the contribution is more empirical and diagnostic than state-of-the-art in performance.", "requested_changes": "1. Please run the amplitude–phase decomposition on ImageNet-pretrained\n    ResNet-50 features (penultimate layer), and if feasible repeat on a\n    standard ViT (e.g., ViT-B/16) to check architecture generality.\n    Report: (a) amplitude-only vs phase-only linear-probe accuracies on\n    ImageNet val (and a small transfer set), and (b) quantitative\n    spectral metrics such as low/mid/high frequency energy fractions,\n    spectral entropy, and a peak-shift index across clean vs\n    synthetically noisy (5–10%) checkpoints. This to establish, at\n    ImageNet scale (and ideally for a transformer too), that\n    discriminative information is predominantly reflected in amplitude\n    while phase is more perturbed by label noise, using both accuracy\n    and spectral statistics rather than descriptive plots alone.\n    \n   \n 2. Please include ablations to verify that the reported gains truly\n    come from **amplitude-guided phase correction** rather than other\n    factors.\n    \n     -   **Remove amplitude guidance:** Apply phase updates without using amplitude residuals.\n        \n    -   **Random or fixed updates:** Replace guided updates with random or constant phase shifts (same step size) to test causality.\n        \n    -   **Template ablation:** Compare fixed vs learnable phase templates to show the importance of adaptivity.\n        \n    -   **Phase source:** Use the original phase (Φₓ) instead of the semantic one (Φₛ) to confirm the design choice.\n        \n    -   **Hyperparameter sensitivity:** Vary ε and test the effect of removing tanh regularization to show stability.\n        \n    -   **Amplitude-only control:** Combine semantic amplitude with original phase to show that phase realignment is necessary.\n    \n3. Please add robustness experiments to verify that the proposed frequency-domain corrections improve stability. Evaluate adversarial robustness (PGD-L∞/L2) and if possible common corruptions (ImageNet-C), and include frequency-restricted attacks to show whether Lorem better resists low- or high-frequency perturbations. These tests will confirm that the claimed frequency alignment also enhances robustness under noise and distribution shifts.\n        \n4. Please report \n    \n    - Error & variability plots (statistical rigor)\n    \n    - Report 5-run mean ± std (already average is reported; add dispersion).\n    \n    - Confidence intervals (95%) or paired t-tests vs strongest baseline per dataset.", "broader_impact_concerns": "No broader concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)