From Pretraining to Pathology: How Noise Leads to Catastrophic Inheritance in Medical Models
Authors: Hao Sun
Zhongyi Han
Hao Chen
Jindong Wang
Xin Gao
Yilong Yin
Venue: NeurIPS 2025 poster ()
Keywords: Catastrophic Inheritance
Pretraining Noise
Distribution Shift
Medical Data Analysis
Primary area: deep_learning

Abstract: Foundation models pretrained on web-scale data drive contemporary transfer learning in vision, language, and multimodal tasks. Recent work shows that mild label noise in these corpora may lift in-distribution accuracy yet sharply reduce out-of-distribution generalization, an effect known as catastrophic inheritance. Medical data is especially sensitive because annotations are scarce, domain shifts are large, and pretraining sources are noisy.
We present the first systematic analysis of catastrophic inheritance in medical models. Controlled label-corruption experiments expose a clear structural collapse: as noise rises, the skewness and kurtosis of feature and logit distributions decline, signaling a flattened representation space and diminished discriminative detail. These higher-order statistics form a compact, interpretable marker of degradation in fine-grained tasks such as histopathology.
Guided by this finding, we introduce a fine-tuning objective that restores skewness and kurtosis through two scalar regularizers added to the task loss. The method leaves the backbone unchanged and incurs negligible overhead. Tests on PLIP models trained with Twitter pathology images, as well as other large-scale vision and language backbones, show consistent gains in robustness and cross-domain accuracy under varied noise levels.

From Pretraining to Pathology: How Noise Leads to Catastrophic Inheritance in Medical Models
Hao Sun', Zhongyi Han'*, Hao Chen", Jindong Wang", Xin Gao Yilong Yin
School of Software, Shandong University *Cannegie Mellon University
Microsoft Research Asia
*Computer Science Program, King Abdullah University of Science and Technology
sunhao_ Sunhao_@mail.. sdu. edu, cn, {zhongyi han, ylyin) edu, en, haoc3dandrew. cmu. edu, jindogg Sang@microsole com, xin. gao@kaust edu.
Abstract
Foundation models pretrained on web-scale data drive contemporary transfer learn- ing in vision, language, and multimodal tasks. Recent work shows that mild label noise in these corpora may lift in-distribution accuracy yet sharply reduce out-of- distribution generalization, an effect known as catastrophic inheritance. Medical data is especially sensitive because annotations are scarce, domain shifts are large, and pretraining sources are noisy. We present the first systematic analysis of catas- trophic inheritance in medical models. Controlled label-corruption experiments expose a clear structural collapse: as noise rises, the skewness and kurtosis feature and logit distributions decline, signaling flattened representation space and diminished discriminative detail. These higher-order statistics form a compact, interpretable marker of degradation in fine-grained tasks such as histopathology. Guided by this finding, we introduce a fine-tuning objective that restores skewness and kurtosis through leaves the backbone unchanged and incurs negligible overhead. Tests on PLIP models trained with Twitter pathology images, well other large-scale vision and language backbones, show consistent gains in robustness and cross-domain accuracy under varied noise levels.
scalar regularizers added the task loss. The method
Introduction
The pretrain-fine fine-tune paradigm (PT-FT) [ is now central to medical artificial intelligence. Rather than train from scratch on small, domain-specinf datasets, practitioners adapt large-scale foundation models (2] that were learned on web-scale images or image-text pairs. PT-FT supports tasks such as disease classification, tumor detection, and report generation, where expert labels are scarce and costly 3 Vision models like CLIP and its medical variant PLIP 66. developed for general visual understanding, now underpin applications in pathology, radiology, and dermatology
[7]. The same pattern holds in biomedical NLP. Models including BioBERT (8), ClinicalBERT 9 and PubMedBERT [10, pretrained on large biomedical corpora, achieve state-of-the-art performance after fine-tuning on clinical named-entity recognition datasets for diseases, drugs, and genes.
Yet the generalization ability of foundation models in real-world medical tasks, especially under distribution shift, remains far from guaranteed. Recent research has explored various fine-tuning strategies to cope with catto--diitribution (OOD) settings, including domain generalization [2]. semi-supervised learning [13,, label imbalance [14), and annotation noise [S] 16]. However, these approaches often assume that the pretrained representations are structurally sound and transferable, premise that becomes fragile under severe distribution shift. In high-stakes domains like medicine.
Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

pre-training is commonly performed on noisy or weakly aligned corpora (e.g., scraped clinical documents, deidentified radiology reports, or loosely paired image-text datasets), where semantic mismatch and label ambiguity are pervasive. While scaling up data is often considered beneficial for generalization [17, . recent findings show that data quality and distributional alignment are more decisive than volume alone [88]. Spurious correlations and structural noise embedded in pre-trained features can propagate to downstream OOD tasks, leading to unexpected degradation. This phenomenon is referred to as catastrophic inheritance [18 [9]. It raises critical concerns for safety when deploying foundation models in clinical environments subject to distribution shift.
Label noise is a direct window into how pretraining noises migrate to downstream tasks. Recently, seminal studies in natural image and general-language benchmarks [88 showed a two-sided effect: a small amount of noise (around five percent) can raise in-domain accuracy after transfer. yet even this mild corruption markedly degrades OOD robustness. Whether the same trade-off exists for medical data remains unknown. How to mitigate the catastrophic inheritance in the medical model has never been explored before. Medical tasks are finer-grained and more sensitive to annotation quality than their natural-image counterparts, and large-scale corpora are noisier. For example, PLIP [6] is pretrained on more than two hundred thousand pathology image-text pairs scraped from Twitter, where many captions are informal, incomplete, or mismatched. Chest-X-ray datasets such as CheXpert 4 and MIMIC-CXR [20] rely on automated report parsing that introduces label ambiguity. Foundation models built on these sources, including MedCLIP variants (7] and BioBERT derivatives [8], inherit this noise but are usually evaluated only in-distribution. Understanding how pretraining noise propagates under clinical distribution shift is therefore an open problem.
This paper presents the first study that system- atically investigates the latent impact of pre- training label noise in medical foundation mod- els, focusing how such noise undermines downstream robustness under distribution shift, While the literature on noisy label learning has proposed numerous techniques to train mod- els robustly on corrupted labels [21] 22 [66. these methods primarily address noise in down- stream supervised datasets. Our setting is fun- damentally different: the label noise originates from the large-scale pre-training stage, where models are trained on noisy image-text pairs or weakly labeled corpora and later deployed in clinical tasks without further modification to their core parameters. Notably, we do not as- sume any label noise in the downstream data; instead, we investigate how noise is inherited from pre-training data under distribution shifts in high-stakes medical applications. The issue is practical, because medical backbones such as PLIP and PubMedBERT are often closed-source, large, and accessed only through frozen checkpoints. Because these models are huge, retraining them not realistic. We usually freeze the backbone and fine-tune only a lightweight head.
Comelyon17
NAM20000
Figure I: Accuracy of downstream medical clas- sifiers under pre-training label noise. We report performance on Camelyon17 and HAM10000 us- ing features from CLIP and ResNet-50 models pre-trained on ImageNet-IK IK and YFCC15M with synthetic label corruption. While the presence of label noise degrade performance, our method (SKD) consistently outperforms the baseline across noise levels and model backbones.
Our study aims answer the following key questions: 1) Influence: Does the noise in pre-training data have an influence on downstream performance in medical settings? 2) Analysis: Why does such influence emerge in the representation space? 3) Mirigation: How can we mitigate this influence through lightweight fine-tuning? We address these questions on large-scale supervised pre-training, followed by adaptation to downstream medical vision and language tasks
Influence:
The label noise in pre-training induces structural degradation in down- stream medical tasks. In Sections 2.1 and 2.2, we conduct controlled experiments with ResNet-50 and CLIP models pre-trained on noisy ImageNet- IK and YFCC15M datasets (3] with label corruption levels ranging from 0% to 30% (0, 5%, 10%, 20%. 30%). We evaluate the resulting models on suite of medical benchmarks on the downstream OOD tasks. Our findings show that even 5% corruption severely impairs robustness and OOD generalization, including Camelyon17, 17, NIHchestXray, and HAM10000, as shown in Figure and Figure 2
2

---

The pre-training noise flattens the representational space by reducing skew- ness and kurtosis. In Section 2.3 we move beyond spectral analysis and focus on higher- order statistics specifccllseiiilllllyy..... skewness and and logit outputs. As noise levels increase, we observe consistent decline in both statistics, reflecting a collapse toward less asymmetric and less peaked distributions. This flatten- ing signals reduced expressiveness in the learned representations, which compromises the model's ability to distinguish fine-grained medical categories. These trends are consistently observed across model types and downstream datasets.
Analysis:
computed over feature embeddings
Mitigation: We propose a distribution-awar fine-tuning strategy that regularizes skewness and kurtosis to counteract representational collapse caused by noisy pre- training. In Section 7 motivated by the observed flattening of the feature space, we design a light-weight fine-tuning algorithm that encourages asymmetry and peakedness in the down- stream representations by explicitly regularizing higher-order statistics. We demonstrate its effectiveness on noisy ResNet-50 and CLIP backbones through extensive evaluations, as shown in Figure T In Section 8 we further apply our method to PLIP and other med- ical foundation models and observe consistent improvements on diverse histopathology benchmarks, highlighting the generality and medical relevance of our approach.
Beyond our core analysis, we argue that this direction is especially important in the medical domain, where pre-trained models are increasingly treated as immutable backbones-either due to their scale, lack of transparency, or access constraints. In such scenarios, fine-tuning often becomes the only controllable lever, and understanding how to correct or compensate for inherited noise is crucial for safe deployment. We believe our findings can inform future work in broader high-stakes settings, including diagnostic support systems, autonomous surgical navigation, and beyond, where noise-induced collapse in representations can have severe real-world implications.
2 Understanding the Label Noise in Pre-trained Models
2.1 Experiments Design
Noisy pre-training datasets. We assume that the supervised pre-training dataset consists of input- label pairs D = {(1;, w4 HEIN of size N with accurate supervisions. In practice, y/ can refer to either a hard label for classification 24 25 or a text description used in contrastive image-text training [23] 51. Due to the scale of web-scale corpora and the high cost of expert annotation-particllaat in domains like medical imaging- seegging--ee-teeee trainininn datasets often contain noisy supervision á that does not accurately reflect the true semantics of input [66] 26). We define such noisy datasets as D (12,, [eN) and denote 7 as the ratio of noisy supervision in D.
Pre-trained models. We adopt standard backbone architectures that serve as the foundation for downstream tasks, composed of feature extractor and projection head. Let fo : X + Fennte the feature extractor parameterized by o, and ge : F + y denote the projection head. We consider two representative pre-training paradigms: (1) fully supervised classification, where is the class label and go is a linear classifier 25 and (2) contrastive learning, where ge aligns image and text pairs via contrastive objectives [ In both cases, we inject synthetic label noise into the pre-training datasets and later evaluate the resulting representations on downstream medical tasks.
Evaluation. To assess how pre-training label noise affects the transferability of learned repre- sentations in realistic clinical scenarios, we exclusively conduct out-of-domain (OOD) evaluation. Following (18), we evaluate the generalization capacity of the pre-trained feature extractor sa under varying levels of synthetic label noise. Given a downstream dataset D' = {( y: y:) [eEMM) we freeze fo and train a C-way linear classifier on top of its extracted features using the standard linear probing (LP) protocol. The linear probing can be viewed as a simple black-box tuning method for pre-trained models that are typically large and difficult or unable to fully fine-tune.
Experiment setup. We use ImageNet- 1K (IN-1K) 24 in fully supervised pre-training and YFCCI5M [3] CLIP pre- pre-training, with ResNet-50
To simulate label noise during pre- training, we uniformly flip the ground truth class label into the other classes IN-IK and randomly swap the text description from another image-text pair in YFCC15M. The noise ratio 7 is set to (0%, 5%, 10%, 20%, 30%). where 7 = 0% corresponds to clean pre-training. For downstream evalu- ation, we focus on clinically relevant OOD tasks using Camelyon17, 17, HAM10000, and NIHChestXray,
25
3

---

200
274
20hh
150 155
Figure 2: Average evaluation results of ImageNet-IK (IN-IK) fully supervised pre-training on downstream tasks with various percentages of data using ResNet-50. The robustness performance constantly decreases once noise is introduced in pre-training.
covering histopathology, dermatology. and thoracic radiology. We pre-train models on the clean and noisy variants of the above datasets, and then evaluate the standard test sets, which are fixed across all pre-training settings. We apply linear probing (LP) on top of frozen representations adaptation method and report accuracy on each OOD task. To ensure consistency, we use the same data split, augmentation, and training protocol across all settings. Additional implementation details are provided in AppendixB
lightweight
This choice of pretraining datasets reflects common practice in medical transfer learning. Due to the scarcity and privacy constraints of labeled medical data, foundation models used in clinical applications, such as PLIP [6] or MedCLIP 77.. are typically pretrained natural image-text corpora before being adapted to medical tasks. Even when medical-scale pretraining data exists (e.g.. Iaaage-caption pairs scraped from social media), it is rarely open-sourced and often suffers from label ambiguity, weak alignment, or platform-specifee bias. PLIP, for instance, is trained on over 200,000 pathology image-texx text pairs from Twitter, but the raw dataset is unavailable and difficult to verify.
2.2 Results: Noisy Pre-training Impairs OOD Performance in Medical Tasks
Figure Figure[prreeents accuracy trends on Camelyon17, HAM10000, and NIH ChestX-ray'l 14 using ResNet- 50 pretrained on ImageNet-IK K with varying noise levels. All models are evaluated via linear probing on frozen representations. Several consistent and task-specific patterns emerge. First, even mild pre-training noise (5% 10%) leads to measurable degradation across all benchmarks, highlighting the sensitivity of medical transfer performance to upstream supervision quality. Second, the severity of degradation varies by task: Camelyon17 exhibits the sharpest decline, likely due to its fine-grained tissue structures and subtle class boundaries. In contrast, NIH ChestX-rayll 14 shows a flatter curve, possibly due to coarser visual categories and greater label redundancy. Third, the performance gap between clean and noisy pretraining widens as more downstream data is used, suggesting that early-stage noise imposes ceiling that cannot be overcome by fine-tuning alone. These results suggest that label noise in pretraining introduces persistent structural damage in the learned features.
2.3 Feature Space and Logit Space Analysis
To understand how label noise in pre-training affects the structure of learned representations, we conduct empirical analysis of higher-order statistical moments. Specifically, compute skewness and kurtosis over feature embeddings and logit outputs on downstream medical datasets. These statisties provide fine-grained view of the representational geometry, offering insights beyond spectral norms or singular values.
Take the feature space dataset D' = {2,, y:) hie[M]. we extract the feature matrix E R M from the frozen backbone Sa. We then compute the skewness and kurtosis of each feature dimension E (1,..., D) over all M samples, where F.y denotes the j-th feature dimension.
example (the same analysis applies to logit space). For each downstream

---

(a) Camelyon17 Feature Space
(b) HAM11000 Feature Space
[c) Logit Space/ Entropy
0.100
-125
Jiber
100 (755)
0.12
1120
126
123
1130
-02
3
-30
0115
1230
55 105
23
1205
0110
126
Figure 3: Effect of pre-training label noise on representation statistics and prediction uncer- tainty. (a) and (b): As the pre-training noise ratio increases, both skewness and kurtosis of the feature embeddings on Camelyon17 and HAM10000 decrease sharply, indicating a structural flattening of the learned feature space. This collapse is especially evident at low to moderate noise levels (5%-10%). (c): Logit entropy on downstream tasks increases monotonically with noise, suggesting higher predictive uncertainty and reduced confidence. Together, these trends highlight the degradation of representational quality and robustness due to label noise in pre-training.
Definition (Feature-wise Skewness). The skewness of feature dimension is defined as:
M
Fij P15-11} 3
Skew(F..)) (M - (M(M 2)
(1)
where μi मे ΣM Pig and oj V D. (Pis - n2)² denote the mean and standard
deviation of F.g. respectively.
Definition 2 (Feature-wiz Kurtosis). The kurtosis of feature dimension j is defined as:
( +)
Fis 13
3(M 1)² (M - (M - 3)
(F.)(F (M - 1)(M 2M 3)
Σ
which quantifies the peakedness and tail heaviness of the distribution.
Analysis. We report the average skewness and kurtosis across feature dimensions to characterize the global shape of learned representations. As shown in Figure both statistics consistently decline with increasing pre-training noise. This trend reflects a structural flattening of the feature space: representations become more symmetric and less peaked. It is indicative of reduced expressiveness and discriminability. In histopathology, where lesion patterns are compact and localized, clean pretraining naturally yields feature distributions with higher skewness and kurtosis. Noise disrupts this structure, degrading the model's ability to distinguish subtle visual cues.
We observe a similar trend in logit space: entropy increases monotonically with noise, suggesting greater predictive uncertainty. Together, these results provide a statistical explanation for the accuracy drop reported in Section 8 They further establish skewness and kurtosis as effective indicators of structural collapse under noisy pre-training. In the next section, we propose fine-tuning strategy that explicitly restores these statistics to recover robustness.
3 Mitigating the Noise with Regularization on Distributional Shape
In this section, we introduce simple and lightweight fine-tuning strategy that restores key distribu- tional properties of the leamned representations. As shown in Section 2.3 pre-training noise reduces the skewness and kurtosis of downstream feature distributions, indicating collapse in asymmetry and peakedness. We hypothesize that preserving these higher-order moments during adaptation can improve generalization, particularly in medical tasks where subtle, localized patterns are critical. Unlike prior work that focuses spectral properties such as singular value entropy [18], our method directly regularizes skewness and kurtosis relative to a clean reference, capturing distributional structure more relevant to clinical signals. Empirically, we find that while [18) improves accuracy on Camelyon17 17 by only 1.93%, our method achieves a 6.51% 1% gain, highlighting its effectiveness in recovering fine-grained discriminative features in medical domains.
5

---

Pre-training
No access
kuvasz
Learnable
Kite
fe
Tead
LcE
Frozen
Black-hox
Inaccessible Noisy Pre-training Data
Back
Transfer
propagation
malignant
CE
benign
MIP
Head
trained fo
+
L
Lskew
Luurt
Downstream Data
SKD (ours)
Overview of the proposed framework. In the pre-training stage (top), backbone fa is trained on large-scale but inaccessible and noisy web data (e.g., ImageNet-IK or YFCC15M), resulting in potential label corruption. This pre-trained model is then transferred to downstream medical tasks (bottom), where only frozen features are accessible due to black-box constraints. Our method (SKD) introduces lightweight MLP and classifier head on top of the frozen encoder. During fine-tuning, we apply three losses: (I) standard cross-entropy LCE, (2) skewness and kurtosis regularization Lskew Lut to restore feature asymmetry and peakedness, and (3) disagreement loss Lai to sharpen output margins.
Figure 4:
3.1 Method
Let fz( denote the frozen pre-trained backbone and Da the classification head. As shown Figure we introduce a lightweight MLP between (z) and ge. yielding transformed features R Rx for mini-hatch of size B and feature dimension D. These transformed features serve the basis for both prediction and structural regularization.
Skewness Regularization. For each feature dimension j (1,..., D}, we compute the skewness
of F as:
B
Fig PJ J
3
Skew Skew(F.,)) = 13-1)(B 2)
1m1
where M3 = B DE= Fr, and oj 8-1 E. deviation of the j-th feature dimension. The skewness loss penalizes deviation from a target skewness Ta
(Fi, 22
denote the mean and standard
Lskew
1
Σ (Skew(F..)) -ral-
(4)
D
Kurtosis Regularization. The kurtosis of each feature dimension is computed
B(B 1)
Fi Mi
3(B - 1)2
Kurt(F..)) (B 2B
Σ
(5)
- 2 3)
and the corresponding loss is defined by deviation from target T3:
Lkout
1 D
Σ (urt(F..) -Tl-
(6)
TN and 7ı: are target statistics that can computed from a clean model or empirically set to enforce non-degenerate representation structure.

---

Disagreement Regularization. Due to the low dimensionality of the logit space, computing skewness and kurtosis on go(F) unstable. Instead, we adopt a disagreement-basse based loss that enhances output separability by enlarging the margin between the ground-truth logit and the average of incorrect logits. Let (x)) == go F)) be the logit output for input x, and let y be the ground- class. We define:
1 1
Cas( v)
log 1 exp exp h(z), log?
M h(T)))
DJ
BFY
where Y is the label set and h())g denotes the logit score for class 1 +
Overall Objective. The final loss combines task supervision with the three regularization terms:
C Lusk XgLakew + Ag + Ad
where A, Ab. and Xa are hyperparameters balancing each term. We set A = 0.1, A = 1, and A 0.1.
Discussion. This strategy explicitly preserves higher-order structural signals in the feature and output spaces, countering the flattening effect of noisy pre-training. It requires changes the pre-trained backbone and applies to frozen or partially fine-tuned settings, making it well-suited for adapting black-box foundation models in medical domains.
3.2 Evaluation on Noisy Medical Pre-training
We evaluate the effectiveness of our proposed SKD on noisy pre-trained models using downstream medical tasks. We compare our method against standard linear probing (LP) and NML [9]].
We consider models pre- pre-trained with various levels of synthetic label noise on ImageNet-1K 1K and YFCC15M, and assess their performance on medical OOD benchmarks, as introduced in Section 2.1 In Table [ plot the average classification accuracy across these datasets. As in prior findings, LP performance drops steadily with increasing noise. While NML demonstrates benefits on general natural image tasks, its effectiveness does not transfer well medical datasets. Due the unique distributional properties of medical data, NML can even introduce adverse effects in certain tasks.
In contrast, incorporating our skewness and kurtosis regularization significantly boosts generalization performance, recovering performance close to or exceeding the clean model baseline. These results support our core claim: that representation collapse under pre-training noise is not soicly function of model size or task loss, but a structural issue in distributional shape. By explicitly regularizing able reshape the learned features toward more expressive, asymmetric,
skewness and kurtosis, and high-peaked distributions, resulting in improved OOD robustness across diverse medical domains.
Experiments
We further validate SKD on practical large-scale vision and language models that are pre- pre-trained on noisy data, and discuss the noisy label learning and running time analysis in this section.
4.1 Vision Models and Datasets
Setup. To further evaluate our method in real-world medical scenarios, we conduct experiments on PLIP 6). vision-language model pre-trained on over 2.3 billion noisy image-text pairs from the web. We use the four official datasets included in PLIP's release for downstream evaluation: Kather colon 27 PanNuke 28). DigestPath (29). and WSSS4LUAD [30] These datasets cover diverse pathology classification tasks with varying levels of granularity. For comparison, we include the Zero-Shot and LP(origin) baselines reported in the original PLIP paper, as well as our implementation of linear probing (LP), the recently proposed NML [8], and our method SKD.
Results. Table 2 reports the F1 and accuracy scores across all datasets. Our method (SKD) con- sistently achieves the best performance, surpassing both the baseline and NML across the board. Notably, SKD improves the Fl score from 0.931 (NML) to 0.959 on Kather colon, and from 0.948 to 0.956 on PanNuke. These gains are observed even when the original PLIP model achieves strong zero-shot performance, demonstrating that structural regularization remains beneficial during fine- tuning. The consistent improvements highlight the robustness of SKD against inherited noise in web-scale pre-training, particularly in pathology tasks that demand fine- iin-grrined discrimination.
7

---

Table 1: Classification accuracy under different pre-training noise ratios on three downstream medical dataset. SKD our proposed method. NML is the previous SOTA. Bold indicates the best performance under each noise level.
Pretrained
Dataset
Method
0 0%
5%
10 %
20 %
30%
Avg Gain
LP
83.94
81.88
81.71
81.06
80.18
Camelyon17 17
GCE
83.12
82.74
82.21
81.57
80.68
0.31
NML
80.61
84.21
83.32
84.68
85.61
1.93
SKD
89.48
89.59
89.50
86.08
86.65
6.51
LP 49.36
47.78
45.88
45.07
45.26
CLIP
HAM10000
GCE
50.54
48.44
47.20
45.89
45.98
0.94
NML.
50.76
51.35
49.03
52.78
50.04
4.12
SKD
55.19
55.88
55.75
53.32
49.84
7.33
44.75
42.00
42,78
41.58
41.75
ChestX-ray
GCE
45,42
42.71
43.11
42.06
42.13
0.51
NML
36.02
35.71
35,92
36.58
37.19
-6.29
SKD
45.92
45.81
45.48
45.45
45.86
3.13
LP
91.16
90.73
89.24
88.57
88.18
Camelyon17
GCE
91.11
91.43
89.48
89.12
88.64
0.38
NML
89.27
92.44
88.09
90.51
91.29
0.74
SKD
91.76
92.50
91.39
89.12
89.02
1.18
LP 55.69
54.62
54.52
54.49
53,88
ResNet50
HAM10000
GCE
55.73
54.79
54.52
54.51
54,46
0.16
NML
54.71
55.16
54,21
54.77
50.67
-0.74
SKD
58.25
57.54
56.94
58.37
57.54
3.09
LP 41.31
35.94
40.77
39.25
36.12
ChestX-ray
GCE
41.37
36.28
41.23
40.02
36.67
0.44
NML
38.94
36.40
36.55
37.38
38.79
-1.07
SKD
44.81
43.37
44.22
44.25
45.39
5.73
4.2 Language Models and Datasets
Setup. We evaluate the robustness of our method on 32 biomedical named entity recognition (NER) datasets spanning various subdomains, including disease, gene, and chemical recognition. Due to space constraints, we report representative results on five widely- benchmarks Table with full results deferred to Appendix C We fine-tune PubMedBERT [10] as the base model under three settings: (1) standard fine-tumie tne-tuning (Baseline), (2) noise- training using the NML framework. and (3) our proposed SKD method. All models are trained on clean downstream data, isolating the impact of noisy pre-training. Evaluation conducted using both Fl score and accuracy metrics.
Results. As shown in Table 3 our method SKD consistently outperforms both the standard Baseline and NML across all five representative datasets, demonstrating superior robustness and discriminative capability under noisy pre-training. For example, on BC2GM, SKD achieves an Fl of 0.9459 and accuracy of 0.9501, improving significantly over Baseline (0.9053/ 0.9222). Notably, while NML improves over Baseline in some cases (e.g., BC4CHEMD), it fails to deliver consistent gains and even underperforms in others, likely due the domain specific challenges of biomedical NER. In contrast, SKD provides stable improvements by regularizing internal distributions, which helps preserve structural integrity even under distribution shift.
4.3 Additional Experiments on Large-Scale Foundation Models
To assess scalability beyond the backbones used in our main study, we further evaluate SKD on large-scale vision and language foundation models, aligning with the decision feedback.
We evaluate the proposed SKD regularization on VIT-L 31 backbone pre-trained on ImageNet-21K with synthetic noisy labels. SKD consistently improves robustness across all medical benchmarks, achieving 93.4% Camelyon17, 60.0% on HAM10000, and 45.9% on NIH ChestXray,
Vision.

---

Table 2: Real-worid evaluation on PLIP using its original medical datasets. SKD consistently outperforms baselines across F1 and accuracy.
Table 3: Real-world evaluation on biomedi- cal NER tasks using PubMedBERT across five datasets. SKD consistently improves both F1 and aceuracy over LP and NML, demonstrating its effectiveness beyond medical imaging.
Model
Dataset
Method
F1
Accuracy
Zero-Shot
0.565
LP(origin)
0.877
Dataset
Method
F1
Accuracy
Kather colon
0.899
0.895
NML
0.931
0.929
I.P
0.9053
0.9222
SKD
0.959
0.959
BC2GM
NML
0.9187
0.9271
SKD
0,9459
0.9501
Zero Shot
Zero-Shot
0.656
LP(origin)
0.902
LP
0.9330
0,9355
PunNuke
LP 0.930
0.930
NCBI-disease- IOB
NML
0.9378
0,9402
NML
0.948
0.948
SKD
0.9459
0,9471
PLIP
0.956
0.956
LP
0.8895
0.8825
Zero-Shot
LPforigin)
0.856
JNIPRA
NML.
0.9032
0.8957
DigestPath
LP 0.968
0.968
SKD
0.9211
0,9128
LP 0,9373
0.9485
SKD
0.976
0.969
BCACHEMD
NML
0,9506
0,9567
Zero-Shot
0.734
SKD
0.9706
0.9725
LP(origin)
0.927
WSSS4LUAD
0.952
0.9286
0,9401
NML
0.956
0.956
BioNLPI IEPI-TOB
NML
0,9362
0.9426
SKD
0.958
0.958
SKD
0.9481
0,9492
outperforming LP, GCE, and NML in every case. Full results and training configurations are provided Appendix D.I.I
Language. Beyond PubMedBERT, we evaluate SKD on GPT-2 for biomedical NER (AnatEM, BC2GM, BC5CDR-chem,
BioNLPO9). SKD consistently matches or surpasses strong baselines, reaching 93.9% (AnatEM), 91.8% (BC2GM), and 95.7% (BC5CDR-chem) with robust improvements over LP/GCE/NML. Detailed tables are in Appendix D.1.2
4.4 Robustness under Structured Noise
In addition to uniform random noise, real-world medical data often exhibit structured or semantic label corruption (e.g.. hierarchical mislabeling within taxonomies co-occurrence-basse confusion). To better reflect such conditions, we follow the reviewers' suggestion and construct a faxonony-awan hierarchical noise setting.
We use ImageNet as the base taxonomy and synthetically replace controlled percentage of image labels with their WordNet hypernyms (e.g.. "Labrador Retriever" - "dog"), simulating realistic structured mislabeling commonly observed annotation pipelines. We then train ResNet-50 model from scratch under different hierarchical noise levels (0-30%) and compare four methods: L.P (standard linear probing), GCE [33], NML (spectrum preuuuuvrre baseline), and our proposed SKD. All hyperparameters follow the random-noise setting for fair comparison.
As shown in Table D.2 SKD remains consistently robust under structured noise. On HAM10000, SKD improves over NML and GCE at every corruption level, e.g., +1.3% at 1% noise and +0.4% 4% at 30%. This demonstrates that skewness kurtosis regularization effectively mitigates representation collapse even when the label corruption is semantically structured. Full tables and visualization examples are provided Appendix D.2
4.5 In-depth Analysis
Ablation study. We conduct controlled ablations to isolate the effects of Lskew, Lxan, and Cais in SKD. Results on Camelyon17 show that removing Lxum leads the largest performance drop (-1.07% on average), followed by Likew (-0.47%) and Las (-0.35%). These findings suggest that each term contributes to structural preservation, with kurtosis regularization playing a particularly important role. Full results provided in Appendix B

---

Hyperparameter sensitivity. We vary the weights of Lakw Luun, and Cais across several magnitudes and observe that SKD remains stable throughout. On Camelyon17, the largest performance fluctuation under weight variation is only 1.12% for Ldis. while Likew varies by just 0.54%. These results indicate that SKD robust to loss loss-weight choices, and coarse grid search is sufficient.
In addition, we analyze the sensitivity of the target hyperparameters тa and Tke which specify the desired skewness and kurtosis of the feature distribution. Varying Tx and ты within [-3, 3] on HAM10000 changes accuracy by less than 1%, indicating that SKD is insensitive to the exact target values. Hence, default settings (T,=0, 00, T2 =0) are sufficient for stable performance. Detailed results are provided in Appendix D.3
5 Conclusion
This paper presents the first study of catastrophic inheritance in medical foundation models, where label noise in pretraining leads to structural collapse in learned representations and harms downstream robustness under distribution shift. Through controlled experiments across supervised and contrastive pretraining settings, we demonstrate that even mild pretraining noise causes consistent degradation in OOD generalization, particularly in high-resolution, fine-grained medical tasks. We trace this degradation to flattening of feature and logit distributions, reflected in lower skewness and kurtosis. To counter this effect, we propose a lightweight fine-tuning strategy that regularizes higher- order moments in both feature and output spaces. Our method introduces no architectural changes and applies to frozen or partially tunable backbones, making it compatible with real-world black-box medical models. Extensive results show that this approach restores representational sharpness and improves accuracy across diverse domains and modalities. Our findings highlight the importance of distribution-awert aware adaptation for safe deployment of foundation models in clinical settings.
Limitation
This work primarily investigates the effect of synthetic label noise in supervised and contrastive pretraining, focusing on its structural impact on downstream medical tasks. While our findings consistent across multiple datasets and domains, the study is limited in two respects. First, all experiments are conducted using mid-sized models such as ResNet-50 and ViT-B/32 due to computational constraints. While these are representative, scaling high-capacity architectures like ViT-L or LLaVA-Med remains unexplored. Second, the noise model assumes uniformly random corupption, which may underestimate the complexity of real-world noise such as co-occurrence bias or semantic drift. Extending the analysis to more realistic noise sources and larger model regimes will be important fully characterize catastrophic inheritance in the wild.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Natural Science Foundation of China under Grants U23A20389 and 62176139, and by the Qilu Young Scholars Program.
References
[1] Kornblith, Shlens, and Q. V. Le, "Do better imagenet models transfer better?" in IEEE Conference Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation IEEE, 2019, pp. 2661-2671. [Online]. Available:
CVPR 2019hhmm/KKrmblitt Do Better Laaerraaaaneeeeaaeeeeeeee Models Transfer Better CVPR 201l__001_____ 1al19_pape.htmll
[2] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill al., "On the opportunities and risks of foundation models," arXiv e-prints, pp. arXiv-2108, 2021.
[3] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. Sánchez, survey on deep learning in medical image analysis," Medical image analysis, vol. 42, pp. 60-88, 2017.
10

---

[4] J. Irvin. P. Rajpurkar, M. Ko, Y. Yu, S. Churea- C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya et al., "Chexpert: large chest radiograph dataset with uncertainty labels and expert comparison," in Proceedings of the AAAI conference on artificial intelligence, vol. 33, no. 01, 2019, pp. 590-597.
[5] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, Clark et al., "Learning transferable visual models from natural language supervi- sion," in International conference on machine learning. PmLR, 2021, pp. 8748-8763.
[6] Z. Huang, F. Bianchi, M. Yuksekgonul, T. J. Montine, and J. Zou, "A visual-Lagguage foundation model for pathology image analysis using medical twitter," Nature medicine, vol. 29, no. 9, pp. 2307-2316, 2023.
[7] Z. Wang, Z. Wu, D. Agarwal, and J. Sun, "Medclip: Contrastive learning from unpaired medical images and text," in Proceedings of the Conference on Empirical Methods in Natural Language P Processing. Conference on Empirical Methods in Natural Language Processing, vol. 2022, 2022, p. 3876.
[8] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, "Biobert: pre-trained biomedical language representation model for biomedical text mining," Bioinformatics, vol. 36, no. 4, pp. 1240- 2020.
[9] E. Alsentzer, J. Murphy, W. Boag, W.-H. Weng. D. Jindi, T. Naumann, and M. McDermott, "Publicly available clinical bert embeddings." in Proceedings of the 2nd clinical natural language processing workshop, 2019, pp. 72-78.
[10] Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, and H. Poon, "Domain-specific language model pretraining for biomedical natural language processing," " ACM Transuctions on Computing for Healthcare (HEALTH), vol. 3, no. 1, pp. 1-23, 2021.
[11] P. W. Koh, S. Sagawa, H. Marklund, M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Ya- sunaga, R. L. Phillips. I. Gao et al., "Wilds: A benchmark of in-the-wild distribution shifts," in International conference on machine learning. PMLR, 2021, pp. 5637-5664.
[12] I. Guirajani and D. Lopez-Paz, "In search of lost domain generalization." arXiv preprint arXiv:2007.0144 2.400444..144, 2020.
[13] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, "Fixmatch: Simplifying comi-superviee learning with consistency and confidence," Advances in neural information processing systems, vol. 33, pp. 596-608, 2020.
[14] B. Kang, Xie, M. Rohrbach, Z. Yan, A. Gordo, J. Feng, and Y. Kalantidis, "Decoupling representation and classifier for long-tailed recognition," arXiv preprint arXiv: 1xXvv9190000921,, 2019.
[15) E. Arazo, D. Ortego, P. Albert, N. O'Connor, and K. McGuinness, "Unsupervised label noise modeling and loss correction," in International conference on machine learning. PMLR, 2019, pp. 312-321.
[16) C. Northeutt, L. Jiang, and I. Chuang, "Confident learning: Estimating uncertainty in dataset labels," Journal of Artificial Intelligence Research, vol. 70, pp. 1373-1411, 2021.
[17] J. Kaplan, S. McCandlish, Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei, "Scaling laws for neural language models," arXiv preprint arXiv:2001.083311.00 2020.
[18] H. Chen, J. Wang, A. Shah, R. Tao, H. Wei, X. Xie, M. Sugiyama, and B. Raj, "Under- standing and mitigating the label noise in pre-training on downstream tasks," arXiv preprint 2r09v 17002, 2023.
[19] H. Chen, B. Raj, X. Xie, and Wang, "On catastrophic inheritance of large foundation models," arXiv preprint arXiv:2402.01999 2024.
11

---

[20] A. E. Johnson, T. J. Pollard, N. R. Greenbaum, M. P. Lungren, C.yy Deng, Y. Peng. Z. Lu, R. G. Mark, S. Berkowitz, and S. Horn. "Mimic-cxr-jpg, large publicly available database of labeled chest radiographs," arXiv preprint arXiv:/901.0700 2019.
[21] A. Ghosh, H. Kumar, and P. S. Sastry, "Robust loss functions under label noise for deep neural networks," in Proceedings of the AAAI conference on artificial intelligence, vol. 31, no. 1, 2017.
[22] J. Li, R. Socher, and S. C. Hoi, "Dividemix: Learning with noisy labels as semi-supervised learning," " arXiv preprint arXiv:2002.07394 2020.
[23] B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, Ni, D. Poland, D. Borth, and L.-J. Li, "Yfec The new data in multimedia research," Communications of the ACM, vol. 59, no. pp. 64-73, 2016.
[24] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., "Imagenet large scale visual recognition challenge," International journal of computer vision, vol. 115, no. 3, pp. 21 211-252, 2015.
[25] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition." " in Proceedings of the IEEE conference computer vision and pattern recognition, 2016, pp. 770-778.
[26] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman et al., "Laion-5b: An open large-scale dataset for training next generation image-text models," Advances in neural information processing systems, vol. 35, pp. 25 278 25 294, 2022.
[27] J. N. Kather, J. Krisam, P. Charoentong, T. Luedde, E. Herpel, C.-A. Weis, T. Gaiser, A. Marx, N. A. Valous, D. Ferber et al., "Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study," PLoS medicine, vol. 16, no. 1, p. e1002730, 2019.
[28] J. Gamper, N. Alemi Koohbanani, K. Benet, A. Khuram, and N. Rajpoot, "Pannuke: an open pan-cancer histology dataset for nuclei instance segmentation and classification," in European congress on digital pathology. Springer, 2019, pp. 11-19.
[29] Q. Da, X. Huang, Z. Li, Y. Zuo, C. Zhang, J. Liu, W. Chen, J. Li, D. Xu, Z. Hu el al., "Digestpath: benchmark dataset with challenge review for the pathological detection and segmentation of sngestive-systemy Medical image analysis, vol. 80, 102485, 2022.
[30] C. Han, X. Pan, L. Yan, H. Lin, B. Li, S. Yao, S. Lv, Z. Shi, J. Mai, J. Lin al., "Wsss4luad: tissue semantic segmentation for lung adenocarcinoma," 2022.
Grand challenge on arXiv preprint
[31] A. Dosovitskiy, "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv: 19929, 2020.
[32] A. Radford, J. Wu, Child, D. Luan, D. Amodei, I. Sutskever et al., "Language models are unsupervised multitask learners," OpenAl blog, vol. 1, no. 8, p. 9, 2019.
[33] Z. Zhang and M. Sabuneu, "Generalized cross entropy loss for training deep neural networks with noisy labels," " Advances in neural information processing systems, vol. 31, 2018.
[34] W. Li, Q. Wang, P. Zhao, and Y. Yin. "Knn transformer with pyramid prompts for few-shot learning," " in Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 1082-1091.
[35] F. Wang, Z. Han, X. Liu, Y. Yin, and X. Gao, "CTPT: continual test-time prompt tuning for vision-language models." " Pattern Recognit. vol. 161, p. 11 111300, 2025. [Online]. Available:
1011 202tcoog.222.1111 2024.111300
[36] W. Li, Q. Wang, X. Meng, 7. Wu, and Yin, "Vt-fsl: Bridging vision and text with lmm for sew-shot learning," arXiv preprint arXiv:2509.25033 2025.
12

---

[37] F. Wang, Z. Han, Z. Zhang, R. He, and Y. Yin, "MHPL: minimum happy points learning for active source free domain adaptation." " in JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. IEEE, 2023, Pp- 20 11000-20000 018.. [Online]. Available: https//doi.org/// /ttp://.....0.0...0.o////2229990
[38] X. Liang, X. Li, F. Li, J. Jiang, Q. Dong, W. Wang, K. Wang, S. Dong, G. Luo, and Li, "Medfilip: Medical fine-graine fineggrained tngguae-image pre-training"" IEEE Journal of Biomedical and Health Informatics, 2025.
[39] C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie, "Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis." in Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 21 372-21 383.
[40] J. Chen, D. Yang, Y. Jiang, Y. Lei, and L. Zhang, "Miss: A generative pre-training and fine- tuning approach for med-vqa," in International Conference on Artificial Neural Networks. Springer, 2024, pp. 299-313.
[41] P. Chambon, C. Bluethgen, C. P. Langlotz, and A. Chaudhari, "Adapting pretrained vision- language foundational models to medical imaging domains," arXiv preprint arXiv:2210.04133, 2022.
[42] F. Wang, Z. Han, Y. Gong, and Y. Yin, "Exploring Nomain-invariant parameters for source free domain adaptation." in IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. IEEE, 2022, pp. 7141-7150. [Online]. Available: https://doi.orr orgg/
WWDPCCVPR52688.22220077777
[43] H. Chen, Z. Wang. R. Tao, H. Wei, X. Xie, M. Sugiyama, B. Raj, and J. Wang, "Impact of noisy supervision in foundation model learning," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[44] Y. Chang. Y. Chang, and Y. Wu, "Ba-lora: Bias-alleviating low-rank adaptation to mitigate catastrophic inheritance in large language models," arXiv preprint arXiv:2408. 04556, 2024.
[45] M. Li, H. Chen, Y. Wang, T. Zhu, W. Zhang, K. Zhu, K.-F. Wong, and J. Wang, "Understanding and mitigating the bias inheritance in llm-basee data augmentation on downstream tasks," arXiv preprint Sili:2502.04412 2025.
13

---

A Related Work
Paradigm in the Medical Domain. The pretrain-finctune (PT-FT) paradigm [ 34 [5 has become central to medical artificial intelligence, allowing models to leverage large-scale pretraining general web-scale datasets followed by task-specific adaptation. This approach has been widely adopted in both medical imaging and biomedical natural language processing (NLP), In medical imaging, models such as PLIP 66 37 6 fine-tune CLIP [ on pathology images and text, enabling effective zero-shot retrieval and classification. MedFILIIP
where labeled data are often scarce and annotation costly [3 36
incorporates domain-spectie supervision into contrastive vision-language pretraining. while MedKLIP (39 enhances medical grounding through knowledge
38
injection. MISS [40] formulates visual question answering generative task and achieves strong results with limited multimodal data. Other efforts [41]44] 7 focus on adapting general-purgy vision- languaee models to medical domains under distribution shift. In biomedical NLP, pretraining on domain-speciff corpora has led to series of strong models: BioBERT [8), ClinicalBERT [9], and PubMedBERT [OO], which all show clear improvements on downstream tasks such as disease mention recognition, drug gene relation extraction, and clinical report parsing.
Catastrophic Inheritance. Chen et al. [8 4 examine this effect of catastrophic inheritance through singular value decomposition, revealing that noise compresses the feature spectrum and diminishes the capacity of principal components. To mitigate this, they introduce NMTune, which enhances robustness by enforcing spectral preservation during fine-tuning. Building on similar insights, Chang et al. [44] propose BA-LORA for large language models, combining low-rank adaptation with spectral regularization. Separately, Li et al. (45) investigate how generative models inherit structural biases from pretraining distributions, with a focus on text generation tasks.
B Understanding The Noisy Labels In Pre-Training Data
We provide additional experiment details for the motivating example of ResNet-50 in this section. We also present the detailed results on each downstream dataset for noisy pre-trained models on both ImageNet- IK and YFCCI5M.
B.1 Pre-training datasets and Plyper-parameters
For analysis in Section 2, we conduct pre- pre-training of ResNet-50 on ImageNet-IK and YFCCI5M. For ImageNet- IK pre-training, we follow the training recipe in Wightman et (2021). To introduce noise in ImageNet-15 K, we use function cleanlab (Northcutt et al., 2021) to introduce symmetric noise in each class. For YFCC15M CLIP pre-training we follow the training recipe in Cherti al. (2023). To introduce noise in YFCC15M, we swap the text description between two randomly sampled image-text pairs until the noise ratio is achieved. We show the validation accuracy on ImageNet- K of the noisy ResNet-50 models pre-trained on ImageNet-1K IK and zero-shot accuracy on ImageNet-1K of the noisy ResNet-50 models pre-trained on YFCC15M in Table The results show that our pre-training achieves the state-of-the-art results (Wightman et al., 2021; Cherti et al., 2023), as basis for our further analysis.
B.2 Downstream Vision Datasets and Hyper-parameters
We present the details of the in-domain (ID) vision datasets in Table 4 and out-of-domain vision datasets Table 5. For ID, we conduct training on the training set and test on the validation set of the downstream dataset. For OOD on DomainNet (Peng et al., 2019), we conduct training on the training set of DomainNet Real or DomainNet Sketch, and test on all the other three DemainNet datasets not used in training. For OOD on ImageNet (Russakovsky et al., 2015), we conduct training on ImageNet training split and test on its variants. To transfer a pre-trained model, we use linear probing (LP) for analysis as shown in Section 2. We train the linear classifier for 30 epochs on each downstream dataset, using Adam (Kingma Ba, 2014) 4) optimizer with cosine scheduler. We do not use weight decay for linear probing and set the learning rate to 0.1 for all tasks.
14

---

Table 4: ImageNet-II IK validation and zero-shot accuracy of ImageNet- I pre-trained and YFCCI5M CLIP pre-trained noisy ResNet-50 models.
Noise Ratio
ImageNet- IK Pre-train Validation Accuracy
YFCC15M CLIP Pre-train
Zero-shot Accuracy
0%
79.96
32.64
5%
79.18
30.86
10%
78.61
29.54
20%
76.27
27.72
30%
73.11
26.53
Table 5: Details of the 3 medical vision datasets used to evaluate transfer performance.
Dataset
Classes
Train Size
Test Size
Evaluation Metric
Camelyon17
2
302,436
33,501
accuracy
HAM10000
8,138
2,000
accuracy
NIH ChestX-ray
14
89,322
25,596
accuracy
B.3 Detailed results
Figure 5 reports the detailed accuracy trends across different levels of synthetic label noise for YFCCI5M and ResNet-50 pre-training. Across all settings, SKD demonstrates consistent robustness compared to LP and NML. On Camelyon17 accuracy under YFCCI5M pre-training (Figure (5) remains above 83% up to 30% noise, with SKD maintaining clear margin over both baselines. For HAM10000 (Figure 5p), the performance more sensitive to noise, but SKD still preserves ~223% improvement under high-noise regimes. A similar trend is observed ChestX-ray (Figure where accuracy degrades stcadily, but SKD slows the collapse. Under supervised ResNet-50 pre-training (Figures 1-)f), SKD consistently achieves top performance across all datasets. For example, Camelyon1 17 (Figure (t),, accuracy stays above 90% even at 30% label noise, whereas LP and NML drop significantly. These trends confirm the effectiveness of SKD in preserving feature integrity under both contrastive and supervised pre-training paradigms, particularly in safety-critical medical tasks.
B.4 Detailed Feature and Logit Results
Skewness and kurtosis degradation under noise. We analyze the feature distributional changes of ResNet-50 models pre-trained with varying noise ratios by reporting the mean and variance of skewness and kurtosis on Camelyon HAM10000, and NIH ChestX-ray datasets. As shown in Figure 66 both skewness and kurtosis consistently degrade with increasing label noise. On Camelyon17. the skewness mean drops from 6.00 to 3.61, and kurtosis mean plummets from 110.02 to 57.28 noise increases from 0% to 30%, accompanied by a sharp decline in kurtosis variance from 232.15 to 93.66. Similar trends are observed on HAM10000 (skewness mean: 5.49 3.28; kurtosis mean: 78.05 - 43.71) and NIH ChestXray (skewness mean: 5.11 + 3.12; kurtosis mean: 73.44 - 24. 14). These shifts indicate a progressive flattening of the representation space under noise-featur dimensions become more symmetric (lower skewness) and less heavy-tailed (lower kurtosis), pointing to collapse of expressive capacity. Such structural degradation motivates our SKD design, which explicitly regularizes these higher-order moments preserve discriminative geometry.
Logit-level analysis. We examine how pre-training noise affects the distribution of logit outputs across different datasets. As shown Figure increasing noise levels lead to consistent rise in logit entropy cestropy_meany reflecting increased prediction uncertainty. For example, in NIHChestXray, entropy increases from 1.5710 at 0% noise 1.6830 at 30% noise. Simultaneously, both logit energy monergy_meant and maximum softmax probability (esp_mean) decrease, indicating lower confidence and greater dispersion in the predictions. On Camelyon 17, energy drops from 5.1 151 to 4.2888, while msp_meat falls from 0.9301 to 0.9006.
15

---

Castkeyy
Xidbs Tasto 2h
Jatee Tado 26
300h Sarta 750
Figure Average evaluation results of ImageNet-1K IK (IN-IK) fully supervised pre-training on downstream tasks with various percentages of data using ResNet-50. The robustness performance constantly decreases once noise is introduced in pre-training.
Camelyon17
NIHchesbXray
6.00
S.1.
Burnoo
Kurtr Meaan
4.10
Kuriaa
2.72
-11-22
$3.7
2:78
2.43
110
2.00
Figure 6: Degradation of skewness and kurtosis under label noise. We visualize the mean and variance of feature-wise skewness and kurtosis for ResNet-50 pre-trained with different levels of label noise. All three datasets (Camelyon 17, HAM10000, NIH ChestXray) exhibit consistent downward trend in both metrics as noise increases, indicating a flattening of the representation space. Skewness becomes closer zero (more symmetric), and kurtosis drops significantly (less peaky), reflecting reduced expressiveness and discriminative structure.
These patterns are tightly connected to the structural degradation observed in feature-level statis- tics spessssicciiclly reduced skewness and kurtosis under noisy supervision. Lower skewness suggests more symmetric and less distinctive feature distributions. whereas reduced kurtosis indicates lack of peakedness and diminished confidence concentration. Together, these shifts in the representation space translate into softer and more ambiguous logit-level predictions, underscoring the downstream impact of representational collapse. Our results support the view that structural noise inherited during pre-training propagates through to the output layer, impairing model reliability in high-stakes clinical settings.
C Experiment
More details of experiments in Section 4 are shown here.
16

---

HAM10000
-30
-M7
372
-333
Litt
-2.33
Staema Meat
-3.03
2.63
-2.80
-2.22 140
2213
10 3.5
25
25 30
33 30
Figure 7: Logit-level changes under pre-training noise. We report logit entropy, energy. and maximum softmax probability (MSP) for CLIP models pre-trained with different noise levels on Camelyon17, 17, HAM10000, and NIH ChestXray. As noise increases, entropy rises while both energy and MSP decrease, indicating higher prediction uncertainty and lower confidence. These trends mirror the flattening of feature-level skewness and kurtosis, suggesting that representational degradation propagates through to the output space.
Table 6: Details of the biomedical NER datasets used for evaluation. Each dataset provides token- level annotations for domain-specitf entity types (e.g., discases, chemicals, genes). Following prior work, we report both Fl score and accuracy.
Dataset
Entity Type
Annotation Scheme
Evaluation Metric
BC2GM
Gene/protein
IOB
FI, Accuracy
BC4CHEMD
Chemicals
IOB
FI, Accuracy
CRAFT
Multiple (e.g., cell, gene)
1OB
FI, Accuracy
RC5CDR-chem
Chemicals
10B
FI, Accuracy
BC5CDR-disease
Diseases
10B
FI, Accuracy
JNLPBA
Biomedical terms
10B
FI, Accuracy
NCBI-disease
Discases
1OB
FI, Accuracy
RioNIP11
Multiple event/mention types
10B
FI, Accuracy
RioNIP13
Multiple entity types (species, cell, etc.)
10B
FI, Accuracy
Ex-PTM
Protein post-translational modifications Anatomical entities
1OB
FI, Accuracy
AnatEM
1OB
FI, Accuracy
C.1 Detailed Setup For Language Model Experiment
To assess the generalizability of our method beyond vision, we evaluate its effectiveness in natural language processing (NLP) via biomedical named entity recognition (NER). Specifically, we use the BiomedBERT model as encoder backbone. This model is a domain-adapted variant of BERT, pre- pre-trained on PubMed abstracts and PMC full-text articles, making it well-suited for biomedical language tasks.
We conduct experiments across 32 standard biomedical NER benchmarks, including BC2GM, BCACHEMD, NCBI-disease, JNLPBA, and multiple BioNLP and CRAFT datasets. These datasets cover wide range of biomedical entity types such as genes, proteins, chemicals, diseases, and anatomical structures. A summary of representative datasets is provided in Table while full results across all benchmarks are included the supplementary materials.
We compare our proposed SKD method against linear probing (Baseline) and a recent baseline NML, reporting both Fl score and accuracy. Results demonstrate that SKD consistently improves upon prior methods across nearly all datasets, highlighting its robustness and transfcrability in language settings.
C.2 Biomedical NER Results.
We evaluate the effectiveness of our method SKD on 32 biomedical named entity recognition (NER) datasets spanning various entity types (e.g., chemical, gene/protein, species, disease) from benchmark corpora such BioNLP, CRAFT, BC2GM, and NCBI-disease. All experiments use the BioMedNLP-
17

---

334
0.80
0344
0313
0300
1393
3.00
384
900
Figure 8: Fl and Accuracy Biomedical NER Benchmarks. We report Fl score and accuracy for 32 biomedical NER datasets using the BioMedBERT backbone under three fine-tuning strategies: linear probing (LP), NML, and our proposed SKD,
model as the backbone, with fine-tuning conducted using three strategies: LP (Baseline), NML, and our proposed SKD.
As shown in Figure SKD achieves consistent improvements across both and accuracy metrics. For instance, B C2GM, SKD achieves Fl score of 0.9459 compared to 0.9053 (LP) and
18

---

0.9187 (NML); on BC4CHEMD, SKD further improves the FI to 0.9706. Across all datasets, SKD outperforms prior methods by an average margin of 1.21% in Fl and 0.57% accuracy. These results highlight SKI's ability to enhance robustness and representation quality in biomedical NLP tasks under noisy transfer settings.
D Extended Results and Analyses
D.1 Large-Scale Foundation Models
D.1.1 Vision: VIT-L (ImageNet-21IK pretrain with noisy labels)
We extend the PLIP setting to a large-scale vision backbone, ViT-L [3I, pre-trained on ImageNet-21K with noisy labels. Across three medical benchmarks, SKD consistently surpasses LP/GCE/NMIL confirming that skewness-kurtosis regularization scales to large backhones.
Table 7: ViT-L under noisy pre-training. Best in bold.
Model
Dataset
LP
GCE
NML
SKD
ViT-L Camelyon17
91.6
92.3
91.8
93.4
HAM10000
58.8
59.3
58.6
60.0
NIHChestXray
43.5
44.3
43.8
45.9
D.1.2 Language: GPT-2 for Biomedical NER
Beyond PubMedBERT, evaluate SKD GPT-2 32
across five representative NER datasets.
SKD matches or surpasses strong baselines on all benchmarks.
Table GPT-2 on biomedical NER. Best in bold.
Model
Dataset
LP
GCE
NML
SKD
GPT-2
AnatEM
93.2
93.5
93.6
93.9
BC2GM
91.3
91.6
91.5
91.8
BC5CDR-chem
95.4
95.5
95.5
95.7
BC5CDR-diseane
95.2
95.2
95.3
95.4
BioNLPO9
91.6
91.8
91.8
92.2
D.2 Robustness under Structured (Hierarchical) Noise
We simulate taxonomy-aae hierarchical mislabeling by replacing labels with WordNet hypernyms at controlled ratios (e.g., "Labrador Retriever" "doo""), SKD remains robust across all corruption levels on HAM10000.
Table 9: Taxonomy-aware hierarchical noise HAM10000.
Dataset
Noise(%)
L.
GCE
NML
SKD
HAM10000
56.9
56.6
57.4
58.4
56.5
56.7
57.0
58.4
56.2
56.4
56.6
57.7
56.0
55.9
56.0
$7.3
55.5
55.6
55.4
56.4
20
55.1
55.1
55.2
56.0
54.2
54.1
54.3
55.8
19

---

D.3
Sensitivity Analyses
D.3.1 Target Statistics (Te, Tk))
To evaluate the influence of the two target hyperparameters T, and тk (which specify the desired skewness and kurtosis), we conduct detailed sensitivity analysis the HAM10000 dataset. These targets were originally set to the negative values of the skewness and kurtosis computed from a clean model, but we also explore range of alternative settings to assess robustness.
The results in Table show that SKD is largely insensitive to the exact choice (Te, T&). This is expected since the primary role of SKD is to encourage non-degenerate skewness and kurtosis, counteracting representation flattening rather than matching precise target values. Across a wide range of target settings, the performance variation is within 1%. confirming that SKD remains stable even when the target statistics are misestimated. In all main experiments, we therefore simply fix Ts 0 and ти for consistency.
Table 10: Sensitivity of SKD to target (Tx: Tk.
on HAM10000 (Accuracy %).
HAM10000
Ta
-3.0
-1.0
0.0
1.0
3.0
Accuracy
56.9
56.9
56.9
56.9
56.8
HAM10000 Tk
-30.0
10.0
0.0
10.0
30.0
Accuracy
56.9
56.8
56.9
56.8
56.6
Acouracy
Accuracy
on Acturacy
92
Hooascy 760
Figure 9: Hyperparameter sensitivity of SKD components on Camelyon17.
D.3.2 Loss Weights (Askew Akurt: Adis)
On Camelyon17, varying loss weights across orders magnitude gives at most 1.12% fluctuation (for Lais), while Likww varies by only 0.54%. See Fig. 9 for full curves.
E Ablation study
We conduct ablation experiments disentangle the contributions of the three components in SKD: skewness regularization (S), kurtosis regularization (K), and logit disagreement regularization (D). As shown in TT removing any individual component consistently reduces performance across all noise levels and datasets. Among the single-componen variants, kurtosis (K) alone typically performs best, aligning with our hypothesis that high-order statisties play a dominant role mitigating representation collapse. Combinations of two terms improve upon single terms, while the full SKD achieves the best average performance on Camelyon17 (90.76%). HAM10000 (57.73%), and NIHChestXray (44.41%), highlighting the complementary effects of the three regularizers.
20

---

Table 11: Ablation study on SKD components using ResNet-S0.. We evaluate variants of SKD by selectively removing components: skewness (S), kurtosis (K), and disagreement (D), across three medical datasets. The full SKD consistently achieves the best performance under varying pre-training noise ratios, confirming the complementary benefits of each regularizer.
Datasct
Method
0
10
20
30
Avg
S
90.10
91.05
88.24
88.77
88.10
89.25
91.41
92.53
88.70
90.57
88.06
90.25
91.02
90.77
89.01
88.27
88.78
89.57
Camelyon17
S&K
91.52
92.11
90.72
88.92
88.79
90.41
S&D
91.09
91.54
90.22
87.88
87.74
89.69
K&D
91,28
92.13
90.72
88.64
88.69
90.29
SKD
91.76
92.50
91.39
89.12
89.02
90.76
56.02
55.25
55.55
57.53
55.72
56.01
56.14
55.21
54.74
57.03
56.08
55.84
D
55.95
53.73
55.62
56.45
55.17
55.38
HAM10000
S&K
57.34
55.98
55.42
57.90
56.87
56.70
S&D
56.87
56.45
55.68
57.44
56.15
56.52
K&D
56.96
55.75
57.37
56.54
56.53
SKD
58.25
57.54
56.94
58.37
57.54
$7.73
S
41.75
39.90
40.66
41.20
41.91
41.08
43.85
41.98
42.52
43.18
43.45
43.00
D
41.70
38.56
41.02
40.35
38.23
39.97
NTHlchestXray
S&K
43.80
41.90
42.56
43.23
43.45
42.99
S&D
43.57
42.08
42.24
42.89
43.58
42.87
K&D
43.62
42.44
42.36
43.15
44.38
43.19
SKD
44.81
43.37
44.22
44.25
45.39
44.41
Table 12: Training time comparison on PanNuke using PLIP.
Dataset
I.P
NMI.
SKD
PanNuke
20(s)
100(s)
150(s)
F
Running Time
To assess computational efficiency, we measure the wall-clock training time of LP, NML, and SKD on the PanNuke dataset using the PLIP model. As shown in Table 12 LP the fastest with only 20 seconds per run, while NML takes approximately 100 seconds. SKD introduces additional overhead due to the skewness, kurtosis, and disagreement regularization losses. requiring 150 seconds. Despite the added complexity, SKD remains lightweight and practical for real-worke deployment, with only 2.5x increase over LP and 1.5x increase over NML, All experiments were conducted on a single NVIDIA A100 40GB GPU.
G
More Discussion
G.1
Future Work
We outline several future directions. One is to evaluate the proposed skewness kurtosis regularization scale, particularly on larger backbones and long-taile tailed medical tasks with high inter-class similarity. Another direction is to generalize the regularization beyond fixed targets: current methods use global statistics as anchors, but instance- or task-adaptive constraints may better preserve local geometry and class separability. Additionally, while our method focuses preserving internal structure, integrating with uncertainty estimation confidence calibration could further improve reliability, especially in settings with weak supervision low-resource deployment.
21

---

G.2 Broader Impact
This study contributes to improving the robustness of foundation models safety- -ciiticll domains such as healthcare. By addressing structural degradation caused by noisy pretraining, our method may reduce silent failures and improve model interpretability under distribution shift. However, as with any representation regularization method, care must be taken to ensure that the learned structure does not inadvertently suppress rare or underrepresented patterns. high-stakes deployment, we recommend incorporating fairness audits, uncertainty quantification, and human oversight to prevent overconfidence and preserve trustworthiness.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims are accurate.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Sec in the appendix for discussion the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All necessary experimental details are provided in Sec 3.1 See and supple- mentary materials.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification: We have included the code and environment configuration in the supplementary material, and will release them publicly after the final organization is completed.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipss.cc/pulli///uuu guwde//ddee/// Cot//ooodeSesiseeeeeeeeee wwwplimmsssioooooooooooooooi for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed explanations of these aspects in the supplementary material of the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed results from five repeated experiments in the supplemen- tary material, demonstrating statistical reliability.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
25

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions of the experimental setup in the supplemen- tary material, and report execution time in Sec 4.5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httt:::/heurripp.cc//bbbbbie
Answer:
[Yes]
Justification:
We fully comply with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential broad impacts of our work in Sec G.2
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our model does not pose high risk of misuse, and the data used comes from publicly available datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly cited all the data and models used in the paper, and have complied with their respective licenses and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide the model installation environment, execution instructions, and detailed parameter settings.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve any of the above.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This question is not applicable our paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
28

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This question is applicable our paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non -ta-dtandrrd components.
Please refer to our LLM policy for what should or should not be described.
29


Reviews:
[{"review_id": "CBa6aUPjdu", "reviewer": "Reviewer_7kcL", "summary": "This paper presents a systematic study of catastrophic inheritance in medical foundation models—where noise in the pretraining stage harms downstream out-of-distribution (OOD) performance. The authors simulate controlled label noise during pretraining of ResNet-50 and CLIP on ImageNet-1K and YFCC15M datasets and evaluate transfer to downstream medical tasks like histopathology, dermatology, and radiology. They show that even minor label corruption significantly degrades OOD generalization, accompanied by a flattening of the representation space (i.e., reduced skewness and kurtosis in features/logits).\n\nTo mitigate this, they introduce SKD, a fine-tuning strategy that regularizes skewness and kurtosis of feature representations. This method requires no modification to frozen backbones and adds minimal computational overhead. Empirical results demonstrate consistent gains across various medical imaging and biomedical NLP benchmarks compared to existing methods like NML.", "questions": "1. Realistic Noise Settings: Can you extend your analysis to structured or semantic label noise (e.g., hierarchical mislabels or co-occurrence noise), which is more reflective of real-world medical corpora?\n2. Target Statistics: How sensitive is SKD to the target skewness/kurtosis values? Are these computed from a clean model, and what happens if these targets are misestimated?\n3. Scalability: Have you tested the method on large foundation models (e.g., ViT-L, MedCLIP, or LLaVA-Med)? This would help demonstrate scalability.\n4. Interpretability & Clinical Relevance: Can you connect changes in skew/kurtosis with specific pathology structures (e.g., lesion shape variability)? This could strengthen interpretability.\n5. Uncertainty Calibration: Since logit entropy increases with noise, have you considered integrating uncertainty calibration methods to complement SKD?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths \n1. Rigorous experiments across multiple model architectures, tasks, and domains.\n2. Clear empirical demonstration that label noise in pretraining harms medical model transferability.\n3. Thorough analysis using higher-order statistics to explain representation collapse.\n4. Well-motivated and effective mitigation strategy with lightweight implementation.\n5. Tackles a highly relevant and underexplored problem in medical AI—how foundation model noise affects clinical deployment.\n6. Provides actionable insight into safe adaptation of large medical models, many of which are frozen or black-box.\n\nWeaknesses\n1. Only considers synthetic uniform label noise. Real-world noise is likely more structured (e.g., co-occurrence biases).\n2. Experiments are limited to mid-sized models (ResNet-50, ViT-B/32); it remains unclear if findings generalize to very large-scale models.\n3. While impactful in the medical domain, its generalization to non-medical tasks is not explored (but appropriately scoped).\n4. Builds on prior work like NML and SVDEntropy; while differentiated, it's an incremental rather than revolutionary advancement.", "was_revised": true, "final_justification": "The rebuttal strengthens an already solid submission. I maintain my recommendation to accept.", "rebuttal": "## Reviewer 7kcL\nWe thank the reviewer 7kcL for your constructive feedback. Below we address each point in turn.\n\n**Q1:** The current analysis focuses on synthetic uniform label noise; extending it to structured or semantic noise (e.g., hierarchical or co-occurrence-based) would better reflect real-world medical scenarios.\n\n**A1:** Thank you for this valuable suggestion. Following your advice, we have extended our analysis to include **hierarchical mislabeling** to better reflect structured noise. Specifically, we used the ImageNet dataset as a base and synthetically introduced taxonomy-aware label noise: a controlled percentage of image labels were randomly replaced with their WordNet hypernyms, simulating realistic structured mislabeling scenarios common in annotation pipelines (e.g., “Labrador Retriever” → “dog”).  \n\nWe then trained a ResNet-50 model from scratch under different noise levels and evaluated four methods:**LP**(standard linear probing), **GCE[1]**(Generalized Cross-Entropy), **NML**(spectrum-preserving baseline), **SKD**(our proposed method)  \n\nResults show the effectiveness of SKD under structured, taxonomy-aware noise: \n\n\n|Dataset|Noise(%)|LP|GCE|NML|SKD|\n|-|-|-|-|-|-|\n|HAM10000|0|56.9|56.6|57.4|58.4|\n||1|56.5|56.7|57.0|58.4|\n||2|56.2|56.4|56.6|57.7|\n||5|56.0|55.9|56.0|57.3|\n||10|55.5|55.6|55.4|56.4|\n||20|55.1|55.1|55.2|56.0|\n||30|54.2|54.1|54.3|55.8|\n\nFinally, we would like to highlight that, while part of our study uses synthetic uniform noise for controlled experiments, our main evaluations—particularly those on **PLIP**—already involve real-world, naturally occurring noise. The noisy image–text pairs in PLIP contain uncontrolled web-scale noise, making PLIP a strong real-world testbed for validating our method.\n\n**Q2:** The current experiments focus on mid-sized models; evaluating on large foundation models is needed to demonstrate scalability and generalizability. \n\n**A2:** Thank you for this important suggestion. Following your advice, we have conducted additional experiments to verify our method on large-scale foundation models.  \n- For vision task, we evaluated on the **ViT-L** backbone.  \n- For natural language processing, we extended our experiments to **GPT-2**.  \n\nThe results demonstrate that SKD continues to achieve consistent improvements, further validating its scalability.  \n\n|Model|Dataset|LP|GCE|NML|SKD|\n|-|-|-|-|-|-|\n|ViT-L|Camelyon17|91.6|92.3|91.8|93.4|\n||HAM10000|58.8|59.3|58.6|60.0|\n||NIHChestXray|43.5|44.3|43.8|45.9|\n \n|Model|Dataset|LP|GCE|NML|SKD|\n|-|-|-|-|-|-|\n|GPT-2|AnatEM|93.2|93.5|93.6|93.9|\n||BC2GM|91.3|91.6|91.5|91.8|\n||BC5CDR-chem|95.4|95.5|95.5|95.7|\n||BC5CDR-disease|95.2|95.2|95.3|95.4|\n||BioNLP09|91.6|91.8|91.8|92.2|\n\n**Q3:** While impactful in the medical domain, its generalization to non-medical tasks is not explored (but appropriately scoped).\n\n**A3:** Following your advice, we have conducted additional experiments on three widely-used non-medical vision benchmarks:**DTD (Describable Texture Dataset)**, **Office-31**, **CIFAR-10**.\n\nThe results consistently show that SKD improves performance under noise across all datasets, further validating our hypothesis. Notably, on **CIFAR-10**, SKD under **30% noise** still outperforms LP under **5% noise**, demonstrating its strong ability to mitigate severe label corruption.\n\n\n|Dataset|Noise(%)|LP|GCE|NML|SKD|\n|-|-|-|-|-|-|\n|DTD|0|70.1|70.3|70.5|70.6|\n||5|70.7|70.6|70.7|71.2|\n||10|69.2|69.2|69.5|69.6|\n||20|68.8|68.8|69.0|68.8|\n||30|67.9|68.2|68.4|68.2|\n|CIFAR-10|0|91.2|91.4|92.1|93.2|\n||5|90.0|90.4|91.6|92.8|\n||10|89.2|89.3|90.7|92.4|\n||20|89.1|89.3|90.4|92.0|\n||30|86.5|86.7|87.9|90.6|\n|Office-31|0|81.3|82.1|82.5|84.3|\n||5|80.9|81.1|81.9|82.7|\n||10|81.7|81.7|82.2|82.5|\n||20|77.9|77.8|78.1|78.7|\n||30|72.3|72.0|72.3|72.5|\n\n**Q4:** Builds on prior work like NML and SVDEntropy; while differentiated, it's an incremental rather than revolutionary advancement.\n\n**A4:** Thank you for this thoughtful comment. While our work relates to prior studies such as NML, it introduces a **fundamentally new perspective** and addresses a **critical gap** in the literature.\n\n1. **New problem setting:** We systematically study **catastrophic inheritance under noisy medical pretraining**, a practically important yet underexplored challenge—especially relevant for large-scale medical vision-language models like **PLIP**. This setting, where frozen noisy features are transferred to downstream classifiers, is distinct from conventional noisy-label learning and demands targeted solutions. Our framework directly targets this high-stakes, underexplored setting.\n\n2. **Novel perspective:** Our method introduces the use of **skewness and kurtosis** as high-order, distribution-level signals to diagnose and mitigate feature collapse. This goes beyond spectrum-preserving or low-order statistics used in prior work, providing a **structurally richer and more interpretable signal** to guide learning. To our knowledge, this is the **first work to explicitly regularize distributional shape** in the context of representation collapse.\n\n3. **Broad impact and generality:** Our method achieves **consistent and significant improvements** over baselines on a range of **challenging medical datasets** (e.g., HAM10000, Camelyon17, NIH ChestXray), and scales effectively to **large foundation models** (e.g., ViT-L, PLIP, GPT-2), and shows **consistent gains across both vision and NLP domains**. These improvements hold even on large-scale foundation models and challenging medical datasets.\n\nWe will emphasize these unique contributions more clearly in the revision to better highlight the **problem focus, conceptual novelty, and broad empirical benefits** of our work.\n\n**Q5:** Target Statistics: How sensitive is SKD to the target skewness/kurtosis values? Are these computed from a clean model, and what happens if these targets are misestimated?\n\n**A5:** Thank you for raising this important question. Following your suggestion, we conducted additional sensitivity experiments for the two hyperparameters on HAM10000 (the target skewness and kurtosis values, denoted as τs and τk).  \n\nThe results demonstrate that SKD is not sensitive to these hyperparameters, which is expected because the primary effect of our method is to counteract representation collapse by encouraging non-degenerate skewness and kurtosis. While these targets were initially set to the negative values of the skewness and kurtosis computed from a clean model, we found that other reasonable values (not excessively large) achieve similar mitigation.  \n\nIn all our main experiments, we simply set **τs = 0** and **τk = 0** for consistency. Below are the sensitivity results:  \n\n|HAM10000|τs|-3.0|-1.0|0.0|1.0|3.0|\n|-|-|-|-|-|-|-|\n||Acc|56.9|56.9|56.9|56.9|56.8|\n\n|HAM10000|τk|-30.0|-10.0|0.0|10.0|30.0|\n|-|-|-|-|-|-|-|\n||Acc|56.9|56.8|56.9|56.8|56.6|\n\n**Q6:** Interpretability & Clinical Relevance: Can you connect changes in skew/kurtosis with specific pathology structures (e.g., lesion shape variability)? This could strengthen interpretability.\n\n**A6:** We thank the reviewer for this thoughtful question. While our core analysis focuses on high-order statistics of feature distributions, we agree that bridging these changes with observable pathology structures would strengthen interpretability.\n\nTo investigate this connection, we conducted a **class-wise analysis** of feature distribution shape and intra-class variance (as a proxy for pathology heterogeneity) on DigestPath. Results are summarized below:\n\n|Class|Intra-Class Variance|Mean Skewness|Mean Kurtosis|\n|-|-|-|-|\n|Benign|64.0|7.00|105|\n|Malignant|76.2|8.77|214|\n\nWe observe that classes with more histological complexity—like malignant tumors—naturally exhibit higher skewness and kurtosis, suggesting that these metrics may reflect lesion-level heterogeneity such as irregular boundaries or intra-class diversity.\n\nTo complement this, we performed a qualitative Grad-CAM analysis comparing models trained under clean and noisy pretraining. We found that activation maps under clean conditions (with higher skew/kurtosis) better captured relevant lesion morphology, while those under noisy pretraining (with collapsed skew/kurt) became more diffuse and less structure-aware. This supports the interpretation that the collapse of distributional shape also signals a loss of clinically meaningful structure.\n\nDue to rebuttal constraints, we omit figures here but will include visualizations in the final version. Thank you for encouraging this line of interpretability exploration.\n\n**Q7:** Since logit entropy increases with noise, have you considered integrating uncertainty calibration methods to complement SKD?\n\n**A7:** We thank the reviewer for this valuable suggestion. Following your suggestion, we applied **Temperature Scaling (TS)[2]**, a widely used post-hoc calibration technique that adjusts softmax confidence via a learned temperature parameter without modifying model weights.\n\nThe table below summarizes classification accuracy:\n\n|Dataset|Noise(%)|SKD|SKD+TS|\n|-|-|-|-|\n|Camelyon17|0|91.8|91.8|\n||5|92.5|92.6|\n||10|91.4|91.4|\n||20|89.1|89.4|\n||30|89.0|89.4|\n|HAM10000|0|58.3|58.4|\n||5|57.5|57.5|\n||10|56.9|57.0|\n||20|58.4|58.4|\n||30|57.5|57.7|\n\nWe observe that SKD already yields robust predictions across noise levels, and temperature scaling provides a modest but consistent boost. We appreciate the reviewer’s suggestion and will include this analysis in the final version.\n\n[1] Zhang, Zhilu, and Mert Sabuncu. \"Generalized cross entropy loss for training deep neural networks with noisy labels.\" Advances in neural information processing systems 31 (2018).\n\n[2] Guo, Chuan, et al. \"On calibration of modern neural networks.\" International conference on machine learning. PMLR, 2017."}, {"review_id": "XQFew9vSue", "reviewer": "Reviewer_rgdp", "summary": "Authors explain in detail how medical imaging models finetuned from pretrained models exhibit a reduced generalization due to label noise in the pre-training corpus. They determine that small amounts of noise lift in-distribution accuracy while harming out-of-distribution (OOD) accuracy; this catastrophic inheritance phenomenon is known in the nautral image domain, but yet to be shown in the medical imaging field. Experiments are performed by swapping labels in nautral image-text pre-training corpuses and perfoming downstream medical imaging tasks. Results show that skewness and kurtosis of downstream features (and logits) decline with increased label noise, indicating a flattening of the feature space. Authors propose a method to recapture generalization by adding a lightweight MLP transforming features from the pre-trained model using 2 scalar losses: (S)kewness and (K)urtosis. A (D)isagreement loss on the logit space is also employed leading to their SKD framework. Analsis of these stats demonstrate that representation collapse under pre-training noise is \"not solely a function of model size or task loss, but a structural issue in distributional shape\".", "questions": "Do authors get any sense of the importance of catastrophic inheritance on different types of downstream task? e.g. how much more severe is the problem on an MRI segmentation task compared with an X-Ray text-summarization task?\nIs the problem also imaging domain variant - in more difficult domains such as ultrasound, and histopathology, is the problem exacerbated compared with images of larger structures, e.g. MRI.", "limitations": "Authors humbly acknowldge the limitations of their work, citing modest model sizes and likely unrealistic unrealistic noise distributions in their experiments.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Authors clearly introduce the problem of catastrophic inheritance and lay out their research agenda.\nThe paper is well-written, clear, and explicitly demonstrates catastrophic inheritance is present in finetuned medical imaging tasks.\nAuthors show that skewness and kurtosis are viable statistics to measure and target when improving the downstream generalization of medical imaging finetuning.\nThe losses are simple and well-informed with little overhead during finetuning.\nThe method requires no alteration to the backbone pre-trained model.", "was_revised": true, "final_justification": "Based on Authors' rebuttal and discussion on these and other Reviewers' comments, I maintain the rating of 5: Accept. I thank the Authors for their timely and constructive discussion.", "rebuttal": "## Reviewer rgdp\nWe thank the reviewer rgdp for your constructive feedback. Below we address each point in turn.\n\n**Q1:** Do authors get any sense of the importance of catastrophic inheritance on different types of downstream task? e.g. how much more severe is the problem on an MRI segmentation task compared with an X-Ray text-summarization task? Is the problem also imaging domain variant - in more difficult domains such as ultrasound, and histopathology, is the problem exacerbated compared with images of larger structures, e.g. MRI?  \n\n**A1:** We sincerely thank the reviewer for this insightful question. We could not agree more that catastrophic inheritance is likely to manifest differently across task types and imaging modalities, and that this deserves a much deeper analysis.  \n\nOur current study primarily focuses on **medical image classification tasks** (e.g., HAM10000, Camelyon17, NIH ChestX-ray), where we observe a clear **flattening of feature distributions**—quantified by reduced skewness and kurtosis—under noisy pretraining. We also include **medical NER experiments** (e.g., PubMedBERT) to show that similar degradation patterns occur in **language models**, suggesting the phenomenon is not modality-specific.  \n\nThat said, we strongly believe that **medical segmentation tasks** (e.g., MRI or ultrasound segmentation) may exhibit *distinct and potentially even more severe forms* of catastrophic inheritance, because noise propagates differently in dense prediction settings and these modalities often involve inherently higher uncertainty. This is a **separate and highly valuable research problem**, and we are committed to investigating it in future work with the level of depth it deserves.  \n\nWe are very grateful to the reviewer for highlighting this direction—it aligns closely with our long-term vision."}, {"review_id": "ZeG4pbPfZa", "reviewer": "Reviewer_Xipk", "summary": "This paper explores catastrophic inheritance in medical models, where label noise in pretraining data impacts downstream tasks and degrades out-of-distribution (OOD) performance. The authors make three key contributions: (1) demonstrating through controlled experiments that even mild pretraining noise (5%) induces structural degradation in downstream medical tasks; (2) demonstrating that noise flattens learned representations by reducing skewness and kurtosis in feature and logit distributions; and (3) proposing SKD (Skewness and Kurtosis Distribution), a lightweight fine-tuning method that regularizes skewness and kurtosis to counteract representational collapse. The proposed SKD method adds regularization terms to encourage target skewness and kurtosis values during fine-tuning, along with a disagreement loss to enhance output separability. \n\nThe paper evaluates corrupting labels in ImageNet-1K and YFCC15M during pretraining, then evaluating on medical benchmarks including Camelyon17, HAM10000, and NIH ChestX-ray. The experiments demonstrate consistent improvements across multiple medical datasets and can be applied to foundation models including PLIP and PubMedBERT.", "questions": "- Line 220: You mention skewness and kurtosis is unstable, thus you compute a disagreement-based loss. Why then keep skewness and kurtosis? And how is that your ablation study shows limited impact of the disagreement loss, and much larger impact to the Kurtosis loss?\n- Why are you evaluating only medical classification while the phenomena you describe should appear in all downstream tasks. Why only evaluating NML as an improved baseline? \n- How do you compute / define the target statistics τs and τk ?\n- What happens under less than 10% of noise ratio for kurosis and swkeness? For isntance at 1%, 2%, 5% ? Figure 3 does not really justify the impact/relation to these statistics beyond 10% noise ratio.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "1-Novelty: \nStrength: The use of higher order moments is novel to study the impact of noised labels\n\n2-Impact: \nStrength: The use-cases covered are critical and noised pre-training data is very common in medical setting\nWeakness: There is no reason the phenomena you are describing and the mitigation solution you are proposing only works in medical settings. Does the phenomena and performance drops (and mitigation with your framework) can be found on non medical settings?\n\n3-Soundness:\nStrength: Expensive ablation and sensitivity analysis, diverse medical classification datasets\nWeaknesses: The perfectly random noise is the only one covered; Only one improved baseline compared; While these limitations are acknowledged, there is nothing that hints that the proposed theory and framework generalize to these settings. The fact that there is correlation between the kurtosis/skewness and the drop of downstream performance is not causation. It is not discussed whether both only reflect some deeper problem. Especially as there is not definitive link between skewness/kurtosis and noise ratio in Figure 3. Indeed, except the drop between 0 and 10%, the metrics are relatively stable (increasing / decreasing slightly around the 10% value) \n\n4-Clarity:\nWeakness: The figure 3 is unreadable because of the legend. There is no way to distringuish the mean and the variation because the legends does not show dashed lines", "was_revised": true, "final_justification": "The authors addressed my concerns sufficiently, and provided additional analyses.\nI have no reason to oppose acceptance anymore.", "rebuttal": "## Reviewer Xipk\nWe thank the reviewer Xipk for your constructive feedback. Below we address each point in turn.\n\n**Q1:** Whether the observed phenomenon and the effectiveness of the proposed mitigation are specific to the medical domain or also apply to non-medical settings.\n\n**A1:** We thank the reviewer for raising this important question. Following your advice, we conducted new experiments on three general-purpose vision benchmarks:DTD, CIFAR-10, Office-31.\n\nWe first analyzed skewness and kurtosis of feature representations. The results confirm that both statistics decrease as noise increases, which supports our claim that feature distributional collapse is a general phenomenon:\n\n|Dataset|Noise(%)|Skewness|Kurtosis|\n|-|-|-|-|\n|CIFAR-10|0|5.92|78.3|\n||5|4.23|38.8|\n||10|4.24|39.2|\n||20|4.13|37.1|\n||30|4.08|36.5|\n\nWe then compared the downstream classification performance of various methods, including:  LP (linear probing), GCE (Generalized Cross Entropy)[1], NML (spectral norm preserving baseline), SKD (ours):\n\n|Dataset|Noise(%)|LP|GCE|NML|SKD|\n|-|-|-|-|-|-|\n|DTD|0|70.1|70.3|70.5|70.6|\n||5|70.7|70.6|70.7|71.2|\n||10|69.2|69.2|69.5|69.6|\n||20|68.8|68.8|69.0|68.8|\n||30|67.9|68.2|68.4|68.2|\n|CIFAR-10|0|91.2|91.4|92.1|93.2|\n||5|90.0|90.4|91.6|92.8|\n||10|89.2|89.3|90.7|92.4|\n||20|89.1|89.3|90.4|92.0|\n||30|86.5|86.7|87.9|90.6|\n|Office-31|0|81.3|82.1|82.5|84.3|\n||5|80.9|81.1|81.9|82.7|\n||10|81.7|81.7|82.2|82.5|\n||20|77.9|77.8|78.1|78.7|\n||30|72.3|72.0|72.3|72.5|\n\nAcross all datasets, SKD consistently outperforms alternatives, demonstrating its effectiveness across both medical and general-purpose domains. On CIFAR-10, SKD under 30% noise even surpasses LP trained with only 5% noise, showcasing its ability to counteract heavy label noise.\n\n**Q2:** The reviewer questions the generalizability of the framework, noting it only addresses random noise and one baseline, and argues that the observed correlation between skewness/kurtosis and performance drop may not imply causation, especially given the weak link to noise ratio in Figure 3.\n\n**A2:**  We sincerely thank the reviewer for this comprehensive and thoughtful comment. Your question touches on several important aspects, and we will address each sub-question one by one in detail:  \n\n1). **Noise type**\n\nThank you very much for this important suggestion. Following your advice, we have conducted additional experiments to explicitly evaluate robustness under structured noise settings.  \n\nWe used the ImageNet dataset as a base and synthetically introduced taxonomy-aware hierarchical label noise, where a controlled percentage of image labels were randomly replaced by their WordNet hypernyms. This setting simulates realistic structured mislabeling commonly observed in real-world annotation pipelines. Below is the results:\n \n|Dataset|Noise(%)|LP|GCE|NML|SKD|\n|-|-|-|-|-|-|\n|HAM10000|0|56.9|56.6|57.4|58.4|\n||1|56.5|56.7|57.0|58.4|\n||2|56.2|56.4|56.6|57.7|\n||5|56.0|55.9|56.0|57.3|\n||10|55.5|55.6|55.4|56.4|\n||20|55.1|55.1|55.2|56.0|\n||30|54.2|54.1|54.3|55.8|\n\nAs shown above, SKD consistently improves robustness across all noise levels, validating its effectiveness in structured label corruption scenarios.\n\nWe would also like to emphasize that our main experiments—particularly those on PLIP—already involve realistic, naturally occurring noise. The noisy image–text pairs in PLIP capture uncontrolled web-scale noise, including co-occurrence bias and semantic drift, making it a strong real-world testbed for validating our method. \n\n2). **Baseline comparison**  \nThank you for pointing this out. We would like to clarify that catastrophic inheritance in noisy medical pretraining is a relatively novel research direction, with very few existing methods specifically designed to address this phenomenon.\n\nGiven this gap, and to ensure a more comprehensive comparison, we have supplemented our original baselines by including **GCE (Generalized Cross-Entropy)[1]**—a well-established method from the noisy label learning literature. Although not originally designed for catastrophic inheritance, GCE offers a strong baseline for robustness evaluation. As shown below, SKD consistently outperforms all other methods:\n\n|Dataset|Noise(%)|LP|GCE|NML|SKD|\n|-|-|-|-|-|-|\n|Camelyon17|0|91.2|91.1|89.3|91.8|\n||5|90.7|91.4|92.4|92.5|\n||10|89.2|89.5|88.1|91.4|\n||20|88.6|89.1|90.5|89.1|\n||30|88.2|88.6|91.3|89.0|\n|HAM10000|0|55.7|55.7|54.7|58.3|\n||5|54.6|54.8|55.2|57.5|\n||10|54.5|54.5|54.2|56.9|\n||20|54.5|54.4|54.8|58.4|\n||30|53.9|54.1|50.7|57.5|\n\n3). **Correlation vs. causation regarding skewness/kurtosis**\n\nWe thank the reviewer for this important comment. To further investigate this connection, we conducted a class-wise analysis of feature distribution shape and intra-class variance (as a proxy for pathology heterogeneity) on DigestPath:\n\n|Class|Intra-Class Variance|Mean Skewness|Mean Kurtosis|\n|-|-|-|-|\n|Benign|64.0|7.00|105|\n|Malignant|76.2|8.77|214|\n\nWe observe that classes with more histological complexity—such as malignant tumors—naturally exhibit higher skewness and kurtosis, suggesting that these metrics reflect lesion-level heterogeneity, including irregular boundaries and intra-class diversity.  \n\nTo complement this analysis, we performed a qualitative Grad-CAM comparison between models trained with clean vs. noisy pretraining. We found that activation maps from clean models better captured relevant lesion morphology. This supports the interpretation that the collapse of distributional shape also signals a loss of clinically meaningful structure.  \n\nWe validated this interpretation under hierarchical label noise, where increasing noise levels led to clear declines in skewness and kurtosis, indicating feature collapse.\n\n|Dataset|Noise(%)|Skewness|Kurtosis|\n|-|-|-|-|\n|HAM10000|0|3.92|52.5|\n||1|3.33|37.2|\n||2|3.25|31.7|\n||5|3.16|29.7|\n||10|3.13|27.3|\n||20|3.11|26.0|\n||30|3.05|22.1|\n\nThese results confirm that skewness and kurtosis reliably reflect feature collapse, even under realistic noise. Full visualizations will be included in the final version.\n\n**Q3:** The figure 3 is unreadable because of the legend. There is no way to distringuish the mean and the variation because the legends does not show dashed lines\n\n**A3:** We thank the reviewer for pointing this out. We acknowledge that Figure 3 currently lacks a clear legend, affecting readability. In the revision, we will add an explicit legend to clearly distinguish mean and variation.  \n\n**Q4:** Why skewness and kurtosis are kept if they are unstable in Line 220, and why the ablation shows disagreement loss has limited impact compared to kurtosis loss.\n\n**A4:** Thank you for highlighting this important point. We apologize for the **potential source of confusion** and would like to clarify the intended meaning.\n\nIn Line 220, the term **unstable** refers specifically to computing skewness and kurtosis **in the logit space**, not the **feature space** used in our method. This distinction is crucial:\n\n- In many medical classification tasks, the logit dimension is very small (e.g., 2 classes), which makes high-order moment statistics such as skewness and kurtosis **numerically unstable and less informative** when applied directly to logits.\n- In contrast, when computed over the **feature representations**, these statistics are **stable, well-defined, and strongly correlated with representation quality**, forming the foundation of our SKD regularization approach.\n\nWe appreciate the opportunity to clarify this point and will revise the relevant sentence in the final version to avoid confusion.\n\nWhile the disagreement loss (D) may appear to contribute a smaller relative improvement compared to the skewness–kurtosis regularization (SK) alone, it still provides **meaningful complementary gains**, with over 1% improvement on both HAM10000 and NIH ChestXray, further reinforcing the effectiveness of our full method.\n\n|Dataset|Method|Acc|\n|-|-|-|\n|Camelyon17|LP|89.6|\n||+SK only|90.4|\n||+SK+D|90.8|\n|HAM10000|LP|54.6|\n||+SK only|56.7|\n||+SK+D|57.7|\n|NIHchestXray|LP|38.7|\n||+SK only|43.0|\n||+SK+D|44.4|\n\n**Q5:** How do you compute / define the target statistics τₛ and τₖ?\n\n**A5:** We appreciate the reviewer for raising this question. Following your advice, we performed additional sensitivity studies on (τₛ) and kurtosis (τₖ).  \n\n|HAM10000|τs|-3.0|-1.0|0.0|1.0|3.0|\n|-|-|-|-|-|-|-|\n||Acc|56.9|56.9|56.9|56.9|56.8|\n\n|HAM10000|τk|-30.0|-10.0|0.0|10.0|30.0|\n|-|-|-|-|-|-|-|\n||Acc|56.9|56.8|56.9|56.8|56.6|\n\nOur experiments show that SKD is highly robust to the choice of these hyperparameters. This aligns with the intuition that the main contribution of SKD lies in alleviating representation collapse by promoting meaningful skewness and kurtosis. Although τₛ and τₖ were initially set to the negative values of those measured from a clean model, we observed that other reasonable values (as long as they are not extreme) lead to very similar improvements. For simplicity and consistency, we set **τₛ = 0** and **τₖ = 0** across all main experiments.\n\n**Q6:** The reviewer asks about the behavior of skewness and kurtosis under low noise levels (<10%).\n\n**A6:** Thank you for the thoughtful comment. As noted earlier, we have added results under 1%, 2%, and 5% hierarchical label noise, including skewness/kurtosis and performance comparisons across LP, GCE, NML, and SKD.\n\nThese results show that even mild noise leads to a decline in skewness/kurtosis, indicating early collapse, and that SKD remains effective in this regime. We will revise Figure 3 and clarify these trends in the final version.\n\n[1] Zhang, Zhilu, and Mert Sabuncu. \"Generalized cross entropy loss for training deep neural networks with noisy labels.\" Advances in neural information processing systems 31 (2018)."}, {"review_id": "cxdVMn0q4A", "reviewer": "Reviewer_PCzQ", "summary": "This work investigates the impact of pretraining on noisy datasets in the context of pathology. The authors demonstrate that using foundation models pretrained on noisy datasets can negatively affect the fine-tuning process on downstream medical tasks. The paper supports its claims with comprehensive experiments across multiple datasets and provides an insightful analysis of the relationship between the amount of noise in the pretraining data and resulting performance when fine-tuning on medical data.\nTo address this issue, the authors introduce a method that improves robustness under noisy pretraining conditions by working on Skewness, Kurtosis and logits.", "questions": "1. As mentioned earlier in the Weaknesses section, I have concerns regarding the choice of the values for $\\tau_s$ and $\\tau_k$. How sensitive is the method to these values? What do the authors mean by \"obtained from a clean model\" or \"set empirically\"? \n\n2. The method demonstrates improved classification across several datasets. However, are these gains consistent across all classes, or are they primarily driven by the some classes in the training set? A per-class performance analysis would clarify whether the method benefits minority or difficult classes.\n\n3. The method is evaluated on ResNet-50 in Table 1. Have the authors tested it on other CNN or ViT architectures? Providing results on more diverse backbones would support the generalizability claims\n\n4. In Table 1, it is unclear whether the \"YFCC15M\" and \"ResNet50\" refer to datasets or architectures. Could the authors clarify this distinction and make the table more explicit regarding which architecture is pretrained on which dataset?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The authors demonstrate the impact of using a pretrained model trained on noisy data. Increasing label noise in the pretraining dataset leads to a progressive degradation in performance when fine-tuning on downstream medical tasks.\n\n2. The authors explain this degradation by analyzing the skewness and kurtosis of feature embeddings and logit outputs. As noise levels increase, both statistics decline consistently.\n\n3. To address this issue, the authors propose a solution to resolve this issue by working on the two previous statistics . In addition, they introduce a disagreement regularization loss on the logits to enhance separability.\n\n4. Also the paper presents several experiments on multiple datasets and backbone architectures under varying levels of label noise to assess the effectiveness of the proposed approach.\n\n\n\n\n\n**Weaknessess:**\n\n1. The proposed method introduces two losses based on skewness and kurtosis based on predefined target values ($\\tau_{s}$ and $\\tau_{k}$) for each statistic. The authors mention that these values can be obtained from a “clean model” or set empirically. However, this aspect is not clearly defined or discussed in detail, which may limit the practical deployment of the method. For instance, it is unclear what authors means by a “clean” model in noisy or real-world settings. Additionally, the sensitivity of the method to the empirical selection of these two values is unclear.\n\n2. The paper lacks comparisons with other methods for dealing with noisy labels. Although it includes results with LP and  NML,  this comparison alone may not provide a comprehensive assessment of the method's robustness.\n\n3. Although the method demonstrates improved classification performance across datasets, it is unclear whether these gains are distributed evenly across all classes. The paper does not provide per-class performance metrics.", "was_revised": true, "final_justification": "The authors’ responses are convincing on several points and highlight the advantages of the proposed method. I have updated my score to accept.", "rebuttal": "## Reviewer PCzQ\nWe thank the reviewer PCzQ for your constructive feedback. Below we address each point in turn.\n\n**Q1:** How the target statistics (τₛ and τₖ) are chosen and how sensitive the method is to these values, noting that the explanation of \"obtained from a clean model\" or \"set empirically\" is unclear and could affect practical deployment.\n\n**A1:** We sincerely thank the reviewer for raising this point. To better clarify, we conducted additional sensitivity analyses on τₛ and τₖ.  \n\n|HAM10000|τs|-3.0|-1.0|0.0|1.0|3.0|\n|-|-|-|-|-|-|-|\n||Acc|56.9|56.9|56.9|56.9|56.8|\n\n|HAM10000|τk|-30.0|-10.0|0.0|10.0|30.0|\n|-|-|-|-|-|-|-|\n||Acc|56.9|56.8|56.9|56.8|56.6|\n\nThe results indicate that **SKD remains stable across a wide range of τₛ and τₖ values**. This observation aligns with our design intuition: the effectiveness of SKD mainly comes from counteracting representation collapse rather than relying on precise target values. Although we initially set τₛ and τₖ to the negative values of those measured from a clean model, we found that other reasonable settings (as long as they are not extreme) yield very similar performance. In practice, these values can indeed be set based on the skewness and kurtosis estimated from a clean model, but the method is **not sensitive to the exact values**—any setting that helps suppress representation collapse works well. Therefore, for simplicity and consistency, we fixed **τₛ = 0** and **τₖ = 0** in all main experiments.\n \n**Q2:** The paper lacks comparisons with other methods for dealing with noisy labels. Although it includes results with LP and NML, this comparison alone may not provide a comprehensive assessment of the method's robustness.\n\n**A2:** We appreciate the reviewer for raising this point. We would like to note that catastrophic inheritance in noisy medical pretraining is still a new research problem, and there are very few existing approaches tailored to this issue—especially for frozen foundation models.  \n\nTo address this gap and provide a more thorough comparison, we have expanded our baseline set to include **GCE (Generalized Cross-Entropy)[1]**, a widely used method from the noisy label learning field. Although GCE was not specifically developed for catastrophic inheritance, it serves as a strong benchmark for evaluating robustness. As illustrated below, SKD consistently surpasses all other methods:\n\n|Dataset|Noise(%)|LP|GCE|NML|SKD|\n|-|-|-|-|-|-|\n|Camelyon17|0|91.2|91.1|89.3|91.8|\n||5|90.7|91.4|92.4|92.5|\n||10|89.2|89.5|88.1|91.4|\n||20|88.6|89.1|90.5|89.1|\n||30|88.2|88.6|91.3|89.0|\n|HAM10000|0|55.7|55.7|54.7|58.3|\n||5|54.6|54.8|55.2|57.5|\n||10|54.5|54.5|54.2|56.9|\n||20|54.5|54.4|54.8|58.4|\n||30|53.9|54.1|50.7|57.5|\n\n**Q3:** Whether the performance gains are consistent across all classes or mainly driven by certain classes, noting that per-class performance metrics are not provided.\n\n**A3:** We thank the reviewer for this valuable suggestion. Following your advice, to evaluate whether the observed improvements are **uniformly distributed across classes**, we analyzed **per-class accuracy** on the Camelyon17 dataset under 10% label noise.\n\nAs shown in the table below, SKD outperforms LP, GCE, and NML across all classes, with no dominant trade-off on minority or majority categories. These results suggest that the performance gains are broadly distributed and that SKD achieves **consistent improvements across all class types**.\n\n|Datasets|Class|LP|GCE|NML|SKD|\n|-|-|-|-|-|-|\n|Camelyon17|Benign|90.1|90.1|89.1|92.4|\n||Malignant|88.3|88.9|87.1|90.6|\n\n**Q4:** The method is evaluated on ResNet-50 in Table 1. Have the authors tested it on other CNN or ViT architectures? Providing results on more diverse backbones would support the generalizability claims.\n\n**A4:** Thank you for this important suggestion. To further support the generalizability of our approach, we have conducted additional experiments on **diverse backbones, including large-scale foundation models**.  \n\n- For **vision tasks**, we evaluated SKD on **ViT-L**, a Transformer-based vision backbone that is substantially larger.  \n\n- For **natural language processing**, we extended our experiments to **GPT-2**, a large Transformer-based language backbone, and tested it on five representative biomedical NER datasets.  \n\nThe results, summarized below, show that **SKD consistently outperforms all baselines on both vision and NLP models**, with approximately **2% gains on ViT-L** tasks. This further confirms the **scalability and generality** of our method across diverse architectures and domains.\n\n|Model|Dataset|LP|GCE|NML|SKD|\n|-|-|-|-|-|-|\n|ViT-L|Camelyon17|91.6|92.3|91.8|93.4|\n||HAM10000|58.8|59.3|58.6|60.0|\n||NIHChestXray|43.5|44.3|43.8|45.9|\n \n|Model|Dataset|LP|GCE|NML|SKD|\n|-|-|-|-|-|-|\n|GPT-2|AnatEM|93.2|93.5|93.6|93.9|\n||BC2GM|91.3|91.6|91.5|91.8|\n||BC5CDR-chem|95.4|95.5|95.5|95.7|\n||BC5CDR-disease|95.2|95.2|95.3|95.4|\n||BioNLP09|91.6|91.8|91.8|92.2|\n\nThese findings demonstrate that **SKD generalizes well across fundamentally different backbones and task types**, including challenging large-scale foundation models.\n\n**Q4:** In Table 1, it is unclear whether the \"YFCC15M\" and \"ResNet50\" refer to datasets or architectures. Could the authors clarify this distinction and make the table more explicit regarding which architecture is pretrained on which dataset?\n\n**A4:** We thank the reviewer for pointing out this ambiguity. In Table 1, **\"ResNet-50\" refers to the backbone architecture**, and **\"YFCC15M\" refers to the dataset used for pretraining**.\n\nWe agree that the current formatting may lead to confusion. In the final version, we will revise the table caption and column labels to explicitly distinguish between the model architecture and the pretraining dataset.\n\nThank you again for the helpful suggestion to improve clarity.\n\n[1] Zhang, Zhilu, and Mert Sabuncu. \"Generalized cross entropy loss for training deep neural networks with noisy labels.\" Advances in neural information processing systems 31 (2018)."}]

Author rebuttal:

Meta-review: This paper studies catastrophic inheritance, poor out-of-distribution generalization, in the context of medical data subject to distribution shift (label noise). It provides a detailed analysis of how does the noise influence the representation space (by reducing skewness and kurtosis in the feature embeddings and logit outputs), and proposes a mitigation framework based on finetuning that regularizes higher order moments in both feature and output spaces. Empirical results demonstrate consistent gains across various medical imaging and biomedical NLP benchmarks compared to existing methods. 

Initial reviews raised several issues, including clarifications of the contributions beyond medical domain, limited empirical evidence (mid-sized models shown, missing baselines when training with noisy labels, and robustness under different noise). The rebuttal successfully addressed most of these concerns, leading to all reviewers recommending acceptance post-rebuttal. After careful deliberation, the decision was made to accept this paper - congratulations to the authors! It is crucial, however, that all the provided improvements and clarifications (experiments on large-scale foundation models, baseline comparisons when training with noisy labels, robustness under structured noise setting, and sensitivity analysis for the two hyperparameters) are incorporated in the final version of the paper.
Decision: Accept (poster)