Robust Fine-Tuning from Non-Robust Pretrained Models: Mitigating Suboptimal Transfer With Epsilon-Scheduling
Authors: Jonas Ngnawe
Maxime Heuillet
Sabyasachi Sahoo
Yann Pequignot
Ola Ahmad
Audrey Durand
Frederic Precioso
Christian Gagné
Venue: ICLR 2026 Poster (2026)
Keywords: robust fine-tuning
adversarial finetuning
adversarial robustness
fine-tuning
transfer learning
image classification
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We study robust fine-tuning (RFT) of non-robust pretrained models and show that robust objectives cause *suboptimal transfer*.  We propose *Epsilon-Scheduling*, which enables optimal transfer and improves *expected robustness*.

Abstract: Fine-tuning pretrained models is a standard and effective workflow in modern machine learning. However, robust fine-tuning (RFT), which aims to simultaneously achieve adaptation to a downstream task and robustness to adversarial examples, remains challenging. Despite the abundance of non-robust pretrained models in open-source repositories, their potential for RFT is less understood. We address this knowledge gap by systematically examining RFT from such non-robust models. Our experiments reveal that fine-tuning non-robust models with a robust objective, even under small perturbations, can lead to poor performance, a phenomenon that we dub _suboptimal transfer_. In challenging scenarios (eg, difficult tasks, high perturbation), the resulting performance can be so low that it may be considered a transfer failure. We find that fine-tuning using a robust objective impedes task adaptation at the beginning of training and eventually prevents optimal transfer. However, we propose a novel heuristic, _Epsilon-Scheduling_, a schedule over perturbation strength used during training that promotes optimal transfer. Additionally, we introduce _expected robustness_, a metric that captures performance across a range of perturbations, providing a more comprehensive evaluation of the accuracy-robustness trade-off of diverse models at test-time. Extensive experiments on wide range of configurations (six pretrained models and five datasets) show that _Epsilon-Scheduling_ successfully prevents _suboptimal transfer_ and consistently improves expected robustness.

Published as a conference paper at ICLR 2026
ROBUST FINE-TUNING FROM NON-ROBUST PRE- TRAINED MODELS: MITIGATING SUBOPTIMAL TRANSFER WITH EPSILON-SCHEDULING
Jonas Ngnawél22
Maxime Heuillet12
Sabyasachi Sahoo
Yann Pequignot
Ola Ahmad
Audrey Durand 1,2,5
Frédéric Precioso"
Christian Gagné
Université Laval (IID) Mila
CCortAIx Labs, Thales Digital Solutions
4 Univerrsité Côte d' Azur, CNRS, INRIA, 13S, Maasai
*Canada CIFAR AI Chair
ABSTRACT
Fine-tuning pretrained models is standard and effective workflow in modern machine learning. However, robust fine-tuning (RFT), which aims to simulta- neously achieve adaptation to downstream task and robustness to adversarial examples, remains challenging. Despite the abundance of non-robust pretrained models in open-source repositories, their potential for RFT is less understood. We address this knowledge gap by systematically examining RFT from such non- robust models. Our experiments reveal that fine-tuning non-robust models with a robust objective, even under small perturbations, can lead to poor performance, a phenomenon that we dub suboptimal transfer. In challenging scenarios (eg, diffi- cult tasks, high perturbation), the resulting performance can be so low that may be considered a transfer failure. We find that fine-tuning using robust objec- tive impedes task adaptation at the beginning of training and eventually prevents optimal transfer. However, propose novel heuristic, Sppilon-SSceddlligg a schedule over perturbation strength used during training that promotes optimal transfer. Additionally, we introduce expected robustness, a metric that captures performance across range of perturbations, providing more comprehensive evaluation of the accuracy-robustnes trade-off for diverse models at test time. Extensive experiments on wide range of configurations (six pretrained models and five datasets) show that transfer and consistently improves expected robustness.
successfully prevents suboptimal
INTRODUCTION
Fine-tuning pretrained models (backbones) on downstream task is the standard workflow in ma- chine learning, spanning natural language processing (Koroteev) (202]) and computer vision (Gold-) blum et al. 2023). This workflow offers clear benefits: (i) reusing a single foundation model
across tasks (Devlin et al., 2019). (ii) faster convergence, (iii) better generalization than training from scratch (Yosinski et al. 20144. and (iv) reduced computation (Weiss et al., 2016). especially
when labelled data is scarce Pan & Yang 2010).
However, in high-stakes applications, adversarial vulnerability remains major concern Biggio ct al., 2013, Goodfellow et al. 2015). Adversarial Training (AT) (Madry et al. 2018) and its vari-
ants (Zhang et al.) 2019: Wang al. 2020; Ding et al. 2020 Shafahi al. 2019a Wong al. 2020) are the most successful empirical defenses (Croce et al. 2021). Robust fine- toning (RFT the integration of these methods in fine-tuning on downstream tasks (Shafahi et al. 2019h Liu
et al. 2023; Xu et al. 2024 Hua et al. (024). Unlike standard fine-tuning, RFT must balance task adaptation with robustness trade-off that makes it considerably harder (Xu al. 2024).
Most prior works assume access to robust backbones (Hua et al. 2024 Liu et al. 2023 Xu et al., 2024 however, in practice, nearly all widely used pretrained models from public repositories are
Corresponding author; jonas.ngnawe. @ulaval.ca

---

Published as a conference paper at ICLR 2026
Swin
VIT
100
100
3.0
80
50
108
Clean Accuracy (%)
20.5
Clean Accuracy (%)
80
600
20.5
20.6
40.1
40
Dogs
Out
Cars
Aienal
-75.4
69.7
Perturbation Strength 6
Perturbation Strength E
Figure 1: RFT can lead to suboptimal transfer even when optimizing for small perturbation strenghts (Eg). The severity is highly model- and dataset-dependent.
non-robust (Wolf et al. 2020). Robust pretraining is costly and less common, and because pre- training pipelines typically prioritize broad general-purpose features, robustness is often treated as a property to be acquired downstream (Heuillet et al. 2025). This makes the development of RFT strategies for non-robust backbones not only consistent with current practice but also necessary to close the gap between research and deployment.
We adopt the standard RFT approach based classical adversarial training (Madry et al. 2018), which optimizes a robust objective using adversarial examples generated at a target perturbation strength (commonly 1/255 or 8/255 in the -nrm). We apply this procedure to robustly fine-tune various backbones across multiple datasets and perturbation levels. Our experiments reveal that, even for small perturbation strengths, standard RFT often leads suboptimal transfer: performance (clean accuracy) falls short of that achieved by standard fine-tuning (without perturbation) and is often too low to qualify backbone and the downstream task (Figure
a successful transfer. The severity of this effect depends on both the
When fine-tumie tinettuning on downstream task with robust objective results in near-random performance, the benefits of using pretrained model are diminished. This raises the question: do standard pre- trained models fail to offer a beneficial initialization for training a robust model? In this study, we explore the challenges associated with robust fine-tuning using standard pretrained models and pro- pose a novel approach to address these obstacles. In contrast to standard fine-tuning, where model adaptation to the downstream task occurs immediately, our study reveals that in robust fine-tuning, task adaptation is delayed until later epochs. This delay seems to eventually lead to suboptimal transfer, and we observe that the duration of the delay correlates negatively with transfer perfor- mance.
Based our findings, propose Epsilon- Dpsiilon-chedduling, sched- ule over the perturbation strength during RFT to encourage op- timal transfer. This novel heuristic is two-hinge linear sched- ule that begins with standard fine-tuning (zero perturbation) for early epochs and linearly increases the target perturbation at final epochs (see Figure 2). This strategy effectively prevents suboptimal transfer and improves both accuracy and robustness.
12223 3
ftt scheduler Standard
2-
nerturbation
Training Epoch
To better evaluate the fine-tuned models, we introduce a comple- mentary evaluation metric, dubbed expected robustness. This proposed metric evaluates the expectation of the model accuracy across the full perturbation range from zero (clean accuracy) to the target robustness threshold. The expected robustness provides a concise, yet comprehensive measure of the accuracy-robustie threat model. We demonstrate that it offers valuable insights for model selection. Under this metric, Apsilon-Scheduun conssiiitiitntttttttlyy consistently outperforms the standard nobust-finetunin even when worst-case robustness at the target threshold similar or lower.
Figure 2:
trade-off, grounded in practical
Summary of Contributions Our main contributions are: (i) We show that robust fine-tuning from non-robust backbones often leads to suboptimal transfer, even at small perturbation strengths, where performance can fall significantly below standard fine-tuning. (ii) We find that robust finetuning sults in task adaptation delay compared to standard finetuning and that this delay strongly correlates with suboptimal transfer. We propose
a two-hinge linear schedule adjust
"oode available at: http:///iiiiccooogggogooo/s///ss

---

Published as a conference paper at ICLR 2026
Swin- Care
Swin Caltach
Swin Aircraft
80
Clean Accuracy (5)
60
60
Ey x 255
40-
20
20
0
2
20
30
40
20
30
40
50
20
30
40
Epoch
Epoch
Epoch
Figure 3: RFT delays task adaptation. Validation clean accuracy under standard fine-tuning (5, = 0) and RFT-fix with Ey E [/2a0, 9/2s] on three datasets. crosses indicate the onset of task adaptation (when validation accuracy exceeds 5%). Stronger perturbations cause longer delays and more severe suboptimal transfer. See Section for analysis.
the training perturbation strength, which effectively mitigates the challenges associated with opti- mizing adversarial loss. (iv) We introduce expected robustness, new evaluation metric capturing the full accuracy-robustme robustness trade-off, and report performance using this metric for the first time. (v) Through extensive experiments, show that Apsilon-Scheduling consistently prevents suboptimal transfer and improves expected robustness across both moderate (4/255) and high (#/255) perturbation regimes.
RELATED WORK
Adversarial Robustness in Transfer Learning with Gobust-FineTuning There are two main ways to achieve adversarial robustness in Transfer Learning: Robust Distillation (Goldblum et al.) 2020 Dong et al. 2024) and Robust Fine- Tun-Tunnng. Prior works on RFT focus on robust backbones (Liu et al., 2023 Xu et al. 2024 Hua al. 2024). TWINS @Luu et al. 2023) employs two net- works with shared parameters to separately track pretraining and downstream batch statistics. How- ever. Liu et al. (203)) do not apply it to non-robust backbones claiming that "robust pre-training is indispensable to downstream robustness". AutoLoRA (Xu et al., 2024) disentangles natural and adversarial objectives using LoRA branch for the former and robust pretrained extractor for the latter, though it relies on TRADES loss (Zhang et al. 2019), which is harder to scale than standard adversarial training (Madry al. (0118) Xu al. (024)) show that robust pretrainning is necessary for AutoLoRA and register the worst performance for the non-robust pretraining. RoLi (Hua et al.) 2024) preserves robustness by initializing the classifier head via adversarial linear probing before performing RFT. (Hua et al. 2024) explicitly argue that robust pretraining is prerequisite for Rol ii by showing that linear probing with robust objective on non-robust backbone fails dramatically and therefore does not provide a good initialization. In summary, all these approaches assume ro- bust pretrained features. In contrast, we are the first to propose an RFT method targeting non non-robust backbones.
Tuning Perturbation Strength Adversarial Training Adapting the perturbation strength dur- ing training has been explored in various forms. Early work used a linear tump-up in Interval Bound Propagation (Gowal et al.] 2018). Ding et al. (2020) connected margin maximization to minimal adversarial loss, motivating adaptive, sample-specific perturbation strengths, though such instance- wise selection (Balaji et al.) 2019) is computationally costly. They also proposed PGDLS (PGD with Linear Scaling), which linearly increases perturbation strength but shows gains only at large perturbations (24/256). Other strategies include sampling the perturbation strength from a Beta dis- tribution (Chamon & Ribeiro 2020) and curriculum schemes that gradually increase the number of attack steps (Cai et al. 2018). Pang et al. (2021) reports that linear warmup provides limited gains in ResNets, whereas Debenedetti et al. (2023) finds that it improves both clean and robust accuracy in vision transformers. Unlike prior works, which apply to adversarial training from scratch, our study is on transfer learning. Our formulation generalizes linear warmup, and we show that the benefits consistently hold across tasks and architectures, including ResNets, using the new expected robustness metric.
Adversarial Defense Evaluation Standard evaluation (Croce et al. 202 compares clean and ro- bust accuracy at a target perturbation strength under strong attacks or ensembles (Carlini & Wagner, 2017 Madry et al., 2018 Croce & Hein 2020b Cinà et al. 2025), ,, yet it obfuscates what happens at intermediate perturbation strengths. A related recommendation is to check that accuracy decreases with stronger perturbations (Carlini al. 2019), but this test is unquantified and serves only as an
3

---

Published as a conference paper at ICLR 2026
R50 Caftech
4,-4055
G-42255
12-5555
ocheddrer
18.3%% schudurer (76.6% 55.7%)
55.0%
15-5 0% 29.781
achocueer (07.0%
8055
CO25
36055
Figure 4: The expected robustness metric offers a valuable perspective for model selection. The larger the area under the curve (shaded area), the higher the expected robustness. The values in the legend indicate the clean accuracy and the evaluation at Eg.
informal validation (Debenedetti et al. 2023 In contrast, our notion of expected robustness for- malizes this decrease and interpolates between clean and worst-case performance at a specific target perturbation strength. Another class of metrics interpolates between worst-case and average-case robustness (Rice et al. 202T] Li et al. 202T), the latter defined against random or natural perturba- tions (Hendrycks & Dietterich 2019 Han et al. (2024). However, this approach does not capture the trade-off between clean and worst-case performance.
3 BACKGROUND
Supervised Fine-Ture TunnnT Consider classification task to map instances x of a d-dimensional input space H to corresponding labels y in the set 3' {1, ..,,...,, K}. Unlike training from scratch to learn classifier fo X' + y from randomly initialized parameters I on training dataset drawn iid from a data distribution D on X' y. supervised fine-tuning uses pretrained feature extractor (backbone) he, X + Z that maps inputs to representation space 2 and randomly initialized classifier head CØ2 fine-tuning where both 02 and 0₂ are trainable parameters. The performance of the fine-tuned model is measured by its accuracy, i.c. the probability that a prediction is correct for an instance drawn from D. We will refer to this clean accuracy (or transfer accuracy).
+ y such that [10,,2,) = co2 heg This work focuses on full
Adversarial Training (AT) Given a target evaluation threshold for perturbation strength gooal (Eg for convenience) in fy-norm (|<=// (E,::)* p 0), adversarial training aims to train a classifier f such that it maximizes the robust accuracy Acce, (f). The robust accuracy is the probability that prediction remains correct for any input x under perturbation 8 of maximum norm Eg. Classical adversarial training (Madry et al. 2018)) minimizes the adversarial risk at perturbation threshold Eg as a surrogate objective for AJ);,(J):
Re (f) = E(x.g))> |
( max (c
where LoB is the cross-entropy loss. In practice, the empirical counterpart of the risk Rg (f) is minimized, and adversarial perturbations 8 are generated using a few iterations of Projected Gradient Descent (PGD) under fx-no no-morm constraint.
In this work, we refer to robust fine-tuning (RFT) as supervised fine-tuning with classical adversar- ial training. The standard practice in RFT for achieving robustness target evaluation threshold Eg is to directly minimize the empirical risk at eg throughout the fine-tuning process (Madry et al., 2018 Hua et al.] 2024). In this setup, the perturbation strength used generate adversarial exam- ples remains fixed at Eg across all fine-tuning epochs. We refer to this baseline strategy as RFT-fix (or fix).
CHARACTERIZING SUBOPTIMAL TRANSFERS IN ROBUST FINE-TUNING
When we perform robust fine-tuning with excessively strong perturbations e, that severely corrupt the inputs, clean accuracy is expected to reach chance level, the training samples will become unrecognisable, drifting away from the task data distribution D (Carlini et al., 2019). Yet, a question remains: how much clean accuracy degrades Eg increases given pretrained model and dataset? To investigate this question, we conduct an experiment with two non-robust backbones: SWIN (Liu et al. 2021) and ViT (Dosovitskiy et al. 2021), across five datasets: Cub (Wah et al. 2011.
Dogs
4

---

Published as a conference paper at ICLR 2026
Dataset
Alrenal
Caliech
Cirs
Dogs
Model
Serting
Clean
Adv:
E. adv.
Clean
Adv.
E. adv.
Clean
Adv.
E. adv.
Clean
Adv.
E.adv:
Clean
Adv.
ndv.
VT
lix
6.40
2,30
4,48
68.14
41.64
55.07
12.70
4.50
8.20
42.82
15.12
27.79
56,40
19.97
36.93
schod
70.69
41.62
SWIN
7:70
6.11
1997
57.16
41.87
61.89
36.99
44.17
sched
73,50
32.00
53.75
85,43
56.39
72.04
84.70
43.20
66.41
82.29
41.61
63.82
72.70
24.32
48.50
43.20
57.52
47,08
68.90
31,11
30.61
whod
78.40
38.00
59.40
89.41
61,45
77.23
58.90
57.70
75.85
85.17
44.99
67.30
78.39
26.31
53.19
R30
lix
8.40
2:30
4.36
67.47
30.02
53.74
4.20
2,001
1.49
49.19
19.35
55-38
57305
19.30
17.73
sched
53.10
11.10
29.40
76.55
34.74
55.67
70.00
19.30
43.44
70.06
19.59
43.62
69.11
15.94
41.11
CaiPiT
4.16
11.91
23001
1.74
1393
7.97
7.0
sched 69.50
33.90
52.79
74,83
46.64
60.99
86.70
58.60
75.01
74.35
38.67
55.54
63.17
20.87
41.05
ClipCNX
lix 3.10
2,50
2.82
61.76
42.13
31.54
2 8000
1.00
2.23
28.89
14.33
20.92
23,00
11.33
17.14
schod
79.06
70.85
23,85
48.19
Table 1: At moderate perturbation regime (4/255,, Spsillon-chedulling, mitigates suboptimal transfers
and consistently improves expected robustness. See Table for Eg 8/256
Khosla et al. 2011). Caltech (Griffin et al. 2007)) Cars Krrasse et al. 2013 and A Aircraft Maji al. 2013)). See Appendix A for details on backbones and their pretraining.
Suboptimal Transfer We apply RFT-fix with target perturbation strengths Eg in |//55, 9/255 and compare performance (clean accuracy) to standard fine-tuning (e, = 0). While low clean accuracies for RFT-Fix from non-robust backbones have been reported on Caltech at Eg = 8/255 (Liu et al. 2023: Hua et al., 2024), our experiments reveal broader picture. Figure I shows distinet trends in clean accuracy degradation as Eg increases. Even for tiny perturbations (Eg = 1/255), accuracy can drop by up 14% compared to the standard finetuning with 6g O. At the commonly used threshold Ey = 4/255, the minimum drop is 10%. In extreme cases, RFT-fix x fails to adapt effectively, with clean accuracy falling below 5%, making it practically unusable. We refer this phenomenon as suboptimal transfer, where RFT-fix yields a clean accuracy substantially lower than standard fine-tuning, at times to the point of ineffectiveness or failure.
Insights on Task Difficulty and Backbone Selection The severity of suboptimal transfer de- pends on both the backbone and the task, with the task being the more differentiating factor (see Figure Figure T}. Easier tasks (e.g., Ca 1 tech, wide variety of objects) exhibit smaller performance drops than more challenging ones (e.g., Ai rera ft. highly similar categories). As expected, the choice of SWIN as backbone is better than ViT (Hua et al. 2024). However, the notion of "task difficultyy" is also model-dependent: (Dogs < Cub for SWIN but Dogs > Cub for ViT). This demonstrates that task difficulty is property that emerges from the interaction between model and dataset (Zilly et al.) 2019; Jankowski et al. 2025), rather than by data features alone (Cao et al., 2024). Our experiments further suggest that with respect to adversarial robustness, the perturbation strength may also play a role: Dogs is easier than Cub for perturbations below Eg 4/255, but becomes comparable more difficult at higher strengths.
Robust Fine-Tuning Delays Task Adaptation Figure 3 shows that in standard fine-tuning (Eg = 0), the model adapts to the downstream task almost immediately - validation accuracy rises from the first epoch - since there is no robustness constraint that competes with task adaptation. With ég > O, RFT-fix distorts telkvrelevant features, which prevents early adaptation and delays the onset of task adaptation. For example, at Eg 4/255, adaptation begins around epoch 10 for Caltech, epoch 25 for Cars, and after epoch 30 for Aircraft. If we precisely measure the delay time as the epoch from which the validation accuracy starts being above 5%. the correlation between the delay times and the severity of the suboptimal transfer very high (above 90%, see Appendix BB. for de- tails). The delay shortens the effective adaptation period, likely leading suboptimal transfer with increased severity at higher perturbation thresholds. To the best of our knowledge, the delayed onset of task adaptation in robust fine- fine-tuning has not been previously reported, which is an interesting insight that could open up new opportunities for improvement.
5 EPSILON-SCHEDULINN AND EXPECTED ROBUSTNESS
The analysis in section demonstrates that a robust objective at high perturbation strengths is detri- mental in RFT from a non-robust pretrained model, eventually leading to suboptimal transfer, with performance subpar of what is expected from the pretraining advantage. Based on this finding, we propose Epssion-Schedullin
simple schedule over the perturbation strength mitigate this effect.
5

---

Published as a conference paper at ICLR 2026
Datsset Serting
Airenit Adv:
Caltech
Cars
Cuib
Dogs
Model
Cleon
E. adv..
Clean
Adv.
E. adv.
Clean
Adv.
E. adid.
Clean
Adv.
E. advdd
Clean
Adv.
E. odv.
WT
fix sched
2.00
2.50
48,95
19.52
31.43
3.00
2.00
2.74
17.40
2.80
8.56
8.64
2.88
27.72
35.18
9,79
5,79
SWIN
fix
4.20
2,70
3.47
68.87
58.10
53.40
13.20
5.00
8.66
45.89
13.60
28.56
46.05
11,08
26.69
sched
69.20
22,40
45.12
80.27
38,67
60.26
78.00
23.50
53.97
74,80
21.07
47.34
60.49
8.73
31.14
1.38
3.98
2.28
7:73
sched
75.00
28.90
50.90
84,99
41.82
64.92
85.60
35,90
65.04
80.69
24.28
53.07
68.94
9.78
36.51
R50
fix
1.30
0.50
0.74
53,59
26.78
50.95
1.50
1 1.20
1.34
70.89
8.27
17:84
27.
6.95
15.61
sched
42,50
5.30
20.38
67.56
25.01
44.03
57.10
8.50
29.56
59.49
8.68
29.95
50.89
6.92
25.26
CliPVI
fix
3.60
2,20
3.05
23.02
7.29
14.52
3.00
2.50
2.75
11:11
2.30
5.73
2.30
1.38
1.77
sched
65.80
25.40
44.84
70.68
33.70
51.67
84.70
38.60
64.47
67.64
18.05
41.79
54,28
8.94
27.78
ClipCNX
6.37
3.97
sched
59.09
56.83
90.00
77.14
73.58
22.75
62.67
11.36
Table 2: At high perturbation regime (*/255), RFT-fix mostly fails and Epsilon-Scheduling preserves performance. See Table for Eg = 4/255.
Spsilon-Schedulingg In contrast with RFT-fix, we perform RFT for target perturbation strength
Eg by minimizing an empirical counterpart of Rg (f) where follows a schedule during the fine-
tuning, given for each epoch & as a proportion (t) 2 0 of Egi a(t) = a(l)eg. We propose a two-hinge linear scheduler illustrated in Figure and defined by:
< Ti St T2 t T2.
a(I) =
Ta T1
This strategy begins by training over Ti epochs without robustness (8 = 0), then linearly increases
from & = to Eg over T, T epochs, to finally remain constant (2 E2,) from epoch T2. Note that this generalizes the linear warmups mentioned in prior work (Pang ct al. 2021: Debenedetti
et al. 2023) for 7i 0, T2 + Ti. while falling back to RFT-fix for Ti 72 O.
From a transfer learning perspective, this strategy can be viewed as follows: begin with task adap- tation, then gradually shift to the robust objective, and conclude by minimizing the target robust objective. Here, Ti denotes the adaptation phase, i.c., the time for the model to reach good clean accuracy, while T2 controls the steepness of the transition from 0 Eg. In the following, to this strategy as RFT-schedules e (or simply schedulers Considering that stronger perturba- tions make task adaptation harder (i.e., cause adaptation delays), RFT-schedu 1 e can be viewed as a curriculum learning strategy that first exposes the model to easier examples before gradually introducing harder ones (Cai al. 2018 Pang al. 202T Debenedetti et al. 2023).
refer
Expected Robustness Evaluation While RFT targets low adversarial risk Rg (f), models are usually evaluated both for clean accuracy Acco(f) and robust accuracy Acc,, (f). We propose to extend this classical evaluation to take into account intermediary perturbation strengths within the range [0, Eg] and define the expected robustness metric as the expectation under uniform distribution U of the accuracy over Eg):
1 1 Eg
("H Acc (f) de AUC,, (f),
1 Eg
A.c/o,e,1 (f) == EenU 0., 0.e,) [Ace (f)
where AUC², (f) represents the area under the accuracy curve from 0 to Eg (Figure 4 panels 2 and 4). This can be estimated using the trapezoidal rule to numerically approximate the integral. See Ap- pendix A for additional details. When comparing two models with similar accuracies, particularly in the presence of clean-robust trade-off, the distinct patterns observed intermediate perturba- tion strengths (Figure 4 panels 1 and 3) can inform model selection, which expected robustness summarizes quantitatively.
The expected robustness metric also evaluates performance under a more realistic threat model where inputs may or may not be altered. Clean accuracy corresponds to the idealized case where inputs are never perturbed, while robust accuracy at Eg assumes that all inputs are perturbed with a budget of Eg. In contrast, expected robustness estimates the accuracy when perturbations of size up to Sn are applied at random (here, uniformly). While our choice of uniform distribution serves as a good option from a case-aggosstii perspective, the distribution of perturbations can be tailored to align with the relevant threat model for a specific application. With the uniform distribution, each adversarial strength is weighted equally, however, for example, with a Dirac distribution centered at 0 (eg), it falls back to clean accuracy (worst case robust accuracy).

---

Published as a conference paper at ICLR 2026
6/255, Per Dataset
41255, Per Model
R255, Per Dataset
-8255, Per Model
60
50
52-
50
Accuracies 60
40
46
36
30-
Clean Accuracy
Accuracy (8)
Accuracy (%)
Clean Accuracy
Clyvil
schelller
Callech
Figure 5: Sheeduln-cheduuling mitigates suboptimal transfers and consistently improves expected robust- ness even when robust accuracy equivalent. Aggregated results from Table and Table
6 EXPERIMENTAL EVALUATION
We provide an overview of the experimental inclu-iicluiingg backbones, datasets, parameters Ti and T2 for Epsilon-Schhe Scheduling. and training and evaluation procedures. Additional details for reference and reproducibility are provided in Appendix A
Backbones: We perform experiments using six non-robust backbones, selected to cover two prominent architecture families (attention-based and convolutional-basee and two pretraining paradigms (supervised and multi-modal). Transformers: Swin-Base (CNX, (Liu et al. 2021) and ViT- (Dosovitskiy et al.] (021);; convolutional architectures: ConvNext- Baaee (Liu et al. 2022) and 50 et al. 2016);; CLIP models (Radford et al. 2021): CLIP- ViT and NenvNee.
Downstream Datasets: We evaluate fine-tuning performance five low-data downstream tasks: bird species classification on CUB-200-2000 (Wah et al. (2011). 200 classes), dog breed classifica- tion Stanford Dogs (Khosla et al. 2011), 120 classes). object recognition Caltech256 (Grif- fin et al. (2007). 257 classes), car model classification on Stanford Cars (Krause et al. (2013). 196 classes), and aircraft variant classification FGVC-Aireraft (Maji et al. (2013), 100 classes).
Choice of T1 and T2 for Epsiton-Scheduling To obtain values of Ti and T2 that are general enough for most cases, we use measurements from the most severe instance of suboptimal transfer in Section 4 (SWIN-Aircraft)
We define the adaptation phase as the epoch when validation accuracy reaches 90% of its final value, which occurs at epoch 11. Accordingly, we Zi 12, corresponding to about 25% of the total training epochs, sufficient for the model to reach high clean accuracy. Since the average aask-adaptation delay in RFT-fix is observed around epoch 37, we set T. = 37, i.e., roughly 75% of the total epochs, so that the model is trained with perturbation strengths smaller than Eg during the delay period.
Training and Evaluation We follow similar setup described in Hua et al. (022)). We train for 50 epochs and generate adversarial examples using APGD (Croce & Hein 2020b) (instead of PGD) with cross-entropy loss as in prior work (Singh al. 2023 Heuillet al.) 2025). which removes the need for manual tuning. The number of APGD steps is 7 for training and 10 for evaluation (Hua et al. 2024). Results of the adversarial accuracy is negligible (Figure 6). The evaluation is conducted in the foc -oomm which is the most widely studied norm in the literature (Croce al. 2021 Ngnawe et al. 2024
reported for the models at the end of training because overfitting
using two standard evaluation thresholds: Eg 4/255 (moderate perturbation) and Eg = =82255 (high perturbation). For each model, we report the clean accuracy (clean), APGD accuracy (adv.), and
the expected APGD accuracy (E. adv.) over the interval [0, Eg). We provide in Appendix [B .2 few additional results for SWIN, with the more expensive evaluation AutoAttack (Croce & Hein (020b),
a diverse ensemble of four attacks containing untargeted APGD-CE, targeted APGD-DLR, targeted
FAB (Croce& & Hein 2020a), and black-box Square Attack (Andriushchenko et al. 2020).
6.1 DASILON-SCIEDDLING PERFORMANCE IN RFT
Results are reported in Table I for the moderate perturbation regime (E, = 1/25s), in Table 2 for the high perturbation regime (s, 8/255), and in Figure 5 for the aggregated analysis.
7

---

Published as a conference paper at ICLR 2026
SWIN Cars Clean
SWIN Caltech Gean
SWIN Aircratt Clean
Clean Accuracy
42
scheduler standard
scheduler stancard
schediler standard FT
Adversarial (1g44)
SWIN Adversarial
(z)m4)
Adversarial (1)m4)
Robust Accuracy
Fix
fx
scheduler
schectler
FX
33
scheouter
standlard FT
Training Spoch
Training Spoch
Training Epoch
Figure 6: Epsilon-Scheduling preserves task adaptation while improving robustness (4255).
SWIN Clean
(2-))
Actada Systart
20
Advemtataal Lake
Chenness
Adrigunaal Liua
Clicn Loia
30
20
50
10
20
40
50
Traniig Epach
Training Spach
Figure 7: Apsiton--Schduuiing discovers different local optimum. Left: Illustrative example of the dif- ference between RFT-fix and RFT-s RFFTscheduller Center and righr: Evolution of validation loss (Clean and Adversarial) for the backbone on the Cars dataset with Eg 4/255.
Moderate Perturbation Regime (eg = 4/255) Table shows that while RFT-fi x often fails to transfer with low clean accuracy, RFT-scheduler achieves high clean accuracy for most models and maintains a decent adversarial accuracy. While RFT-fix sometimes achieves better adversarial accuracy (in out of 30 configurations), our scheduling strategy consistently yields higher clean and expected accuracy. These results show that even at moderate perturbations (*/255), Epsilon- Scheduling prevents the steep degradation incurred by RFT-fix, allowing models to retain strong clean performance while achieving improved or similar adversarial accuracy at non-trivial levels.
High Perturbation Regime (Eg = 8/255) At stronger perturbations, performance naturally de- clines, as shown in Table Z RFT-fix almost always fails to transfer, yielding very low accuracies. In contrast, RFT-s cheulerrr consistently improves clean accuracy and achieves higher expected robustness in all 30 configurations. For adversarial accuracy alone, the scheduler outperforms in 28 out of 30 cases.
Overall, shown in the aggregated results (Figure 5). Sehelun-Scheduling consistently improves expected robustness through significant gains in clean accuracy, even when a robustness-accume trade-off exists when robustness is similar across datasets and backbones. This contrasts with lin- ear warmups in adversarial training from scratch, which benefit vision transformers but not residual networks (Pang al. 2021: Debenedetti et al. 2023
6.2 RESULTS ANALYSIS
In order to further analyze els of task difficulty, as determined by the severity suboptimal transfer in Section Aircraft (high), Cars (medium), and Caltech (low).
we consider three datasets representing different lev-
promotes task adaptation while improving robustness Figure the validation accuracy during training. Standard fine-tuning quickly reaches high clean accuracy with- out robustness, whereas RFT-fix improves robustness but degrades clean accuracy. In contrast, RFT-schedules achieves high clean accuracy during the first stage and once perturbation strengths start passing above zero, robust accuracy rises while clean accuracy remains surprisingly high and stable.
8

---

Published as a conference paper at ICLR 2026
Fy Vas
Afreraft
Callech
Cirs
Aironaft
Caliich
Cars
Medel
Setting
Clean
Adv:
E, Adv.
Clium
Adv.
E. Adv:
Clean
Adv.
E, Adv.
Clean
Adv.
E. Adv.
Cleun
Adv.
E. Adv.
CClean
Adv.
E. Adv.
RobNIT
63,00
63.50
43.30
53,29
78.67
N2.22
57.73
70,86
08.39
78.00
12,90
49.30
62,23
62.05
65.10
51.60
25.10
23.00
37.91
42.88
73.16
41.12
41.89
57.48
62.10
78.60)
24.40
52.39
43.16
RODSWIN
74,00
56.50
66.19
82.64
61.12
12.39
83.10
60.10
72.42
21.60
43.30
$7.84
77.39
45.87
61.95
67.00
31.50
50.62
sched
77.20
49.90
61.99
$4.88
58,71
73.11
86,60
51.20
73.15
74,90
38.00
57.44
$1.86
43.43
63.88
64.20
39.10
63.88
RobCNN
78.30
62,00
70.74
55.20
87.92
67.11
17.03
78.53
86.10
68.40
78.09
14,70
48 06
02.22
79.37
49.87
65 65.30
80.30
50,00
67.51
70.38
RobR50
sched
63.10
66.90
41.50
56.30
52.14
51,14
71.25
75.49
47.60
48.83
61.79
00:29
78.10
72.10
44 30
45.30
02.29
59.61
60.30
66.90
24.70
32.50
44.26
45.62
05:25
71.90
36,60
34.08
52.35
50.19
59.66
76,80
24.30
22.00
50.37
41.89
Table 3:
Epsilon-Scheduulng robust backbones. The scheduler (sched) improves clean accuracy the cost of a decrease in robustness achieved in fix, but overall, the expected robustness is still improved.
Insight on the optimization process with Epsilon-Scheduling From an optimization standpoint, RFT-schedulle seems to converge to a distinct local minimum of the adversarial loss compared to the one achieved by RFT-fix, as illustrated in Figure 7 (left). The local minimum attained by RFT-schedulle is notably characterized by a lower value of the clean loss, whilst reaching a comparable value of the adversarial loss around epoch T2 37. Xu et al. (2024) found that the gradient of clean loss and the adversarial loss can point in opposite directions. Our experiments appear to confirm that this indeed happens when initializing at non-robust pretrained model. We observe that standard fine-tuning effectively minimizes the clean loss (Figure 7 center), but this comes at the expense of increasing the adversarial loss (Figure 7 right). In contrast, during the first 20 epochs, RFT-fix appears to struggle to reduce the adversarial loss while the clean loss remains nearly equal to the adversarial loss. However, the optimization trajectory of RFT-schedulee er initially aligns with that of standard fine-tuning, resulting in a low clean loss value. Subsequently, RFT-s AFT-schedulex effectively reduces the adversarial loss while maintaining minimal degradation in clean loss. This allows RFT-sched:: 1 e to achieve a balance that appears difficult for RFT- f x.
Effeet Epsilon-scheduling on robust backbones Table Ssolss shows that robust backbones are indeed more resilient to large perturbations under RFT-fix than their non-robust counterparts, reducing the Nevertheless, RFT-s ctedulehe still consistently boosts clean accu-
need for racy relative to RFT-fix, although at the cost of reduced robustness. On the easy task ec)), sub-
the trade-off is in favour of the scheduler. A key takeaway is that Apsilon-Scheduling stantially reduces the large clean- accuracy gap previously observed between RFT from non-robust backbones and their robust equivalents (L.uu et al. 2023 Hua et al. 2024), even if robustness at target cy is not fully matched.
6.3 ABLATION AND SENSITIVITY ANALYSIS
We summarize the effect of the hyperparameters Tj and T2 of Epsilon-Scheduuing in Ap- pendix C.1 follows. (i) T2 have the most significant influence with the control of steepness (4/(I=TT) Ti), Ti * T2). When T2 is close to Ti, clean accuracy decreases, whereas robust accuracy increases; this eventually leads suboptimal transfer. This is in line with the motivation for linear warmup in Debenedetti et al. 2023). although they do not study this effect. (ii) Increasing T2 in- creases clean accuracy, up to a threshold beyond which further increasing Tj has no apparent effect.
Special cases Only delaying the robust objective without follwing with gradual linear increase, i.c., a schedule that switches directly from to E, (Zi accuracy drops sharply to its initial value and does not recover during training unless Tj is small enough. Linear warmups (T₁ = 0, T2 > 0) without the delay still improve over fix, x, provided T5 is sufficiently large to ensure low steepness, thus having only very small perturbations early in training to avoid distorting features. The end-to-end linear schedule (Ti = 0, Ta = 50) comes close to the performance of the scheduler, though the latter remains superior.
0, Ti T2), is unstable: validation
Targeting directly the expected robustness A possible strategy to directly minimize the expected robustness risk 1e is via Monte Carlo estimation with single sample, which is equivalent to training with an e randomly drawn from U(0, en] at each epoch. Results in Appendix C show that the random uniform strategy (uniform often results in suboptimal transfer, except on relatively easy datasets such Caltech. This behaviour is normal: the expected perturbation strength is 64/2, making it likely that high perturbation levels appear early in training, thereby im- peding effective transfer.
9

---

Published as a conference paper at ICLR 2026
Vas
Model
Setting
Aircralt Adl-
Cahech
Can
Alneratt
Caieed
Can
Clean
E.Adv.
Cleas
Ads:
E.AdV
Cleas
Adk.
E.Adk
Cleus
Ads.
E.AdV:
Clous
Adv.
E.A.V
Cleas
Adv.
E.Adx
73.30
71.29
64.30
42.99
59.92
7.60
4.50
5.66
83.27
61.54
13.05
69.60
43.20
57.52
1.60
1.50
1.48
99.85
33.95
46.34
5.30
2,60
3,96
UNX
Schted
78.40
30.00
59.00
89.41
01.45
77:23
88.14
07:70
15.00
75.00
28,80
50,90
$4.99
40.82
04.11
35.60
35, 10
11
4660
79.00
31.60
36.61
90.14
58.30
76.26
89.00
50.30
72.56
76.20
23.60
49.65
86.48
38.30
64.35
85.20
29.00
$3.40
Table 4: Results
automated scheduler derived from analysis for SWIN and ConvNext
(CNX) on Aircraft, Caltech, and Cars, at e = 4/255 (left block) and 8/255 (right block).
Automated Scheduler Based our analysis, can derive simple automatic epsilon-schedullr (auto) driven by the validation accuracy. The procedure starts with = 0 and then initiates linear increase from T1 to the end of training. where Ti is automatically selected as the point at which the validation accuracy converges. Convergence is detected by monitoring the change in validation accuracy with patience of epochs and tolerance of 2%. Table A presents the results obtained with this automatic scheduler, which show that although it has less expected robustness compared to RFT-s chedulers effectively mitigates suboptimal transfer and provides strong performance across tasks.
CONCLUSION
We present the phenomenon of suboptimal transfer in robust fine-tuning from non-robust backbones and its connection with delayed task adaptation. To address this, we propose Sihilonn-cheduling. heuristic perturbation schedule over perturbation strength, and demonstrate that it effectively miti- gates this phenomenon, using commonly used metrics as well as the introduced expected robustness. Our findings underscore the practical potential of scheduling in robust transfer learning and motivate further exploration of fine-tuning strategies from non-robust pretrained backbones.
Limitations and Future Work. Although Epsilon-Scheduling yields significant improvements, robustness can still be limited even when clean accuracy is high, highlighting the potential for future research to further enhance performance. This work opens doors exploring other scheduling strategies, either heuristic, theoretically motivated, or learning-based. Extending the analysis to other vision tasks, such detection segmentation, applying the framework to parameter-efficient methods like LoRA, and investigating whether similar dynamics occur in other modalities, such as natural language processing, remain open questions. Studying these cases may require special considerations such as task-specific losses or hyperparameters.
From a theoretical perspective, although we offer explanation based on the discrepancy between the clean and robust loss landscapes in the vicinity of the pretrained model, deeper understanding of robust fine-tuning in this setting remains an open challenge. Our findings point to several important open problems: (i) What mechanisms underlie suboptimal transfer: is delayed task adaptation the only cause of suboptimal transfer or are there other factors? (ii) Can we find other approaches to mitigate delayed task adaptation different from Epsilon-SSheddulnng? (iii) What mathematical theory can account for suboptimal transfer or delayed task adaptation? (iv) If robust pretraining is not indispensable, what specific properties (if any) in pretraining really matter for downstream robustness and allow effective robust fine-tuning?
Pursuing these directions promises to unlock more effecctiee strategies for robust fine-tuning and yield more substantial progress towards achieving robustness in downstream tasks.
ACKNOWLEDGEMENTS
This work is supported by the DEEL Project CRDPJ 537462-18 funded by the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Consortium for Research and Innovation in Aerospace in Québec (CRIAQ), together with its industrial partners Thales Canada inc, Bell Textron Canada Limited, CAE inc and Bombardier inc
httpss//deel. http://eee.uuu.beee.ee....e quebec
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
Our study is designed to be fully reproducible. All backbones and datasets are publicly available, with details and references provided in Section 6 and Appendix A where we also cite the prior work underlying our design choices. Details on the estimation of expected robustness are given in Appendix
We provide an anonymized GitHub repository containing the implementation, the results of the hyperparameter optimization, all the data used generate the paper's figures and tables, and a script to reproduce them. The repository also includes step-by-step instructions for downloading datasets and pretrained models, creating Python environments, and launching experiments.
Finally, details on compute resources and expected run times are reported in Appendix
Link public github repository: http:///hwhh...oooop////////ppps
REFERENCES
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square at- tack: a query-efficient black-box adversarial attack via random search. In European conference on computer vision, 2020.
Yogesh Balaji, Tom Goldstein, and Judy Hoffman. Instance adaptive adversarial training: Improved accuracy tradeoffs in neural nets. arXiv preprint 1000001110.08051, 2019.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šmdii Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning test time. Joint European conference machine learning and knowledge discovery in databases, 2013.
Qi-Zhi Cai, Chang Liu, and Dawn Song. Curriculum adversarial training. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, 2018.
Bryan Bo Cao, Abhinav Sharma, Lawrence O'Gorman, Michael Coss, and Shubham Jain. A lightweight measure of classification difficulty from application dataset characteristics. In ternational Conference on Pattern Recognition, pp. 439-455 Springer, 2024.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy, 2017.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint 1902,v:0 2019.
Luiz Chamon and Alejandro Ribeiro. Probably approximately correct constrained learning. Ad- vances in Neural Information Processing Systems, 2020.
Antonio Emanuele Cinà, Jérôme Rony, Maura Pintor, Luca Demetrio. Ambra Demontis, Battista Biggio, Ismail Ben Ayed, and Fabio Roli. Attackbench: Evaluating gradient-based attacks for adversarial examples. In Proceedings of the AAAI Conference Artificial Intelligence, 2025.
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with fast adaptive boundary attack. In International Conference on Machine Learning. 2020a.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter- free attacks. In International conference on machine learning, 2020b.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flam- marion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversar- ial robustness benchmark. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
Edoardo Debenedetti, Vikash Sehwag, and Prateek Mittal. A light recipe to train robust vision transformers. In 2023 IEFE Conference Secure and Trustworthy Machine Learning, 2023.

---

Published as a conference paper at ICLR 2026
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, 2019.
Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma training: Direct input space margin maximization through adversarial training. In International Conference on Learning Reprexentations, 2020.
Junhao Dong, Piotr Koniusz, Junxi Chen, Jane Wang, and Yew-Soon Ong. Robust distillation via untargeted and targeted intermediate adversarial samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. In Proceedings of the AAAI conference on artificial intelligence, 2020.
Micah Goldblum, Hossein Souri, Renkun Ni, Manli Shu, Viraj Prabhu, Gowthami Somepalli, Prithvijit Chattopadhyay, Mark Ibrahim, Adrien Bardes, Judy Hoffman, et al. Battle of the back- bones: A large-scale comparison of pretrained models across computer vision tasks. Advances in Neural Information Processing Systems, 2023.
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, 2015.
Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Ue- sato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint urXiv:18100 12715, 2018.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. Technical
report, California Institute of Technology, 2007.
Tessa Han, Suraj Srinivas, and Himabindu Lakkaraju. Characterizing data point vulnerability as average-case robustness. Proceedings of the Fortieth Conference on Uncertainty Artificial Intelligence, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings the IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In International Conference on Learning Representations, 2019.
Maxime Heuillet, Rishika Bhagwatkar, Jonas Ngnawé, Yann Pequignot, Alexandre Larouche, Chris- tian Gagné, Irina Rish, Ola Ahmad, and Audrey Durand. A guide to robust generalization: The impact of architecture, pre-training, and optimization strategy. arXiv preprint arXiv:2508.1400 2025.
Andong Hua, Jindong Gu, Zhiyu Xue, Nicholas Carlini, Eric Wong, and Yao Qin. Initialization mat- ters for adversarial transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Gabriel Ilbarco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongscok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021.
Robert Jankowski, Filippo Radicchi. M Serrano. Marián Boguñá, and Santo Fortunato. Task complexity shapes internal representations and robustness in neural networks. arXiv preprint arXiv:2508. 05463, 2025.
12

---

Published as a conference paper at ICLR 2026
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-grained image categorization: Stanford dogs. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition Workshop Fine-Grained Visual Categorization, 2011.
Mikhail Koroteev. Bert: a review of applications in natural language processing and understand- ing. arXiv preprint arXiv:2103.11943. 2021.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshop on Fine-Grained Visual Categorization, 2013.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 2018
Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. Tilted empirical risk minimization. In International Conference on Learning Representations, 2021.
Chang Liu, Yinpeng Dong, Wenzhao Xiang, Xiao Yang, Hang Su, Jun Zhu, Yuefeng Chen, Yuan He, Hui Xue, and Shibao Zheng. A comprehensive study on robustness of image classification models:
Benchmarking and rethinking. International Journal of Computer Vision, 2025.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
Zhuang Liu, Hanzi Mao, Chao-Yuse Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEF/CVF Conference on Computer Vision and
Pattern Recognition, 2022.
Ziquan Liu, Yi Xu, Xiangyang Ji, and Antoni B Chan. Twins: A fine-tuning framework for improved transferability of adversarial robustness and generalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition Workshop on Fine- Grained Visual Categorization, 2013.
S. Marcel and R. Rodriguez. Torchvision image transformations, 2016.
Jonas Ngnawe, Sabyasachi Sahoo, Yann Batiste Pequignot, Frederic Precioso, and Christian Gagné. Detecting brittle decisions for free: Leveraging margin consistency in deep robust classifiers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge
and data engineering. 2010.
Tianyu Pang. Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu. Bag of tricks for adversarial training. In International Conference on Learning Representations, 2021.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Confer- ence on Machine Learning, 2021.
Leslie Rice, Anna Bair, Huan Zhang, and J Zico Kolter. Robustness between the worst and average case. Advances in Neural Information Processing Systems, 2021.
Christoph al. Schuhmann. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, Datasets and Bench- marks Track, 2022. arXiv:2210.08400.
13

---

Published as a conference paper at ICLR 2026
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Ad- vances in neural information processing systems, 2019a.
Ali Shafahi, Parsa Saadatpanah, Chen Zhu, Amin Ghiasi, Christoph Studer, David Jacobs, and Tom Goldstein. Adversarially robust transfer learning. arXiv preprint 1rXiv:1905.08232 2019b.
Naman Deep Singh, Francesco Croce, and Matthias Hein. Revisiting adversarial training for ima- genet: Architectures, training and generalization across threat models. Advances in Neural Infor- mation Processing Systems, 2023.
Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions Machine Learning Research, 2022.
Ole Tange. Gnu parallel 20230522 ('charles'), May 2023. URLlaatpss//ei..rr 5281/ zenodo. 7 7958356 GNU Parallel is a general parallelizer to run multiple serial command line programs in parallel without changing them.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. Technical report, California Institute of Technology, 2011.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International Conference on Learning Representations, 2020.
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big Data, 2016.
Ross Wightman, Hugo Touvron, and Herve Jegou. Resnet strikes back: An improved training procedure timm. In NeurlPS 2021 Workshop on ImageNet: Past, Present, and Future, 2021.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020.
Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training.
In International Conference on Learning Representations, 2020.
Xilie Xu, Jingfeng Zhang, and Mohan Kankanhalli. Autolora: automated robust fine-tuning
framework. In The Twelfth International Conference on Learning Representations, 2024.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances neural information processing systems, 2014.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning. 2019.
Julian Zilly, Lorenz Hetzel, Andrea Censi, and Emilio Frazzoli. Quantifying the effect of represen- tations task complexity. arXiv preprint 1rXiv:1912.080 2019.
14

---

Published as a conference paper at ICLR 2026
Shorthand (Contgurraton Name)
HuggingFace ID
References
VIT (vi..b,sup.nnkk)
timm//it_ www.basc.pppee patchTt. 224.augrog_in1
Steiner ot a 2022
SWIN (swin bysup.in22k
patch4 window? 224.ms in22k
CNX (conenext b.sup.in22k-ini innkk)))
timn/ecnvnest buse. ib in22k inik
Liu al.
2022
ClipVIT' wwl.pin.lppaloo2200
timm/vit base, parch clip. laiona
Ibarco of a.
22221
ClpCCNX (convnext_ wttps/___.ccii..ioon22...
Schulamann
2022)
R50 (resnct50.sup.nnlg)
timm/iesnet50all
Wigtmmnn
RobCNX (robustt_com _co.oneettttcccmccmmmmm cot.weett...uu.iiiibb
Lau cl al
21.35)
RobSWIN (robusi swin b.sup.in22kiinlk.
Liu el al 2025
RobR50 (pubust Serpeii.rrseetS.....iilllll
RobViT (eebus..i....uu..ikkk)
Liu ct al.
2025
Table 5: Pretrained non-robust and robust models used with HuggingFace IDs and references. The model name indicates the architecture (ivit, swin, convnext, resnet50)), the training type (sup: supervised, clip: multimodal), and the dataset: inlk ImageNet-Ik, in22k ImageNet-22k, in22k- inlk pretrained on ImageNet-224 22k then fine-tuned on ImageNet-Ik, laion2b = LAION-2B.
APPENDIX
A EXPERIMENTAL SETUP DETAILS
The non-robust backbones come from timm (PyTorch Image Models) and are publicly available on HuggingFace. The robust models are publicly released by ARES and can be accessed at githubb.cm//hh -ml/ares//// summary of all models used in this work is provided in T5
Pretrained Models
Training Splits and Data Augmentations We use train-val-test split from Hua al. (2024) for Caltech, Cub, Stanford Dogs; and from Heuillet et al. (2025) for Airoraft and Stanford Cars. Training augmentations consist of standard preprocessing methods commonly used for ImageNet and high-resolution images (Marcel & Rodriguez, 2016): random horizontal flips (p 0.5), color jitter (brightness, contrast, and saturation set 0.25), and random rotations. As done in Robustbench (Croce et al., 2021), images are resized to 224x224 with pixel values in the range [0, 1]. and data normalization and standardization are directly integrated into the model.
Hyperparameters Optimization We use the Adam W optimizer with a cosine learning rate sched- uler that includes a warmup period. select the learning rate and weight decay via hyperparameter optimization (HPO) based on clean accuracy. HPO is performed only for the fix setting, and the resulting hyperparameters are reused for the s schedul er setting to ensure a fair comparison. We search learning rate and weight decay values in the range 10 5 to 10-1, using the ASHAS scheduler, a variant of Hyperband Li et al. (2018).. The exploration budget is minutes for all configurations. HPO results available in the code repository.
Additional Evaluation Details The expected robustness is estimated by using the trapezoidal rule with evaluations made with steps 1/255, so for example with Eg 4/255:
3 Acc +. ( Ace - (/) 286
AUCa/asa (
2
T=U
Compute Resources Experiments were conducted using a 4xNVIDIA H100 GPU with 80GB of Memory. The duration for single case of robust fine-tuning ranges from approximately 15 minutes to one hour in distributed mode. An evaluation of robust accuracy for Eg from 0 16 can run in 5 minutes or less with APGD. The same evaluations with AutoAttack require a minimum of hours; the most expensive models can go up 24 hours more.
15

---

Published as a conference paper at ICLR 2026
Carg
Cars
Celeeh Nossh
Calach
Alrohh
Dalay Time (spacha)
40
30
20
6 Perturbation Strength
Perturbation Strength
Figure 8: Delay times increases with perturbation strength. We take the delay time here as the epoch from which the validation accuracy starts being above 5%. In some cases, the model never goes beyond this threshold until the end of training at 50 epochs. See Section
Final Epoch Accuracy (%6
Cars (corr: -0.96)
80
Cars (corr:-0..
80
Caltech (corr: -0.91)
Caltech (corr: -0.98)
Aircraft (corr: -0.94)
60
Aircraft (corr: -0.55)
Cub (corr: -0.86)
Cub (corr: -0.98)
Dogs (corr: -0.95)
Dogs (corr: -0.96)
40
20
10
20
30 (apochs)
40
50
10
20
30
40
50
(epochs)
Figure 9: Delay times strongly correlates with suboptimal transfer performance. The final vali- dation accuracy lower because task adaptation starts at later epochs. See Section
B ADDITIONAL RESULTS
B.1
TASK ADAPTATION DELAYS
We report detailed results the increase in task adaptation delay time with growing perturbation strength (Figure 8) as well as the correlation between delay times and the severity of suboptimal transfer (Figure
B.2 AUTOATTACK RESULTS
AutoAttack (Croce & Hein 2020b) is stronger and more diverse attack on the models, but is more expensive. We evaluate a few cases (SWIN on (Cars, Ai Aircrafe ft) x (4/255, $/255)). Results can be found in Table Figure 10 Although takes substantially more time, the results are close to evaluations with APGD.
C ABLATION AND SENSITIVITY ANALYSIS
C.I ABLATION AND SENSITIVITY ANALYSIS
To evaluate the influence of Ti and T2 the performance of Epsilon-Schedulling, we consider multiple configurations, illustrated in Figure II (moderate perturbation, 4/25s) and Figure 12 (high perturbation, $/2ss). These figures illustrate the evolution of validation losses and accuracies during training. along with test set evaluations, showcasing the distinct trends. The corresponding numeri- cal results the test set reported in Table 7
16

---

Published as a conference paper at ICLR 2026
Aircraft, C244285
Airoraft, 6,42235
Aircratt (y48/55
Alroraft, (y-2293
fy
0.3% 5.55
schdiwr (0.25. 66.5%
---- 2chel.la (73.8% 35.1h)
16.2% 1.25)
33-
6055
8D53
12055
14250
1055
4035
4053
N250
18055
19200
6055
0250
Carl 6,54055
Cara. 4ym5555
Cars, 58255
Cara, (,-0225
(66.7%
(6.5%, 2.4%)
sreddrr 78.5%, $2.851
22-
20-
16
achod.lo (34. 68.2%)
43
10
33
6059
14/000
2053
370
400m
LNCO3
4053
Figure 10: Evaluation with AutoAttack. Numerical values are in Table
Attack
Setting
Aircraft Adv.
Cars Adv.
Clean Acc
E, Adv.
Clean Acc
E, Adv.
fix
7.70
4.80
6.11
60.20
31.9
45.89
APGD
sched
73.80
32.00
53.75
84.70
43.20
66.41
1/255
fix 7.70
3.30
4.97
60.20
29.90
44.96
AutoAttack
sched
73.80
31.40
53.10
84.70
42.00
66.24
fix 4.20
2.70
3.47
6.50
3.2
4.49
APGD
sched
69.20
22.40
45.12
78.00
23.50
53.57
8/255
fix
4.20
0.80
2.16
6.50
2.40
3.97
AutoAAttack
sched
69.20
21.50
44.51
78.00
22.90
52.81
Table 6: AutoAttack results
C.2 DIRECT MINIMIZATION EXPECTED ROBUSTNESS
Since SesilonnScheduling consistently improves expected robustness, we compare with a direct min- imization of the expected robustness risk. The results in Table show Spsilon-Scheduling still superior, and the uniform strategy often leads to suboptimal transfer due to early sampling of high perturbations.
D STATISTICAL SIGNIFICANCE
We report paired (-test statistics comparing RFT-Fix and RFT-Scheduler at e = 4/255 and 8/255 in Table These tests assess whether performance differences between the two strategies are statistically significant across downstream tasks. A paired f-test measures whether the mean performance difference between methods is reliably non-zero; small p-values indicate that the observed differences are unlikely to occur by chance.
We also report the averages for each metric per model (Table 10) and per dataset (Table
E ADDITIONAL. RESULTS ON IMAGENETTE
We provide additional results for ImageNette in Table 12
17

---

Published as a conference paper at ICLR 2026
Eg = 4/255
Eg = 8/255
255
T1
T2
Clean
Adv.
E. Adv.
Clean
Adv.
E. Adv.
0
0
60.20
31.90
45.89
6.50
3.20
4.49
12
67.70
40.20
54.19
36.20
10.50
21.47
30
78.40
44.70
63.41
63.30
21.80
43.34
50
82.30
40.30
63.95
75.40
19.90
49.33
5
5
64.20
29.50
47.26
4.20
2.40
3.26
12
78.40
48.20
64.95
68.00
24.20
47.06
25
81.10
48.60
66.64
74.80
26.00
52.54
50
84.50
35.90
63.32
80.30
18.00
51.58
12
I2
1.90
1.40
1.74
1.30
1.30
1.28
30
83.00
47.30
67.09
78.10
26.60
55.06
37
84.70
43.20
66.41
78.00
23.50
53.57
50
84.80
35.80
63.25
81.10
16.60
51.79
25
25
0.80
0.80
0.80
0.80
0.80
0.80
37
84.00
39.50
64.61
78.40
21.00
51.51
50
84.30
24.80
57.46
81.00
12.10
46.94
Table 7: Effect of hyperparameters Tj and Tg. The training dynamics can be found Figure for Eg 4/255 and Figure 12 for Eg #/255. RFT-scheeuler reported in main text
12, T2 = 37).
f/258
1/255
Aireraft
Caltech
Curs
Aircraft
Caktech
Curs
Sotting
Clean
Adv.
Adv.
E.
7:20
4.30
6:11
7030
57.16
69.19
60.20
20.70
44.74
4.20
2.70
3.47
65.87
38.10
$3.40
13.20
5:00
K.65
SWIN
unifoms
30.10
7.90
18.57
83.45
55.49
70.27
70.90
35.30
54.68
7,80
2,00
4.85
76.74
37.31
57.34
53.70
11.30
30.16
sched
73,80
32.00
53.75
85.43
56.39
72.04
84.70
43.20
66.41
69.20
22.40
45.12
80.27
38.67
60.36
78.00
23.50
53.57
1.34
unifomm
41,50
22.04
74.95
8:00
27.
3.70
12:26
22.03
6.20
3.36
school
53.10
11.10
29.40
76.55
34.74
55.67
70.00
19.30
43.44
42,80
5.30
20.38
67.56
23.01
44.03
57.10
8.50
29.56
2.50
61.76
02.13
31.51
1.80
1.00
2.23
1.80
1.30
1,02
28.37
39.14
1.30
1,25
ClipCNX
unifonn
59.66
3.90
2,68
30.35
2.28
$1.19
Table 8: Epsilon-Scheduling still has better expected robustness than a direct optimization for the ex- pected robustness risk. In fact the approximation with uni form can still lead to suboptimal transfer.
6 4/255
6 8/255
Metric
n pairs
f-stat
p-value
t-stat
p-value
Clean Acc
30
7.823294
1.255470 x 10-8
12.387491
4.170254 x 10-13 13
Adv.
30
4.348780
1.540867 x 10-4
5,447550
7.317049 x 10-6
E. Adv.
30
6.595568
3.153155 x 10-7
9.270810
3.572919 10-10
Table 9: t-test statistics between RFT-fix and RFT-scheduler for two perturbation magnitudes.
18

---

Published as a conference paper at ICLR 2026
T,013
Yxar find
Spack
Todkrey Epers
Vaddng 5amm
Tram Ranch
Yearg Kprhh
5.3
Voarr Karch
Figure 11: Effect of hyperparameters on Cars for target Eg 9/255. The numerical results are presented Table Same plot at Ey = 8/255 are in Figure
Model
Setting
6 4/255
6 8/255
Clean
Ady.
E. Adv.
Clean
Ady.
E. Adv.
fix 37.29
16.89
26.49
15.59
5.84
10.12
VIT
scheduler
70.96
22.66
46.42
63.91
11.63
34.38
fix 56.40
32.08
44.35
35.64
14.21
24.16
SWIN
scheduler
79.78
39.50
60.90
72.55
22.87
47.49
fix 61.14
37.59
49.93
19.82
9.61
14.33
CNX
scheduler
84.05
45.69
66.59
79.04
28.12
54.09
fix 37.26
16,99
26.62
22.88
8.82
15,09
R50
scheduler
67.76
20.13
42.65
55.57
10.48
29.84
fix 12.73
5.74
8.85
8.59
3.13
5.56
ClipVIT
scheduler
73.77
39.14
57.08
68.62
24.94
46.11
fix 24.09
14.38
18.93
13.95
7.41
10.47
ClipCNX
scheduler
80.74
49.09
65.91
76.40
32.20
54.93
Table 10: Average per model of the clean accuracy, adversarial accuracy, and expected adversarial accuracy for e = 4/255 and e 8/255.
19

---

Published as a conference paper at ICLR 2026
Techkig
Tadiyg faoth
Yearng 1anch
Rravee lannh
Tratrng Epech
10
Thatrny Epack
Tonding
10
3.2
26
24
10
26
2-10
Figure 12: Effect of hyperparameters on SWIN-Care for target Eg 8/255. The numerical results can be found in Table 7
Dataset
Setting
= 4
= 8
Clean
Adv.
E. Adv.
Clean
Adv.
E. Adv.
fix 6.37
3.47
4.66
2.58
1.77
2.14
Aircraft
scheduler
69,23
29.82
49.69
64.83
20.52
41.34
fix 65,42
43.00
54.27
50.37
25.67
37.51
Caltech
scheduler
81.02
48.93
65.73
75.48
33.55
54.50
fix 25.73
14.22
19.99
4.65
2.50
3.45
Cars
scheduler
82.43
45.33
65.29
77.25
28.45
54.16
fix 47.24
23,56
35.07
19.45
5.26
11.38
Cub
scheduler
77.39
34.60
56.64
70.16
17.44
42.31
fix 46.00
18.82
31.99
19.95
5.66
11.95
Dogs
scheduler
70.82
21.50
45.61
59.02
8.59
30.06
Table 11: Average per dataset of the clean accuracy, adversarial accuracy, and expected adversarial accuracy for e = 4/255 and c = 8/255.
20

---

Published as a conference paper at ICLR 2026
Model
Setting
4/255
8/255
Clean
Adv.
E. Adv.
Clean
Adv.
E. Adv.
fix
97.15
85.07
92.22
94.19
66.17
82.43
SWIN
scheduler
98.62
85.48
93.67
97.50
69.08
86.93
fix 97.81
88.44
94.09
94.80
68.87
84.08
CNX
scheduler
99.29
88.23
95.36
98.27
71.32
89.03
Table 12: Imagenette results for 4/255 and 8/255.
21


Reviews:
[{"review_id": "7PDxpGGLnL", "reviewer": "Reviewer_AYiL", "summary": "This paper studies how to improve adversarial robustness via robust fine-tuning when the pre-trained model is vanilla/non-robust. Firstly, the authors identify the phenomenon, “suboptimal transfer”, when standard fine-tuning with adversarial training leads to low clean accuracy on downstream tasks. Since this can be attributed to the delayed task adaption by the authors when robust objectives are applied from the start of the fine-tuning, they propose Epsilon-Scheduling, a two-hinge linear schedule that starts with standard fine-tuning before gradually increasing perturbation strength to the target epsilon. Finally, the authors introduce expected robustness, a metric that evaluates model accuracy across the full perturbation range. Epsilon-Scheduling consistently shows improved expected robustness compared to fixed-perturbation RFT with the experimental results.", "questions": "- Could you provide some theoretical intuition/hypothesis on why robust objectives delay adaptation would occur? For example, is there a connection to the loss landscape geometry? Are there other factors that could be involved – size of the dataset, optimization objectives, etc? \n- Can you provide direct comparisons with TWINS, AutoLoRA, or RoLi with the same experimental setup? Even if they assume robust backbones, understanding relative performance would be valuable. \n- Can you add some error bars/confidence intervals with multiple runs? Since sometimes the values are very close to each other, this will help mitigate noises.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- **Studying an important and practical problem.** The focus of the paper on non-robust pretrained models is highly relevant. Nearly all publicly available pretrained foundation models are not adversarially trained. Improving robust fine-tuning techniques with those models would help model security in practice. \n- **Interesting insights on task adaptation delay and proposing simple yet effective solution.** The observation that robust fine-tuning delays task adaption (figure 3) is well-introduced. Especially, the strong correlation (>90%) between delay time and suboptimal transfer severity provides valuable understanding of the underlying mechanisms. This makes the idea of the proposed epsilon-scheduling method intuitive. It is an easy approach which generalizes prior linear warmup strategies. \n- **New evaluation metric, strong reproducibility.** Expected robustness helps measure the accuracy-robustness tradeoff than point estimates at a certain epsilon perturbation alone. It is well-grounded in a realistic threat model. Also, an anonymized code repository is provided with detailed hyperparameters and configurations to support reproducibility for future research in the field.", "weaknesses": "We thank the authors for submitting the paper to ICLR 2026! There are a few weaknesses listed below which I believe can make the paper better.\n- **Limited theoretical understanding.** The paper is purely based on empirical results. While those observations are compelling, the paper lacks theoretical analysis of why robust objectives delay task adaptation or why Epsilon-Scheduling finds better local minima. Some theoretical perspective can make the claim (such as figure 7) much more grounded.\n- **Lack of comparisons with other robust fine-tuning strategies.** There’s no comparison with other methods mentioned in the related works section such as TWINS, AutoLoRA, or RoLi. It would be interesting to see how it competes with other RFT methods and if they could be combined to make further improvements given the proposed scheduling method could be compatible with other architectural changes. \n- **Concerns on the expected robustness metric.** Since it is aiming to measure the trade-off between robustness and accuracy, it would be good to consider cases with very high clean accuracy, when the metric could be overemphasizing clean accuracy. Also, some discussion on how to make decisions when / whether to use this metric vs. traditional worse-case robustness would be informative for practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R2w2iVGSRq", "reviewer": "Reviewer_SXJd", "summary": "This paper studies robust fine-tuning (RFT) of non-robust pretrained models and observes suboptimal transfer, where robust objectives hinder task adaptation and degrade performance. The authors show that RFT initially delays learning, which results in the poor performance in clean performance. To address this, they propose Epsilon-Scheduling, a simple yet effective heuristic that gradually adjusts perturbation strength during training, improving both adaptation and robustness. Extensive experiments demonstrate consistent gains in expected robustness across multiple models and datasets.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and logically presented\nThe observation that RFT delays task adaptation is interesting, as it explains the low clean performance observed during RFT. \nThe proposed mitigation method is straightforward yet effective.", "weaknesses": "I noticed that in Figure 4, scheduler shows lower robustness compared to fixed, while in Table 1 it achieves higher adversarial robustness in most cases. Could the authors clarify this discrepancy or provide insights into why this happens?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8qmYtOwhsR", "reviewer": "Reviewer_Vfsn", "summary": "This paper explores finetuning a non-robust pretrained model with adversarial examples. The paper shows that direct adversarial finetuning, even with a small perturbation, leads to poor performance. To this end, the paper proposes epsilon-scheduling, which schedules perturbation strength as finetuning progresses. Experiments show that training curve under scheduling shows quicker clean accuracy convergence, achieves better tradeoff between accuracy and robustness, and leads to more optimal minima.", "questions": "1. Is there an automated way to decide when to start and end the scheduling based on the trianing progress?\n2. How would the scheduling help with more recent robust finetuning methods and on larger tasks?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Robust finetuning of non-robust pretrained model is an important research question. The proposed method is simple, solid, and interesting. The method is well motivated from the observation of \"delay time\", a novel observation for robust finetuning. Experimental results show effective improvements for most cases. The visualization of training curve and converged minima are helpful for the understanding of the process.", "weaknesses": "1. The evaluation is limited to PGD adversarial training, which is known to be prone to overfitting and diminished clean accuracy in the finetuning setting. More advanced finetuning method like AutoLoRA [1] or CLAT [2] should be attempted to see if the proposed scheduling is also effective.\n2. The current performance is highly relevant to baseline, task, and choice of hyperparameters. This makes the heuristic exploration of T1 T2 scheduling hard to be optimal for a new task and backbone. Some criteria to automatically decide when to start and end the epsilon scheduling based on things like gradient, loss, or accuracy would be helpful for better generalizability.\n3. The evaluation is limited to finetuning on small-scale tasks, which may be an overkill with the large pretrained model. Having some exploration on ImageNet variants would better show the effectiveness of the proposed method under more complicated settings.\n\n[1] Xu, X., Zhang, J., & Kankanhalli, M. (2023). Autolora: A parameter-free automated robust fine-tuning framework. arXiv preprint arXiv:2310.01818.\n\n[2] Gopal, B., Yang, H., Zhang, J., Horton, M., & Chen, Y. Boosting Adversarial Robustness with CLAT: Criticality Leveraged Adversarial Training. In Forty-second International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)