LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders
Authors: Borna khodabandeh
Amirabbas Afzali
Amirhossein Afsharrad
Seyed Shahabeddin Mousavi
Sanjay Lall
Sajjad Amini
Seyed-Mohsen Moosavi-Dezfooli
Venue: NeurIPS 2025 poster ()
Keywords: Adversarial finetuning
Adversarial Robustness
Unsupervised Learning
Visual Encoders
Constrained Optimization
Primary area: general_machine_learning
TL;DR: A new algorithm on unsupervised adversarial finetuning of visual encoder models via constrained optimization.

Abstract: Visual encoders have become fundamental components in modern computer vision pipelines. However, ensuring robustness against adversarial perturbations remains a critical challenge. Recent efforts have explored both supervised and unsupervised adversarial fine-tuning strategies. We identify two key limitations in these approaches: (i) they often suffer from instability, especially during the early stages of fine-tuning, resulting in suboptimal convergence and degraded performance on clean data, and (ii) they exhibit a suboptimal trade-off between robustness and clean data accuracy, hindering the simultaneous optimization of both objectives. To overcome these challenges, we propose **L**agrangian-**O**ptimized **R**obust **E**mbeddings (LORE), a novel unsupervised adversarial fine-tuning framework. LORE utilizes constrained optimization, which offers a principled approach to balancing competing goals, such as improving robustness while preserving nominal performance. By enforcing embedding-space proximity constraints, LORE effectively maintains clean data performance throughout adversarial fine-tuning. Extensive experiments show that LORE stabilizes training and significantly improves zero-shot adversarial robustness with minimal degradation in clean data accuracy. Furthermore, we demonstrate the effectiveness of the adversarially fine-tuned image encoder in out-of-distribution generalization and enhancing the interpretability of image embeddings. The code is available on [GitHub](https://github.com/Theborna/LORE-Lagrangian-Optimized-Robust-Embeddings).

LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders
Borna Khodabandeh
Amirabbas 2
Amirhossein Afsharrad1.2
Seyed Shahabeddin Mousavi Sanjay Lall"
Sajjad Amini
Seyed-Mohsen Moosavi-Dezfoolf
Stanford University
2 Aktus AI
UUniversity of Massachusetts Amherst AApple
{bornakh, afsharrad, la1l)@stanford. edu cumousav@cs.stanforii amirabbas. Erizali@aktun ai
samini@umasa edu
smoosavi@apple.commmm
Abstract
Visual encoders have become fundamental components in modern computer vision pipelines. However, ensuring robustness against adversarial perturbations remains a critical challenge. Recent efforts have explored both supervised and unsuper- vised adversarial fine- sinettuning strategies. We identify two key limitations in these approaches: (i) they often suffer from instability, especially during the early stages of fine-tuning, resulting in suboptimal convergence and degraded performance clean data, and (ii) they exhibit a suboptimal trade-off between robustness and clean data accuracy, hindering the simultaneous optimization of both objectives. To over- come these challenges, we propose Eegrrangian-pptimiie Robust Embeddings (LORE), novel unsupervised adversarial fine-tuning framework. LORE utilizes constrained optimization, which offers principled approach to balancing compet- ing goals, such as improving robustness while preserving nominal performance. By enforcing embedding-space space proximity constraints, LORE effectively maintains clean data performance throughout adversarial fine-tuning. Extensive experiments show that LORE stabilizes training and significantly improves zero-shot adver- sarial robustness with minimal degradation clean data accuracy. Furthermore, we demonstrate the effectiveness of the adversarially fine-tuned image encoder in out-of-distribution generalization and enhancing the interpretability image embeddings. The code is available on GitHub
Introduction
In recent years, embeddings from foundation Vision-Language Models (VLMss,, such as CLIP [Radford et al. 2021], have transformed downstream tasks, including classification [Conde and Turgutlu, 2021] object detection [Zhong al., [202], segmentation [Xu et al., [022]. and image generation [Saharia et al.) 2022). These models, trained large-scale web data, semantically align inputs from different modalities into a joint embedding space. enabling remarkable zero- shot generalization capabilities like zero-shot image classification, where virtually any class can be encoded via its textual description. [Radford et al. 2021 Qian and Hu 2024
However, visual encoders, such as CLIP models, exhibit significant vulnerabilities. Adversarial attacks [Goodfellow et al.) 20155) and backdoor vulnerabilities (Bai al. 2024] Liang et al. 2024 can lead to erroneous and misleading embeddings. These challenges highlight the need for robust alignment methods in large, pre-trained visual encoders. To address this, adversarial fine-tuning plays crucial role in enhancing the robustness of deep neural networks against adversarial examples
Borna Khodabandch and Amirabbas Afzali contributed equally to this work.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Pang et al. 2020 Kurakin et al. 2017 Madry et al., 2019]. Moreover, maintaining accuracy and correct embeddings is vital for downstream applications, as it ensures the model's versatility and reliability without compromising clean data performance, i.e., its nominal performance [Tiipras et al. 2019 [Dbbriban et al. 2022]. While recent works have proposed effective methods to enhance the robustness of visual encoders [Schlarmann et al. 2024 Mao et al., 2023]. they often struggle to manage the trade-off between robustness and nominal performance effectively. Therefore, principled approaches are required to explicitly balance this trade-off (Zhang al. 2019a].
In this work, we focus on the vision encoder and introduce Lagrangian iangrannginnOpttmmzzel Robust Embeddings (LORE), a novel unsupervised adversarial fine-tuning framework that effectively balances robustness and nominal performance through a novel constrained optimization framework. While we primarily target CLIP's image encoder, due to its widespread adoption and strong zero-shot capabilities, we also demonstrate the applicability of LORE to other foundation models such as DINOv2 [Oquab et al.. 2024]. LORE enforces proximity to reference model in the embedding space by leveraging the Lagrangian dual method [Bertsekas 1997]. enabling robustness improvements without compromising semantic fidelity on clean data.
Our main contributions are as follows: 1) We propose LORE, novel unsupervised adversarial fine-tuning framework that achieves superior empirical balance on the trade-off between robustness and nominal performance, with a hyperparameter (p) enabling principled control over this trade- off. 2) We demonstrate the effectiveness of LORE in image classification, significantly improving zero-shot adversarial robustness over CLIP while maintaining minimal degradation in clean accuracy. 3) We analyze the behavior of the adversarially fine-tuned LORR-CLIP encoder in terms of out- of-distribution generalization and show that it improves the interpretability of image embeddings, highlighting its advantages over other unsupervised adversarial fine-tuning baselines.
2 Related Work
Attacks and Defenses. Adversarial attacks exploit neural network vulnerabilities by crafting imperceptible perturbations, leading misclassifications [Goodfellow et al. 2015 Szegedy et al. 2014]. Defenses like adversarial training improve robustness but struggle 2019]. Balancing robustness and nominal performance remains challenging [Tsiiras et al. 2019 Zhang et al. 2019b). necessitating more versatile defenses, especially in multimodal settings, A principled approach managing the trade-off Constrained Optimization: [Robey et al. 2021] pioneered using a Lagrangian duality framework to enforce explicit constraints, avoiding heuristic loss weighting.
Adversarial Robustness:
with generalization [Madry
Robustness in Foundation Encoders. Recent studies have highlighted the vulnerability of foundation visual encoders, e.g. CLIP and DINOv2, to adversarial attacks, demonstrating that embeddings be disrupted [Zhang et al. 2024]. To address this, supervised defenses like TeCoA [Mao et al.) 2202)] and MMCoA Zhou al. 2024 improve robustness by aligning adversarial and clean embeddings, often relying on auxiliary information like text, using simplistic text descriptions, or attention maps. Yu
al. 2024], assuming normalized embeddings. PMG-AFT (Wang al. 2024 extends this approach by minimizing the distance between adversarial features and those of the pre-trained model. In the label-free setting, Unsupervised Adversarial Fine-Tuning seeks to achieve robustness by modifying contrastive losses, leading to methods like Robust Contrastive Learning (RoCL) [Kim et al. 2020
and the decoupled approach of DeACL [Zhang al. 2022 Among unsupervised fine-tuning
strategies for the CLIP vision encoder, FARE [Schlarmann et al. 2024] stands out as direct baseline. However, FARE and similar methods often lack explicit mechanisms to balance robustness and task performance, resulting in a suboptimal trade-off. Due to its unsupervised nature, we consider FARE a baseline for evaluating our proposed approach.
3 Background
Adversarial Attacks. Adversarial attacks aim to deceive machine learning models by adding imperceptible perturbations to inputs. A prominent example is the Projected Gradient Descent (PGD) attack, which iteratively maximizes loss function J (e.g., cross-entropy) to craft adversarial examples. Starting from an input x, PGD updates:
1/1 = Projecne)) (14 sign(( J(z
(1)
2

---

Clean 1DILONE)
A-05
1-5.0
80
p
Am02
Creey Accunny 290
20
hari
1-0.0
Baraaion
0
(a)
(b)
Figure 1: (a) clean data accuracy during adversarial fine-tuning with different training perturbation strengths E, using the loss from Eq. 2 Larger & values result in substantial drops clean data accuracy and early training instability. LORE (with p = 0.1) mitigates this effect, even (e 10), maintaining stable and higher clean data accuracy. (b) Pareto frontier comparison between naive regularization of Eq. 2 (blue, varying A) and LORE (orange, varying p). LORE yields strictly better empirical Pareto front, demonstrating superior trade-offs.
where y is the true label, 1 is the step size, and bounds the fox norm of the perturbation. The projection Projs(z.e) ensures x' stays within the allowed perturbation set. This procedure reveals the susceptibility of models like CLIP to adversarial manipulations.
Unsupervised Adversarial Fine-Tuning. FARE [Schlarmann al. 2024 proposed unsupervised method for fine-tuning CLIP vision encoder é. FARE's goal is to minimize the Euclidean distance between clean and perturbed embeddings under worst-case perturbation conditions:
CFARE(ÓO, 6:||6]| Se
max || + 6) Φe, (1)
(2)
where Φo, refers the original (frozen) CLIP encoder, and 5 represents the adversarial perturbation. Unlike methods like TeCOA and MMCoA, FARE does not require text inputs or labels, relying solely on the image encoder. This loss function aims to (i) enhance the robustness image embeddings generated by Φg. and (ii) ensure that the image encoder deg remains close to the original encoder 69. maintain performance clean data, preserving nominal performance.
Preliminary Experiments
To motivate approach, conduct series of preliminary experiments that reveal critical limita- tions in adversarial fine-tuning with the loss defined in Eq. (2). In particular, we examine two key challenges: (i) instability during early fine-tuning, and (ii) poor abbstness-accuraan trade-offs under naive regularization.
(i) Fig. la depicts the clean data accuracy of CLIP image encoder during adversarial fine-tuning using the loss function defined in (2). The results indicate substantial declines clean data accuracy, particularly at elevated perturbation strengths (e.g., = 8, 10 Additionally, there is notable instability during the early stages of training. This instability hampers optimal convergence and diminishes performance clean data. Consequently, the model may converge to local minima, losing significant pretrained knowledge, despite achieving robustness.
(ii) As straightforward approach to mitigate the degradation of clean data accuracy, one may introduce a naive regularization term to the loss ( aiming to minimize the following objective:
LEARE æ) Al|óe(æ) dogg()) (2)
where serves as a hyperparameter to regulate the strength of the regularization. We assessed the trade-off between clean and robust accuracy by plotting the Pareto frontier across various regularization strengths, as illustrated in Fig. [b The blue curve represents naive regularization (with
Throughout the paper, all values of refer to normalized lao perturbation bounds. i.e., e/255. For example. corresponds to perturbation magnitude of 2/255.

---

input
Algorithm Lagrangian-Optimizzz Robust Em- beddings (LORE)
O. 0o
Encoder
for each epoch do for batch ~ DD do < ATTACKALG((d(do, (x 6): 60. (+)))
Gg
for i - 1 KooKun do
Dou (2)
Libbst t- d(Øn, (x + 8), (2)(22))
Lelenn t- (00, (I)) 140 (x)) pm((a)
de (x)
prrn pan
Liotal + robust + (2)) Lekean
Appoot
9 24- 0 1/8 V Lootal
end for
+ Tww Lelean (w)
end for end for return 8, w
EmbeddingSpace
Figure 2: (Left)) Detailed steps of LORE. (Right) A conceptual overview of LORE. Given a clean anchor. LORE optimizes the encoder do, (1) to
input, the pre-trained model do, (ar) serves maintain proximity to Φo, (1) for clean inputs, while also improving robustness against adversarial perturbations 60, (z: + 5), which are pushed away from the anchor. The green region denotes the constraint margin, preserving semantic alignment and mitigating degradation in nominal performance.
varying λ), revealing steep trade-off, with robustness gains costing a considerable expense to clean data accuracy. Conversely, the proposed method, LORE (depicted by orange stars, with varying p), demonstrates superior Pareto front, consistently attaining higher robust accuracy at comparable or enhanced levels of clean data accuracy. Moreover, LORE maintains stable clean data accuracy throughout training (Fig. Ta even with large perturbation strengths.
5 LORE: Lagrangian-Optinine
Robust Embeddings
Our findings in Section Tighhlight the need for a principled method that both stabilizes adversarial fine-tuning and offers favorable robustness-accurae trade-off. Constrained learning offers structured approach balancing competing objectives like robustness and nominal accuracy, The constrained optimization problem for adversarial robustness in classification, explored in Robey ct al. [2021], can formulated as:
min BR max ((fox + (), #C6
SEA
(4)
s.t. e(fo(z), S p. for almost every (2, E
where is the loss function, 2 controls the nominal performance level, fo is the classifier parameterized by 0, and A denotes the of possible perturbations. The presence of infinitely many constraints indexed by the input space makes this a semi-infinite learning problem, which allows the application of tools from semi-infie amii-iniinite optimization theory.
Extending the optimization framework of an unsupervised setting, we propose enforcing proximity reference model ds, (e.g., pre-trained model) in the embedding space R$ where denotes the embedding dimension, thereby eliminating the dependence on y. Our formulation
min #eB
EL-D max d(oo(< + de, (z)) SEa
de,
s.t.
d(de(w), de (x)) pm(()) for almost every D.
Here, d: R2 x 1 + is divergence metric, and X + + is sample-specific tolerance margin, independent 8 used to modulate the constraint across inputs, defining a per-sample margin within which deviations setting m(x) = I, (2) ensures the constraint becomes ccale-invarian with respect to the original embedding magnitude.
considered acceptable. For example, when using Euclidean distance,
Solving the Constrained Problem. Constrained optimization deep learning presents significant challenges due to the high-dimensional and non-convex nature of neural networks. Therefore,

---

following [Robey et al. 2021] Chamon et al. 2020
Chamon and Ribeiro 2021], we leverage
Lagrangian duality to obtain approximate solutions for ( that generalize effectively with respect to
both adversarial and nominal performance. Specifically, we solve the dual optimization problem:
max min E well 0E
maxd (0o(x 6), de, (x)) (z) (do(())) Po. (x)) pm(()) SE
(6)
where d_ : + R is the dual network parameterized by w. with outputs made non non- non-negative via Softplus activation, which ensures valid Lagrange multipliers and avoids the sparse gradients of ReLU. This network simplifies the original optimization over all functions À : X - IR+ By jointly optimizing and w, our method achieves two key objectives: (i) enhancing the robustness of the model's embeddings against adversarial examples through the worst-case perturbation ö; and (ii) maintaining proximity to the reference model's ' embeddings on clean data, i.e., dee (z), to preserve nominal performance. Note that this adaptive framework fundamentally differs from regularization, and is automatically enforced if embeddings diverge, The theoretical gaps and limitations introduced by our formulation and parameterization are detailed in Appendix B
Practical Implementation. For computational efficiency, we implement the dual network as (ar) = (00, (z)),, where Fu is a lightweight two-layer MLP that takes the reference model's embeddings as input. This leverages readily available semantic information from de (2) for input-
dependent constraint weighting with no extra embedding computation. In Appendix G we provide a
detailed analysis of the X. architecture and its impact on performance. Specifically, we empirically demonstrate that modeling the dual variable as an input-dependent network, i.e., (z), significantly outperforms the input-independent variant A We instantiate the divergence metric as the squared &2 distance, d(21, #2) ||| 22 and set m=x) = l, (ar) to enforce scale-invariant constraints. Substituting these choices into our general framework yields the following optimization problem:
max min EX-D max SEA
Il (x + de (=) + (x) (1lo(()) Φ (x) plloe, (x)]]²)
(7)
WER #EB
The adversarial perturbation 5 is approximated using projected gradient descent (PGD) within an -bounded set. Problem ( employs a primal-dual framework to balance robustness and accuracy dynamically. The primal step optimize de for robustness while maintaining proximity da, on clean data. The dual step updates parameters w to strengthen constraints where d(())()), do (a)) exceeds pm()),, preventing overly conservative solutions. This approach is crucial for vision-language models, ensuring semantic alignment via embedding proximity, and eliminates heuristic loss weighting by treating robustness and accuracy as competing objectives within Lagrangian duality framework.
Intuitively, our scale-invarial constraint [z)) - óe, 313 S pllge, (x)113 preserves angular re- lationships. As detailed in Appendix [F] this bounds the deviation in cosine similarity Sc(u, v) uT [|[[[[]]]]|)) ensuring |Sc (0o, (z), v) Sc (z) u) < 2vp for any vector v E R". This preserves image-text alignment in vision-language models during adversarial fine-tuning.
Optimization. We optimize the LORE objective using an alternating training procedure. For each batch x sampled from the dataset D, we first generate adversarial examples 6. Then, we perform K steps of gradient descent to update the encoder parameters 0. After the K primal updates, we take a single gradient step to update the dual network parameters w, using the clean loss Cetean (as defined in Algorithm to dynamically balance robustness and clean performance, with K acting hyperparameter that controls the frequency of primal updates. The full procedure is summarized in Algorithm [ Although non-convex convergence guarantees
[ to guide the adjustment of (x)). This alternating update strategy enables the model
challenging. this alternating optimization is empirically stable and effective in our experiments. Ablation studies on the choice of K and constraint threshold p are provided in Appendix
6 Main Results
Although LORE and FARE optimize the identical adversarial loss, LORE's proximity constraints serve non-trivial, sample-based regularizer that steers training away from poor local minima, yielding consistently stronger robustness without sacrificing clean accuracy. We now present a suite of experiments substantiate this claim.
We begin by demonstrating how LORE enables controlled trade-off's between robustness and nominal performance. Next, we evaluate zero-shot image classification under both clean and adversarial
5

---

Accursry Comparison
Costne Siitrrty Companison
L2
Companionn
120
0.900
024
0275
4.80
922
Crsan
120
R110
casippt
0280
0.403
22.0
0.000
o20
75
0.50
8010 54
Hho
Figure 3: Influence of constraint threshold p on model behavior. As p increases, robustness improves the cost of clean data accuracy, cosine alignment, and embedding fidelity, highlighting the effec- tiveness of controlling the trade-off between robustness and fidelity by tuning p in LORE.
settings, We then analyze LORE's benefits for embedding interpretability and out-of-distribution robustness. Further results and ablations are provided in Appendix
Experimental Setup. To empirically validate our approach, we conduct the majority of our experiments using the ViT-B/32 CLIP model. In the second part of our evaluation (Section 6.3 we further assess the effectiveness of LORE on alternative architectures such ConvNeet-B CLIP and DINOv2. We use both ImageNet and ImageNet- 100, a curated subset of ImageNet, as training datasets throughout our experiments. Additional details for each experiment, including figures and tables, are provided in Appendix C. Throughout the results, superscripts denote the &oo perturbation bound used during adversarial fine-tuning; for instance, LORE² refers to a model trained with 2/255, We consistently compare our method with FARE, which serves as the unconstrained counterpart to LORE and also utilizes unsupervised adversarial fine-tuning. All models were trained for 2 epochs on ImageNet or 5 epochs on ImageNet-100. Experiments were conducted using 8 NVIDIA HGX H100 80GB GPUs.
6.1 Controlling the Robustness-Accurac Trade-off
The parameter p directly governs the allowed clean embedding deviation, offering a more precise control knob for the robustness-accuracy trade-off than heuristic weightings. While such trade-off's have been extensively studied in supervised adversarial robustness [Zhang et al. 2019a Xiao et al.
2024 Javanmard al. 2020). they remain underexplored in unsupervised adversarial fine-tuning.
We apply our constrained optimization framework to probe cobustness-accurag trade-offs. Using proximity constraint, restrict the hypothesis space from the full family of encoders H to the fidelity-constrained subset H, = o E H d(((#), de, (x)) Σ pm(() ()) for a.e. x E X}. Intuitively, H, is neighborhood of the clean encoder Φa, for small p. it's tight ball around (880 and p grows it expands until it recovers the full class H.
We then define the pointwise adversarial loss as ladv xy )o,)) A maxsea d(plx + 09, (x)), which induces the optimization
arg min EgED [faav x; 60,)] argmin Laav (dd (b0)).
(8)
"E H2
dEH,
Constraining the search space stabilizes training by avoiding extreme encoder shifts. However, it does so at the cost of theoretical optimality- since restricting H may exclude the global minimizer. In practice, the constrained formulation can nonetheless outperform the unconstrained one by steering clear of poor local minima.
Assumption 6.1. Assume d(.,-) .) is nonnegative metric the embedding space. Moreover, for any fixed reference embedding c, the map u + d(u, c) is La-Liscchitz in its first argument, i.c., (d(u),) - d(u2, c) < La || - w2]l for some norm || ||. Theorem 6.2 (Robustness Suboptimality Bounds). Let:
R min láāv(ó; do.). #EH
Rp min laav (o;; do..). oEH,
Since Hpp H. we have RS R. Moreover the suboptimality gap satisfies:
0 < Rp R S Vk (L, + L')
10, 0¹11

---

0m0
Averga
1010
tenson
20000or
Training Evaduation Epoilso 1d
(a)
(b)
Figure (a) Robustness common corruptions on ImageNet-C Embedding interpretability assessment based on average cosine similarity between clean image embeddings and 150 corresponding GPT4- agT4-generated text templates.
OOD evaluation. (b)
Here o= and o are the minimizers HH and Hpp respectively; Ti and T/ are their associated Lipschitz constants; and udversarial perturbations are -bounded by IR.,, (Proof in Appendix A
The bound shows that the excess loss R - R grows with both the attack radius E and the encoder shift || - o= I|. While we employ squared & geometry only affects the numerical
optimization convenience, this choice into the embedding norms and local Lipschitz terms.
Intuitively, increasing p relaxes the proximity constraint, granting the model more flexibility (and thus greater robustness) at the expense of nominal accuracy, whereas smaller keeps the model tightly aligned with the pre-trained encoder. favoring clean performance.
We empirically evaluate LORE's control over the robustness bbustness-acccraacy frontier by training ten models with varying p. Figure [b] shows the resulting Pareto frontier compared additive-regularizzane baselines (Eq.2), and Figure reports clean-data accuracy and adversarial success rate on the test These experiments confirm both the practical tightness of our theoretical bound and the superior trade-off achieved by our constrained-optiizzatii framework.
6.2 Out-of-Distribution Robustness and Embedding Interpretability
To assess how unsupervised adversarial fine-tuning affects
(OOD) robustness, we evaluate a ViT-B/32 CLIP model fine-tuned with LORE on ImageNet, using the ImageNet- C (Hendrycks and Dietterich 2019], which applies common corruptions to simulate distribution shifts. Since adversarial fine-tuning often reduces clean accuracy, it can lead to degraded OOD performance. As shown in Fig. 4a FARE suffers substantial drop under corruptions, whereas LORE² maintains much stronger generalization. fact, for certain perturbations (e.g.. jpeg compression), LORE² even outperforms the original pretrained CLIP model, underscoring its robustness to distribution shift.
Adversarial training also known to enhance embedding interpretability and cross-mode modal alignment [Croce al., 2025]. To measure this effect in the CLIP vision-language space, generate 150 natural-language templates using GPT-4 (c.g.. "This a photo of (J") to capture class semantics, then compute the average cosine similarity between each clean image embedding and its corresponding text embedding. Figure 4b shows that both FARE² and LORE² improve over the pretrained baseline, but LORE² consistently achieves higher alignment across all training and evaluation E settings, demonstrating superior semantic representation.
6.3 Image Classification
Zero-shot Image Classification. We evaluate adversarial robustness of the VIT-B/32 CLIP vision encoder on zero-shot benchmarks from CLIP -rrnhbbechmarf]]] all originally trained on ImageNet. For each dataset, we compare LORE against FARE under both clean and adversarial conditions.
hhtps//gtttub.com/LLltoAAAUU A benchmark

---

Table 1: Clean and adversarial accuracy for in-domain image classification on ImageNet- 100 across different CLIP vision encoders, evaluated using the APGD attack.
Table 2: Clean and adversarial accuracy for in-domain image classification on ImageNet across different DI- NOv2 variants. Adversarial robustness is evaluated using APGD attack.
Clean
2=1
C-2
FARE"
VIT-B/16
70,40
55.0
34,9
8.8
0.05
Method
Backbone
Clean
2
4
FARE
LORE
VII-Br16
VII-B/16
74.7
58.1
67.7
62.5
42,2
37.1
20.8
19.0
2.22
0.74
FARE
ViT-S/14
69.2
60.7
51.2
30.7
2.91
LORE
VIT-B/16
LORE
ViT-S/14
77.3
60.8
50.0
30,3
5.8
FARE
VIT-B332 LATON
65.4
41.0
19.0
2.02
0.02
FARE"
ViT-S/14
55.1
48.9
42.7
30.0
8.13
LORE
VII-H332 LAION
70.2
51.8
31,4
7.26
0.04
LORE
ViT-S/14
75.1
55.9
48.8
36.8
13.7
FARE
ViT B/3 LATON
52.7
36.7
23.4
6.72
0.20
LORE
VIT-B/32
FARE
VIT-B/14
78.3
71.9
64.1
44.0
6.51
FARE2
Com-NeXi-B
74.2
61.6
46.1
16.7
0.22
LORE
VIT-B/14
80.2
73.5
67.1
49.6
11.2
LORE
Com-NoXt-B
75.6
64.9
$2.4
25.6
1.04
FARE
VIT-B/14
69.4
63.8
57.8
44.1
16.0
FARE!
Com-NoXi-
70.6
61.6
12.3
12.7
6.48
LORE
ViT-B/14
80.5
65.0
59.7
48.5
21.8
ConeNoX--B
66.0
70%
LORE
LORE
60%
Clean Anturuyy (82
Robuat Acnuracy (34)
-396
5.%
6.
4.%%
20%
JL/h 15%
50%
Training Evaluation Epsiion
Training Evaluation Epsilon
Figure 5: Comparison of LORE and FARE across different training and evaluation perturbations (s). LORE consistently outperforms FARE, particularly at higher values, achieving higher robust accuracy while maintaining better clean performance, especially at higher perturbation intensities.
Robustness measured using the first two Auto/ttack methods [Croce and Hein 2020]: APGD with cross-entropy loss and APGD with targeted DLR loss, each run for 100 iterations under an foo constraint. shown in Table IT] the base CLIP model has negligible adversarial robustness, whereas LORE consistently outperforms FARE across all settings, improving adversarial accuracy without sacrificing clean performance.
In-domain Image Classification. We further assess LORE on in-domain tasks across various vision architectures. Table reports clean and adversarial accuracy on ImageNet-100 for several CLIP-style backbones using APGD. We also fine-tune the DINOv2 visual encoder with a fixed classification head on the full ImageNet dataset. Results in Table show that LORE enhances robustness not only for CLIP-style models but also for other foundation models such as DINOv2 [Oquab et al. 2024]. demonstrating its broad applicability across visual encoder architecture
Robustness at High Adversarial Intensity. A key challenge in improving robustness against large adversarial perturbations is the instability introduced by high training perturbation budgets (e). Training with large values often causes a sharp drop in nominal performance and a loss of alignment with the original model's 's semantics, making naive adversarial fine-tuning impractical in high-s regimes. As illustrated in Fig. Ia and further supported by higher evaluation in Fig. 5) this degradation is particularly evident in the early stages of training. showing that adversarial fine-tuning using the loss in Eq. (2) rapidly disrupts the proximity between clean inputs and their reference embeddings. This degradation intensifies with increasing E, resulting in more pronounced collapse in clean data accuracy. In contrast, Fig. 66 compares LORE and FARE, showing that LORE explicitly enforces constraint satisfaction over time, thereby preserving proximity between clean embeddings and their pre-trained references, while FARE exhibits progressive increase deviation during training. This constraint-guide optimization plays a critical role in mitigating early instability and enables LORE to achieve robust and stable learning even under high adversarial intensity.
(opendixxpproviccee further insight,
For each model, bold font highlights the best result, and underlined text denotes the second-ber result.

---

8.
Slop
Treniog SHDp
6.1
Distunee
6.
Distonce
Figure 6: The plot shows the & distance between the fine-tuned clean embedding. dbe (1), and the frozen reference embedding, (0, (I), over training steps. (Left) LORE* effectively regulates this deviation through its constraint-awes aware mechanism. gradually realigning the embeddings within p threshold. (Right) FARE". which lacks such constraint, exhibits a significant upward drift the & distance, indicating a collapse in nominal performance and emhedding fidelity.
Table 3: A comprehensive evaluation of clean and adversarial performance is conducted across various image classification datasets using the ViT-B/32 CLIP model. All models are trained on ImageNet and evaluated in zero-shot setting across diverse benchmarks. Our method consistently achieves a performance increase (f) relative the corresponding FARE models.
Zero shot datasets
Vision
ImageNer
CalTech
Cars
CIFAR 10
CIFAR100
DTD
EaroSAT
FGVC
Flowers
ImagoNer-S
PCAM
OxfordPers
STL-10
Average
Eval.
encoder
Image
Zero-shot
CLIP
$9.8
84.1
59.6
89.7
63.3
44.4
46.1
19.6
66.3
69.3
42.3
62,3
87.5
97.2
64.0
FARE
56.6
84.0
56.3
86.4
61.1
40.5
27.2
18.1
62.0
66.4
40.5
55.5
86.1
95.8
60.0
elenn
LORE
84.4
64.5
40.1
29.9
16.7
61.3
67.2
53.8
86.9
96.3
60.5
10.5
FARE
52.9
82.2
49.7
76.3
51.1
36.4
15.7
53.3
60.4
48.2
82.7
54.1
LORE
55.7
83.0
51.0
83.4
59.7
37.2
23.0
15.9
54.5
63.4
39.3
51.2
84.3
94.5
57.0
129
FARE
42.6
78.1
36.5
55.9
35.8
28.8
15.7
10.6
36,1
49.3
27.1
50.0
71.8
85.6
44.7
LORE
50.1
80.3
40.1
72.4
49.6
32.4
17.7
11.4
39.7
55.1
33.6
50.0
79.3
90.4
50.2
15.5
CLIP
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.1
0.0
0.0
0.0
0.0
FARE
68.6
16.1
35.6
22.5
6.1
30.6
34.4
22.5
24.7
55.8
35.6
==
LORE!
71.0
33.5
13.1
FARE
34.3
75.2
22.6
35.4
24.7
5.3
33,9
39.7
30.4
64.8
83.3
39.4
LORE3
39.3
76.3
23.3
67.0
43.2
26.4
12.3
6.5
35.8
42.4
26.4
39.0
68.5
85.6
42.5
13.1
FARE
33.2
74.8
21.4
44.9
28.0
22.4
14.0
5.8
27.3
37.1
21.3
50.2
59.3
77.7
37.2
LORE"
41.8
77.2
24.1
61.2
39.9
24.5
14.3
30.2
41.6
25.5
50.2
68.8
83.2
42.2
13.0
CLIP
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.1
0.0
0.0
0.0
0.0
FARE
43.5
14.7
6.8
14.1
15.9
17.0
==
LORE!
49.0
3.3
37.9
19.0
14.2
10.1
19.1
23.1
20.8
FARE
19.3
59.9
7.7
41.2
22.8
17.8
9.6
1.5
16.4
24.2
15.9
23.4
38.6
68.6
26.7
LORE
24.0
63.3
8.6
47.2
27.2
18.2
10.6
1.7
18.5
26.0
18.4
28.0
44.4
73.1
29.6
12.9
FARE
24.1
65.5
10.4
36.0
21.6
18.8
12.3
2.7
17.9
27.7
15.8
50.0
44.4
68.8
30.1
LORE
32.6
69.5
12.4
50.8
29.6
20.9
13.0
21.6
32.3
20.0
50.1
55.9
76.1
35.0
14.9
CLIP
0.0
0.0
0.0
0.0
0.0
0.0
0.0
FARE
6.3
0.0
2.0
2.3
0.1
2.6
5.3
1.0
LORE"
9.7
0.0
4.0
0.2
3.8
2.8
2.7
0.0
3.0
FARE
3.2
27.5
0.5
12.3
7.0
7.7
4.3
0.0
2.4
6.8
5.1
15.8
3.0
30.1
9.4
LORE
5.7
31.1
0.7
13.0
8.2
9.7
0.8
0.0
3.1
8.3
6.5
18.2
7.2
33.5
10.8
11.4
FARE
46.3
1.5
19.7
11.8
11.
10.2
6.4
11.4
45.2
16.2
46.1
18.2
LORE
54.2
8.0
48.4
25.5
22.5
14.3
6.4
Analysis: Additional Discussions and Ablation Studies
For further analysis, present additional discussions and ablations through the following Q&As.

---

Q: Why use sample-specific tolerance margin m(x) instead of fixed one?
Table 4: Ablation on Adaptive Margin specific tolerance margin, m(x) øl (z:)11,, which m(x). Adversarial Accuracy is evalu- dynamically adjusts the allowable perturbation based on the feature magnitude of each input sample.
A: The core of the LORE framework relies on a sample-
ated at the same E as training.
Table 4 compares LORE with the adaptive margin (LORE m(x) p 0.1) against LORE with several fixed margins (p (p= 0.6, 0.8, and 1.5) as well as the FARE base- line. Experiments were conducted using ViT-B/32 CLIP models fine-tuned for 5 epochs on ImageNet-100. The results show that fixed margins are either overly restric- tive, with p = 0.6 severely degrading = 4 robustness, or overly loose, with p = 1.5 leading to performance similar to FARE. In contrast, the adaptive margin LOREm(a) con- sistently achieves the best robust accuracy, validating our design choice.
Method
p
Clean
Robust
LORE m(x)
0.1
68.81
33.71
LORE
0.6
70.12
18.65
LORE LORE
0.8
69.65
29.45
1.5
63.45
25,47
FARE
62.23
23.15
LORE ma(a))
0.1
66.91
17.58
LORE
0.6
67.21
1.23
LORE
0.8
64.32
14.68
LORE
1.5
55.24
11.74
Further discussion and results are Appendix J.6
FARE
53.68
12.87
Q: Can LORE's proximity principle be extended supervised adversarial fine-tuning?
A: We found that LORE's s core principle of using proximity constraints to maintain clean performance during adversar- ial fine-tuning, is transferable and highly effective in the supervised setting. This demonstrates the method's d's broader applicability beyond its original unsupervised design. We applied LORE's s embedding space proximity regular- ization (E2) to TeCoA MMa et al.) [2023), creating the com- bined method TeCoA + E2. We also explored a variant constraining the KI. divergence of the model's output dis- tribution (TeCoA + KI.).
Table 5: Extension to Supervised Fine- Tuning. Adversarial Accuracy is eval- uated at the same E as training.
Method
Clean
Robust
TeCoA²
60.04
35.94
TeCoA + E2
75.97
39.12
TeCoA KL
66.50
39.53
TeCoA*
49.55
19.8
Table 5 presents the results, showing that TeCoA + l consistently achieves the highest clean accuracy, with mas- sive gains at high adversarial training budgets. Experi- ments were performed using ViT-B/32 CLIP models fine- tuned for 5 epochs ImageNet- 100. This validates that proximity-based constraints, implemented via LORE's La- grangian framework, are generally effective mechanism for stabilizing and improving both clean and robust perfor- mance in diverse adversarial fine-tuning scenarios. Further discussion and results are in Appendix J.7
TeCoA + L2
73.12
49.12
TeCoA* KL
56.01
22.18
TeCoA*
30.22
2.35
TeCoA* +
67.23
4.23
TeCoA* KI.
38.06
2.82
7 Conclusion, Limitations and Future work
Summary. We proposed LORE, an unsupervised adversarial fine-tuning framework that enhances the robustness of visual encoders while preserving nominal performance without relying heuristic loss weighting. Extensive experiments show that LORE consistently outperforms FARE, particularly under stronger attacks, achieving superior robustness with better clean data accuracy. LORE's robust visual encoders improve reliability in critical applications, fostering trust in AI. Maintaining high, clean data accuracy ensures effective performance in standard operational environments.
Limitations and future work. While LORE is effective, it has several limitations that suggest directions for future work. (I) A deeper theoretical analysis of the trade-offs in unsupervised adversarial fine- tuning is needed; our constrained framework provides a useful starting point. (2) The effectiveness and performance ceiling of LORE, which relies on the pretrained model as a fixed anchor (do,). are inherently limited by the quality and fidelity of that frozen reference model. (3) Our use of a neural network to model Lagrange multipliers is heuristic; better parameterizations could improve efficiency and reduce duality gaps. (4) While we adopt Lagrangian duality with manageable gaps, alternative constrained optimization techniques may offer stronger guarantees. Future work could also explore supervised LORE variants.
10

---

References
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square
attack: query-efficieet black-box adversarial attack via random search, 2020. URL https
//arxiv. wwwrv..org/ab//9112.. 00049
Jiawang Bai, Kuofeng Gao, Shaobo Min, Shu-Tao Xia, Zhifeng Li, and Wei Liu. BadCLIP: Trigger-
Aware Prompt Learning for Backdoor Attacks on CLIP, March 2024. URL http:///rxiv org/ arXiv:2311.6619 [cs].
abs/2311. 16194
D. P. Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3): 334-334, Mar 1997. ISSN 1476-9360. doi: 10. 100.00///pagrave.josso666000 URL https
7/doi. orr.ioorg11... 1057 wtt.:/1101.///aaaeeeeeppp pttp:/aalgavv.rrrr. -60.26044255 2600425
Luiz O. Chamon and Alejandro Ribeiro. Probably approximately correct constrained learning.
2021. /tRLppttps://arxi.... org abs/ 05487
Luiz F. O. Chamon, Santiago Paternain, Miguel Calvo-Fullana, and Alejandro Ribeiro. The empirical
duality gap of constrained statistical learning. ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8374 8378, 2020. URLhttps
7/api. seemanticscholaa. org/ :
Marcos V. Conde and Kerem Turgutlu. CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classification. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3951-3955 June 2021. doi: 10.1 119/CC/PR530098.... 2021 1.0044.
URL http:://axiv. org/ /tg/abs///// abs//as///// 2204. 14244 arXiv:2204.1422 [cs].
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustmess with an ensemble of diverse parameter- attacks. Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 2206-2216. PMLR, 13-18 Jul 2020. URL /ttt/tsss///prreeediin
mlr. a.r.press/vvii wwtpssssvvv/// crt.e20b.bb.ooooo vttp///cecccooebbbbbb html
Francesco Croce, Christian Schlarmann, Naman Deep Singh, and Matthias Hein. Adversarially robust clip models can induce better (robust) perceptual metrics. In SaTML, 2025.
Edgar Dobriban, Hamed Hassani, David Hong, and Alexander Robey, Provable tradeoff's in adversar- ially robust classification, 2022. URL https: org abs//b/s/6660 201.220000000660011
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2015. URL htap///ppp//axxvooo/////////2122
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations, 2019. URL http:/ttt::/ariiv.orrr org/
Adel Javanmard, Mahdi Soltanolkotabi, and Hamed Hassani. Precise tradeoffs in adversarial training for linear regression, 2020. / wttpo//aa/b/00..1117777.
Minscon Kim. Jihoon Tack, and Sung Ju Hwang. Adversarial self-supervised contrastive learning, 2020. htt // //txxvxxvvvva////a// org/ ab0/ 07589
Antoni Kowalczuk, Jan Dubiński, Atiyeh Ashari Ghomi, Yi Sui, George Stein, Jiapeng Wu, Jesse C. Cresswell, Franziska Boenisch, and Adam Dziedzie. Benchmarking robust self supervise learning across diverse downstream tasks, 2024. URL https://arxiv..rrg abwp////..... 12588
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale, 2017. URL tittps://arxinn httpaaawww....r//bba http//bb//1111 1etps//111...11666661
Siyuan Liang, Jiawei Liang, Tianyu Pang, Chao Du, Aishan Liu, Fe-Chien Chang, and Xiaochun
Cao. Revisiting backdoor attacks against large vision-language models, 2024. URL https
//arxiv. .rgg/v.oor//aoo www.aa/.....2. 18844
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, 2019. URI /tattps://arxiv.... org/ abs/ 06083
11

---

Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models, 2023. URL https://aaxxi...rrgg wttp://wwwww....///////./...
16
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2024. URL httptt:tt///rtte
org/ 2300/ 2304. 07193
Tianyu Pang, Kun Xu, Yinpeng Dong, Chao Du Ning Chen, and Jun Zhu. Rethinking softmax cross- entropy loss for adversarial robustness, 2020. URL https:///axxii wwt.://gw.bbb....noooo 10626
Qi Qian and Juhua Hu. Online zero-shot classification with clip, 2024, URL https://arxiv.. abb/2408. 13320
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pumela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021. URL
http://arxiv. http://axxiio.../aaaeo hwtpgg/ab/22103 00020 arXiv: 103,00020 [cs].
Alexander Robey, Luiz F. O. Chamon, George J. Pappas, Hamed Hassani, and Alejandro Ribeiro. Adversarial robustness with semi-infinite constrained learning, 2021. URL /RL.tttss:////// httpt:t/atxtii......///
//abs/2110.1574
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. 2022. URL httph:tt////aitti http://aaxxvv..r/abbbaaa barg//bb/// 05. 11487
Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. ICML, 2024.
Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks, 2014. URL httpps:/ wrg. /bb
s/1312. 6199
Dimitris Tsipras. Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry, Robustness may be odds with accuracy, 2019. URL https://aaxxi...tt. org/ hargg//b///// 18050 12152
Turner,
Sibo Wang, Jie Zhang, Zheng Yuan, and Shiguang Shan. Pre-trained model guided fine-tuning for zero-shot adversarial robustness. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14502-245111 2024. doi: 10.109/CVVV6373322224..22
Jiancong Xiao, Ruoyu Sun, Qi Long, and Weijie J. Su. Bridging the gap: Rademacher complexity in robust and standard generalization, 2024. URL huttts://arrai http:///axxxvvoooaaa/ htrg/abaa/406.... 05372
Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model, 2022. 14757
URL
Lu Yu, Haiyang Zhang, and Changsheng Xu. Text-guided attention is all you need for zero-shot robustness in vision-language models. In A. Globerson, I. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 96424-96448. Curran Associates, Inc., 2024. https p/tpcceedp////rooroed///sss/s neurips. cc
fww.flllepppe httpg//ie..aeeeeo.....cc..c.oooe Papp:eaeseeeccttttooooeeeeeeeeee Paper--ooreeeeeeeeee Confe rence. pdf
Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Axi Niu, Jiu Feng, Chang D. Yoo, and In So Kweon. Decoupled adversarial contrastive learning for self-supervised adversarial robustness,
2022. URL/nttps:/ // arxiv. org/ abs//2bs/7722 10899
12

---

Hongyang Zhang. Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustmess and accuracy. Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 7472-7482, PMIR, 09-15 Jun 2019a.
mlr. prwss/ /v/sss/vv///// http//999/2//1//1/ zhang 19p html
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy, 2019b. URL https //arxiv. org/ abs 1901 08573
Tingwei Zhang, Rishi Jha, Eugene Bagdasaryan, and Vitaly Shmatikov. Adversarial illusions in multi-modal embeddings, 2024. URL
11804
Yiwu Zhong, Jianwei Yang. Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. RegionCLIP: Region- based Language-Imase Pretraining, December 2021. URL arXiv:2112.09 [es].
org http//ab////1/11111 09106
Wanqi Zhou, Shuanghao Bai, Danilo P. Mandic, Qibin Zhao, and Badong Chen. Revisiting the adversarial robustness of vision language models; multimodal perspective, 2024. URL https
// //arxiv... org/ abaaab///a 2204/2404.000 19287
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction accurately reflect the paper's contributions and scope. LORE, novel unsupervised adversarial fine-tuning framework, enhances robustness while maintaining model fidelity, as supported by the experimental results.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Our approach entails certain limitations. The limitations of the work are discussed in the conclusion section of the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
The paper provides the full set of assumptions and complete proofs for its theoretical results, specifically in Section where the Lagrangian Multiplier Optimization is derived, and in the Appendix, which contains further details and proofs for the propositions. While the paper includes theoretical results, it is not heavily theoretical paper. Guidelines:
Justification:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The paper provides sufficient details to reproduce the main experimental
results. o comprehensive information on the datasets used, implementation specifics, and evaluation metrics. Furthermore, the authors have stated that the code will be made publicly available, ensuring full reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
15

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The authors state in the paper that the code will made publicly available. This, combined with the detailed descriptions of the datasets, experimental setup, and evaluation metrics, ensures that the main experimental results can be faithfully reproduced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.com cc/pu
Saic/guides/Codeff
ssioiiiccyyi for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper provides sufficient training and test details in Section 6 and the Appendix. It describes the datasets used, data splits, hyperparameter settings, the method used for hyperparameter selection, and the optimizers used for training.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: While the paper does not report error bars, the statistical significance of experiments is ensured through evaluation across multiple diverse datasets and settings, using standard benchmark datasets. Furthermore, retrainining the model with multiple random seeds for each experiment is computationally expensive
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section provide comprehensive details the computational resources used and the duration of the training process.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httph::t///// /tttur://neurips. wwteerrss.ccppulii/eeeee hetppllic/hhhccsciddeeeeeeeee
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethies. The paper focuses improving the robustness of visual
coders against adversarial perturbations, which relevant concern for responsible AI development.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
17

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses the potential positive and negative societal impacts of the work performed in the conclusion section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not involve the release of data or models that have a high risk of misuse, such as pretrained language models, image generators, or scraped datasets. The research focuses on improving the robustness of visual encoders against adversarial perturbations.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
18

---

Justification:
The paper properly credits the creators or original owners of assets used. The datasets and models used are standard benchmarks in the field, and appropriate citations are provided.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of the package should be provided. For popular datasets, paperswi serrswithhcdee thcode. com dataasts has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper introduces novel unsupervised adversarial fine-tuning frame- documentation. The contribution is the algorithm itself.
work (LORE), but it does not introduce new data, code, or models that require separate
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
19

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve research with human subjects. The research focuses on improving the robustness of visual encoders against adversarial perturbations.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration not required.
Answer: [NA]
Justification: The paper does not describe the use of LLMs part of its core methodology. and LLMs were not involved the design, implementation, or evaluation of this framework.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: httpp::////uriss.... httppp////uripppscc///feeerrreee /025 /LLM) for what should or should be described.
20

---

Partl I
Appendix
Table of Contents
A Proof of Theorem 6.2
22
B The Parametrization Gap
23
Additional Experimental Details
D Constraint Satisfaction in L.OEE
26
E
Generalization Gap in Adversarial Training
26
F
Deviation Between Cosine Similarities
27
Impact Dual Network on Model Performance
28
Comparison Altemative Archilectres for the Dual Function
28
G.2 Effect of the Dual Network (3) on Clean Accuracy
29
H
Revisiting Embedding Models
29
Impact of on Model Performance
31
Additional Experimental Results
31
J.1
Square Attack Evaluation
................................
32
J.2
Evaluation Under Gaussian Noise Corruption
32
J.3
In-domain Image Classification
................................
33
J.4
Zero-shot Image Classification
33
J.5
Out-of- Distribution Robustness
33
J.6
Ablation on Sample-Specine Margin v((2))
33
J.7
LORE Supervised Adversarial Fine- Tuning
34
J.8
Validation on Downstream Tasks (Segmentation)
K
Computation and Efficiency Analysis
37
K.
Convergence Efficiency.
37
K 2 Impact of X. architecture on Training Time
37
K.3 Impact of K on Training Time.
38
21

---

A Proof of Theorem 6.2
We define the following quantities:
R min adv (o: Φo), Rp
min (Ø: do). dEHr
PEH
And
0 argmin ENND ( (a, 2; 00,)) argmin fadr (Ø; poo), dEN,,
oEH,
(9)
where fadv x; 20,) - max d(ó(a 5), 60 (z)).
Since Hp C Hr we have RS 2 R. The sub-optimality gap is bounded by:
0 R - R vk. Lipschitz(o, - 6²)=+ 10 - o' S Vk(L L')E 1 0¹
Proof. Let oj arg max d( (x + 8), (x)))) denote the perturbation that maximizes the adversarial loss for model Φ1 relative to the reference model do. For any two models Φ1, Φ2, we aim to bound the difference in their adversarial losses, considering assumption 6.1:
Paav 2; 00) - (12, x; do)1
(10)
max di Φ1 (z 8), oo(x)) max d( + + (o(x))
(11)
MEA
SEA
d( (I (o())) d(ó2(z æ + 87), 0o(x))
(12)
1 (5) 62 61)
(13)
Alternatively, we set the Lipschitz constant for brevity; however, it is worth noting that in general, this constant appears explicitly in the bound and should be carried through the analysis. Note that without loss of generality, we have assumed that maxsea d( (x + 6), dn(x)) maxsea d(02(x 5), )u(x)) 2 O. that is not the case, then we have to replace of with 82 arg maxáça d( 5), (z)) throughout the proof. We now decompose this difference using the triangle inequality:
[ + 81) 22(x + 81)
(14)
[1(bn (xr + o1) - (x))) - (d2(x 81) - (2(x)) + (b)( (2())) < [1(0) + d1) <2( + 67)) (x) +2(x))] + |1 (x) (z())
(15)
(16)
Assuming that the function Φ1 Φ is Lipschitz continuous with constant L, we obtain:
[1(مb (xr + 5)) Φ2 + 61) ( (z) d2(x)) < L|
(17)
Assuming oj RR* with | < s which corresponds assuming that is Lipschitz with respect to the Euclidean norm, and that e is bounded in the Eo norm-we have:
Plav (01, 2; Φo) las (d2, x; do)l Lvke (1)) 62(x)))
(18)
We now extend this pointwise bound to the expected adversarial loss:
(dv: Φo) - (02; Φu) Eaglaav (01, Φo) las (02, 2; 20)1
(19)
< E. [|eavv (6), x; Φo) 102 do)l]
(20)
s E. Lvke + (() - +2(x))]
(21)
LVVe + || Ø211.
(22)
where || Φ211 denotes the expected difference in their outputs over the input distribution.
Similarly, setting E 0, the adversarial loss reduces to the clean loss, yielding corresponding bound:
eeecan (tn (th, 6o) Lekcan (02, Φb)1 S || - Ø211-
Finally, applying this to the case di o and dh2 = o², we obtain the desired bound on the deviation in adversarial loss due to constraining the hypothesis space.
D
22

---

B The Parametrization Gap
In this paper, solved our constrained optimization problem, by optiming its dual problem, and using neural networks for the lagrangian multipliers. particularly solving:
d' = max min E max d(oo(z + 6). + ()) (u()) - pm(x)))
NEW AEB
.E
Instead of the original constrained optimization problem. in this section interested in deriving some bounds on the duality gap, following the proof from Robey et al. [201]]
Proof.
Assumption B.1. For all g conv(H,, there exists i ④ such that
1 9'1l S
where 1) > a sufficiently small constant.
First, ignoring the parametrization of X, and assuming À can be any function from A = {> :. R+, ), we consider the Lagrangian:
L( A) =
max d d((( - 00, + X())(d((())) - pm((z)) BEA
d' sup inf L(do, A) XEd AEB
Now consider the original problem:
p" inf ENND
max ((x + 6), oo(z)) DEA
s.t. (0, S pm(()), for almost every æ E X
function class parametrized by were convex, this would convex program. Since by definition, do E H {sha 0 E }), there exists 0 E O such that d(cbo <o(x)) = pm((1) for all X. Thus, Slater's condition satisfied.
Therefore, if H. were convex, we would have strong duality, i.e., p' = d' However, for most typical neural networks, H. is non-convex. By weak duality, we always have: d' ppp To derive lower bound, consider the following problem for some positive constant >> 0:
inf gEconv( (H)
EEDD
max d(g(x + 8), (0())) GEA
s.t.
(g S pm(x) - 7) for almost every æ E X'
This is now a convex program. Since du itself satisfies d( (z)) (z))))
pm(x) 7,, Slater's condition is again satisfied. Hence, strong duality holds, and the Lagrangian becomes:
i(g, A) = max d(g(æ + (),))(z)) + )(x)(d(((x), do(x)) - pm((x) + mn) = L(g, +)+11((((x) dEA
Thus,
n' sup inf i(g, A)
xEA gEcun((H))
Assuming the infimum and supremum are attained at 9" and A we have:
d' sup inf L(ds. A) inf L( X²) inf L(0, 2 inf L(g. Xt)
X lEe
gEconv(H)
Using the relation between L and L, we obtain:
d' 2 inf Líg, X+) = inf [(g. X²) - n i(g","1' nB[A(x)) = [n))
gEconv(HH)
23

---

Since 9" is strictly feasible for the relaxed constraint, the complementary slackness condition implies
X (z) = O for almost every 2. Therefore:
2 p
max (x (1) Lavv (g", O0)
SEA
Using the suboptimality bound from Lemma A we have: Madv ,9"
(D)) fadv (Øg. Φa)1 I Lvke + Ilg" dgll. [ 20) felean (6g. Øo)1 S Ilg* اilه
Since 9" is feasible in the relaxed problem with the stricter constraint d(gg (1)) )o(x))) S mm(x) " for some 0 ④ approximating 9 such that l|g" Φg. ll n, the function Øg. is feasible in the
original problem, because for almost every z E X'
Lotcan (óg- (z), (z)) S fetean (g" (g"((((()))))))) (x), (0())) |1g Since p" is the minimum over all feasible E θ it follows that p" S ladv (
oz pm(x) 17 + 11g 2z- < pm(x))
d' ž (g Φ0) Z fudv (0g- Ø0) Lv ke 1lg" - Φz- 2 Par (0g- (00g-:00) 00) Therefore, since 8 is the optimal solution to the original problem:
LLVkE
d' pp Lvke
Finally, we note that although we have treated the dual space as the full adfnite-dimensiona set A {X X' + [0, oo) in this work have restricted À to a inite-dimensionnl family {> Jwen A that uniformly approximates its elements. Concretely, for every À E A there exists w E with ||l (=D &, then replacing
sup inf L(d, A) x PEH
by
sup inf L(,,Aw) WEll PEH
only incurs arbitrarily small error O All weak-duality arguments carry immediately, and under Slater's 's condition the resulting strong-duality statement remains valid to this negligible approximation.
One viewpoint is that limiting the expressivity of A through parametrization effectively relaxes the constraints, the network cannot fully ensure the constraints are met. As an extreme case, consider when (z) = w for x E X. In this case:
xa)x) = w, 2 O.
Then the (relaxed) Lagrangian becomes
L(o, w) = max d(ó(1 - 5), (0()) + (d(o(((()) - pm(x)].
LaEA
Optimizing first over o (so that E/max; d] is fixed) and then taking the supremum 2 0 forces
[[(()(x) ([(((x)))((()))) [n()))] E[m(z)) < 0,
otherwise ((),w) too +oo. In other words, the constant-A relaxation exactly enforces
E[d(á(z), (n(x))
p [[m(())]
Thus, by limiting the expressivity of X, we move from the original per-sample constraint
d(o(x), oo(x)) S pm(x) Væ,
the weaker but still meaningful average constraint
((d() - p pB[m(x)).
24

---

C Additional Experimental Details
In this appendix, we provide further experimental details beyond those given in the main text. Experiments were conducted using 8 NVIDIA HGX H100 80GB GPUs.
Training hyperparameters. We report below the training settings used across all experiments. Unless otherwise noted, all models were trained using A with a weight decay of x 10 cosine learning rate scheduler, and adversarial training with PGD (10 iterations, step size s/4) under an < constraint. Each À network used the 2-layer linear __par__lpppppe architecture, with a hidden dimension of 512, and was optimized via K 5 inner primal updates with learning rate 10-4 More experimental details are provided in Table Additional information about all figures and tables in the paper summarized in Table
Table 6: Training hyperparameters for all models trained with LORE. All models trained with FARE use the same number of epochs and learning rate as the corresponding LORE setting.
Training
Batch size
Model
Dataset
Epochs
(per device)
LR
P
K-iter
ALR
LORE
CLIP VYT-B/32
ImageNet ImageNet-100 CIFARIO
448
lc-5
0.1
Sc-4
CLIP VIT-B/32
448
lc-5
0.1
Sc-4
CLIP VIT-B/32
448
le-5
0.01
5e-4
LORE
CLIP VIT-B/16
ImageNet- 100
128
1e-5
0.1
5
5e-4
CLIP VIT-B/32-LIIO
ImageNet-100
448
le-5
0.15
5
Se-4
CLIP ConvNeXt-B
ImageNet-100 100
64
le-5
0.15
5
Se-4
CLIP VIT-B/32
ImageNet
448
lc-5
0.1
Se-4
CLIP VIT-B/32
ImageNet- 100
448
le-5
0.1
Se-4
CLIP VIT-B/32
CIFAR10
448
1e-5
0.01
Se-4
LORE#
CLIP VIT-B/16
ImageNet-100
128
le-5
0,2
Se-4
CLIP ViT-B/32-LAION
ImageNet-100
448
le-5
0.15
5c-4
CLIP Con
ImageNet-
64
le-5
0.15
5
5e-4
DINOv2 S/14
ImageNet
2
128
le-5
0.05
5
5e-4
DINOv2 VIT-B/14
ImageNet
64
1e-5
0.1
5e-4
CLIP VTT-B/32
ImageNet
448
1e-5
0.1
Se-4
CLIP VIT-B/32
ImageNet- 100
448
le-5
0.1
5e-4
CLIP VIT-B/32
CIFARIO
448
le-5
0.01
5
5e-4
LORE
CLIP VIT-B/32
ImageNet ImageNet-100
448
1e-5
0.2
5e-4
CLIP VIT-B/32
448
1e-5
0.1
5e-4
LORE
CLIP VIT-B/16
ImageNet-100
128
le-5
0.2
Sc-4
CLIP VIT-B/32-LAION
ImageNet-100
448
le-5
0.15
Sc-4
CLIP Con
ImageNet-100
le-5
0.15
5e-4
DINOv2 ViT-S/14
ImageNet
le-5
0.05
5e-4
DINOv2 VIT-B/14
ImageNet
5e-4
CLIP VIT-B/32
ImageNet
448
0.2
Se-4
CLIP VIT-B/32
ImageNet-1Ue 100
448
le-5
5e-4
LORE
CLIP VIT-B/32
ImageNet ImageNet-100
448
le-5
0.2
5
5e-4
CLIP VIT-B/32
448
1e-5
0.1
5
5e-4
LORE
DINOv2 VIT-S/14 DINOv2 VIT-B/14
ImageNet
128
1e-5
0.05
Se-4
ImageNet
64
le-5
0.1
5e-4
25

---

Table 7: Details of each Figure and Table used in the paper.
Figure/Table
Model
Training Dataset
Additional Notes
Figure I-a
FARE2.4.6..00 LORE
ImageNet- 100
Initial performance drop
Figure 1-b
FARE?. LORE
ImagcNet-100
Pareto front
Figure 3
LORE"
ImageNet- 100
Controllability LORE
Figure 4-a
FARE*, LORE
ImageNet
OOD robustness
Figure 4-b
FARE12446.8 LORE
ImageNet
Effect on image embedding interpretability
Figure 5
FARE124.6...10 . ..0RE
ImageNet
Accuracy & robust accuracy across
Figure 6
LORE
ImageNet
Fidelity analysis of LORE
Table I
FARE2 T.RRE24
ImageNet- 100
In-domain image classification
Table 2
FARE LORB
ImageNet- 100
In-domain image classification (DINOv2)
Table 3
FARE 12,4 .LOEE
ImageNet
Zero-shot image classification
D Constraint Satisfaction in LORE
To further understand the behavior of constraint enforcement under varying adversarial budgets, visualize the distribution of distances between clean embeddings and their corresponding pre- trained references throughout training for = 1,2, and in Figure 7 As we can observe, larger perturbation strengths lead to greater deviation from the pre- pre-trained reference in the early stages of training. This early-stage divergence results in a more pronounced initial drop in the model's nominal performance. However, LORE is able to effectively regulate this deviation through its constraint- mechanism, gradually aligning the clean embeddings back within the threshold. This demonstrates the robustness and practicality of LORE in preserving clean performance even under severe adversarial training regimes.
In contrast, due to the lack of such constraint regulation in FARE, the distance between clean embeddings and pre-trained references cannot be reliably controlled. As result, FARE experiences catastrophic initial drop in nominal accuracy, particularly under larger perturbation budgets. This failure to maintain embedding fidelity further underscores the importance of the dual network in LORE for stabilizing the training process and preserving clean accuracy.
To better illustrate this behavior, all subfigures in Figure 7 show the distance distributions beginning from the 20th training iteration onward. These comparisons clearly highlight the contrast between LORE's effective enforcement of the proximity constraint and FARE's limited capability to manage deviation across increasing adversarial strengths.
E
Generalization Gap in Adversarial Training
Theorem E.1 (Generalization Gap in Adversarial Training). It well known that the generalization gap for given loss function is upper bounded by complexity measures, giving rise to theoretical justifications of the bias-variance trade-off. Assuming bounded norm embeddings, i.e., [| S K for all x, Φ, we see that the uniform loss bound B satisfies:
B = sup x)) sup max 6.2
|| + ( ~o(xx)l2 I sup o max[]lo(r + +6]l2 + [[+o())122) 2K.
0.2
Therefore, with probubility at least -25,
E. 12 x)]
1
Σ (()
S (CH) B
log(1/8) 2
< (CH + K
2 2lg((10)
D
where Lu ((( o H3 is the loss class induced by hypothesis class H. Ri (Lx) is the empirical Rademacher complexity of LH, and sup [((Ø, x²| < B.
In adversarial training, the loss class Lu becomes extremely complex due the inner max, operation, leading to large Rademacher complexity R. (Cx). This explains why adversarial training requires significantly more samples for generalization compared to standard training.
When we restrict to simpler hypothesis classes Hp C Hory (through techniques like Lipschitz con- straints norm bounds), the Rademacher complexity decreases, potentially improving generalization.
26

---

COrp
Treking
Dateue
Dateure
Dateue
(a) LORE,
(b) LORE, E
(c) LORE, 1
Stay
(d) FARE,
(c) FARE,
(f) FARE,
Figure 7: Constraint satisfaction comparison between LORE and FARE across adversarial training budgets E = 2,2,4. 4. (Top): LORE maintains strong proximity between clean embeddings and pre- trained references throughout training, with distances concentrating below the threshold. (Bottom): FARE exhibits weaker fidelity preservation and fails to effectively regulate distance under increasing adversarial strength.
However, the bounds we derived are notoriously crude; they fail to capture important phenomena like double descent and often dramatically overestimate the actual generalization gap in practice.
Deviation Between Cosine Similarities
Assumption F.1. For each input let
u
(org(1),
à = do(x),
and suppose
11 < p 1|ull1 for some p [0, 1). Proposition F.2. Under the above assumption, for any nonzero v R"
Se(u, v) - Sc(ú v)l s 2vp.
We show that enforcing implies uniform bound on the change in cosine similarity to any fixed vector IR..
[()o(x) Dorg (z)11 < p (1)))
Proof. Write
" a |
Sc(u,v)
Sc(0,u)
|
27

---

50
(c( So(ū, v) -
||ll|
Now decompose
i
)
(
+
|| || so by the triangle inequality,
|
|
||
|/u al
||
Tall
[
|u||
[|
M
Since ||||||
||u|||
s
I|ue
al|
and ||
S
+
I|u
a|l one shows easily
all
|
||u||
Hence
||u úl
Sc(u, (a, wi]
S
2VP.
|u1||
as claimed.
Remark. In vision-language models one may take v - (b)), the text embedding of prompt 4, so the same bound guarantees
(t)) Sc(os ((t)) s 2v).)
G Impact of Dual Network on Model Performance
G.1 Comparison of Alternative Architectures for the Dual Function
In Figure g we compare the clean and robust accuracy achieved by two different architectures used for the dual function: simple scalar value and network-based (sample-dependent function, as adopted the current LORE setting. While both configurations perform comparably in terms of clean accuracy, the network-based dual function generalizes substantially better adversarial examples, leading to consistently higher robust accuracy throughout training. This highlights the importance of flexible, sample-adaptive mechanism in effectively enforcing robustness constraints during adversarial fine-tuning.
Gean
Robust Accutac
Figure 8: Comparison of clean and robust accuracy when using different architectures for the dual function in LORE. (a): Clean accuracy over training steps. (b): Robust accuracy over training steps. The sewwork-based dual function (sample- based) used in the current LORE setting leads to significantly higher robust accuracy compared Scalar baseline, while maintaining competitive clean accuracy.
28

---

G.2 Effect of the Dual Network (z) on Clean Accuracy
Figure illlustrate the impact of the dual network A (2) on model performance. As shown, the initial steps, higher values of (x) help maintain the model's nominal performance, ensuring it performs well on clean data. In contrast, for FARE, due to the absence of such a proximity constraint during the early iterations, the model, lacking robustness, passes through suboptimal states, leading significant drop in nominal performance.
To further support this observation, we present comprehensive results in Figure and Figure showcasing the behavior of the dual network and its impact across different architectures. In Figure 10 experiments on DINOv2 models (base and small) demonstrate that LORE consistently achieves higher clean accuracy compared FARE, especially in the early stages of training, while adaptively modulating (a:) to control constraint satisfaction. Similarly, Figure 11 reports the performance of ViT-B/16 and ConvNeXt-B models. confirming the effectiveness and generality of the dual network across various architectures and perturbation strengths. These results highlight that LORE's constraint-awart mechanism is stable, avoiding the sharp degradation commonly observed in FARE adversarial fine-tuning.
Accuraty
10
botation
Rectation
500
154
24
500
(a)
(b)
Average the Dual Network Outpurt
15
65
Figure 9: Comparison of the performance of two methods and the output of the dual network. (a) Clean accuracies over iterations, (b) Robust accuracies over iterations, (c) Average output of the dual network (x)).
H Revisiting Embedding Models
CLIP [Radford et al.) 2021 A major part of experiments builds upon CLIP, which consists of an image encoder and a text encoder & that map images and text descriptions into a shared embedding space. For zero-shot classification, textual descriptions are typically formatted as "This is photo of a [CLS]", where [CLS] represents class labels. The probability of assigning an image & to class ý is computed via softmax over cosine similarities:
p(O x)) exp(eos(é((4))
(23)
Σ exp(coo( (tt)), (())))
where is temperature parameter, and K denotes the number of classes.
DINOv2 Oquab et al. 2024 In addition to CLIP, we incorporate DINOv2, a powerful self- supervised visual transformer-batt based encoder, into our exploration of embedding models. While
29

---

Accurecy
Aserage
20
0
(a) DINOv2-base. Clean accuareay (e = 16)
(b) DINOv2-base base, Dual Network Output (e = 16)
Accurecy
Aserage
Dovition
Ml
(c) DINOv2-base, Clean accuarcay (5 = 8)
(d) DINOv2-base Dual Network Output (E 8)
Accurncy
Aserage
0
100
350
(c) DINOv2-small, Clean accuarcay (5 16)
(f) DINOv2-small, Dual Network Output (E = 16)
Accurscy
Average
60
30
Ierutton
(g) DINOv2-small, Clean accuarcay (s 8)
(h) DINOv2-small, Dual Network Output (E = 8)
Figure 10: Comparison of LORE and FARE on DINOv2-base and DINOv2-small models under different adversarial budgets E {8, 16). Left: Clean accuracy over training iterations, illustrating LORE's superior stability and performance. Right: Average output of the dual network A (x) across iterations, highlighting how LORE dynamically adjusts its constraint enforcement.
CLIP provides joint image-text embedding space, DINOv2 focuses solely visual representation learning. This complementary perspective allows us to compare and leverage both multimodal and unimodal embedding paradigms.
30

---

Accurecy
Average
230
600
153
203
1.24
3.4k
(a) VIT-B/16, Clean accuarcay (e = 8)
(b) VIT-B/16, Dual Network Output (e = 8)
Accurecy
Aserage
00
20
Dorticn
Ieration
(c) ConyNeXI-B, Clean accuarcay (6 = 4)
(d) Con-Ne Dual Network Output (5 = 4)
Figure 11: Comparison of LORE and FARE on ViT-B/16 and ConvNeXt-B models under adversarial fine-tuning Left: Clean accuracy over training iterations (s = for VIT-B/16 and = 4 for ConvNeXt-B), showing LORE's improved stability and performance. Right: Average output of the dual network (z), indicating LORE's s dynamic constraint adjustment during training.
DINOv2 learns visual features by minimizing cross-view prediction loss between student and teacher networks. Given N image views, the loss is computed as:
DINO
qia log Pies
(24)
i=1
where tie and Pie. are the teacher and student probabilities for class and view i, respectively, and C the number of output dimensions.
Our broader work centers around embedding models, with primary emphasis on CLIP, while also investigating the capabilities and representations of models like DINOv2. In general, found that the DINOv2 model has much richer image embeddings than CLIP and can achieve much higher robustness over the same perturbation and dataset.
I Impact of K on Model Performance
LORE alternates between K steps of updating the primal encoder and one step of updating the dual network. In this section, study the effect of the hyperparameter K on final performance. As shown in Table & increasing K leads to improved clean accuracy (Acc) and robust accuracy (RAce), particularly when moving from K 1 K primal updates between dual updates help stabilize training and improve performance. Based on this observation, we choose K 5 as the default in our final LORE implementation. For further discussion on how K impacts computational cost and training time, see Appendix K
5. This demonstrates that more frequent
J Additional Experimental Results
In this section, we present additional experiments to further validate robustness and generalization capabilities of our proposed method. These evaluations span multiple settings, including black- box adversarial attacks (e.g., Square Attack), Gaussian noise corruption, in-domain and zero-shot (OOD) robustness. By comparing against the FARE baseline
classification, and
31

---

Table Effect of the hyperparameter on model performance. Clean accuracy (Ace) and robust accuracy (RAcc) (%) are reported for various values of K. Results are based on ViT-B/32 trained with LORE2 evaluated on ImageNet- 100 under 2/255 APGD attack.
K
2
3
5
7
10
Acc (%)
64.11 11
66.43
64.58
60.19
59.96
56.46
RAcc (%)
9.54
13.78
19.48
27.01
31.81
39.71
Table 9: Evaluation on Square Attack, a Black-Box attacks, averaged over the previous mentioned 13 zero-shot datasets
Method
Backbone
Clean
= I
E = 2
E =
4
6
FARE
ViT-B/32
42.6
40.0
36.4
30.0
23.6
LORE
ViT-B/32
50.1
43.9
40.3
33,5
27.1
across diverse conditions and datasets, we demonstrate that LORE consistently achieves superior performance, particularly under challenging threat models and distributional shifts.
J.1 Square Attack Evaluation
In this section, we evaluate the robustness of our fine-tuning approach against a black-box adversarial attack known as the Square Attack (Andriushchenko et al.] [020].. Unlike gradient-based methods, Square Attack operates without access to model gradients and perturbs the input using a query- efficient, score-based strategy. This makes it strong candidate for evaluating real-world robustness where white-box access is not feasible. We conduct experiments on LORE* which is adversarially fine-tuned ImageNet, assess how well the model generalizes to such black- -ox settings. The results, summarized in Table 9 show that our method consistently outperforms the baseline under this challenging threat model.
J.2 Evaluation Under Gaussian Noise Corruption
To further assess the robustness of our method, we evaluate the performance of LORE* and FARE under varying levels of Gaussian noise corruption. We use the ViT-B/32 CLIP model, with both methods fine-tuned on ImageNet. accuracy than FARE across wide range of noise strengths (σ)), especially in low to moderate noise settings. The right subplot illustrates the accuracy gap, highlighting LORE*ss robustness advantage up to 40, beyond which the performance of both models converges the corruption becomes extreme. This evaluation further supports the generalization capabilities our method in the presence of unseen corruptions. Figure 12 provides visual illustration of how single image degrades under increasing levels of Gaussian noise.
shown in Figure 13 LORE4 consistently maintains higher
Noise:
Neise: 32
Noise: 64
Noise: 128
Figure 12: Visualization of single image under increasing levels of Gaussian noise (a = 0, 32, 64, 128). This figure helps set reasonable expectations for model performance by illustrating how perceptual degradation progresses with noise intensity.
32

---

FARE
Azvva
15 2010
Figure 13: Robustness evaluation under Gaussian noise corruption. Left: Classification accuracy of LORE4 and FARE" under increasing Gaussian noise strength (or). LORE consistently outperforms FARE4, particularly in moderate noise regimes. Right: Accuracy gap between LORE and FARE showing a stable and significant margin up to a 40, after which the gap decreases as both models degrade under extreme noise conditions.
Table 10: Clean and adversarial accuracy for in nn-domiin image classification on ImageNet- 100 across different CLIP vision encoders, evaluated using the APGD attack.
Method
Backbone
Clean
==
=
2
= 8
FARES
VIT-B/16
26.5
20.4
17.0
10.3
2.3
LORES
VIT-B/16
70.5
53.6
48.5
37.8
17.8
FARE
ViT-B/32 LAION
17.0
11.3
7.3
3.16
LORES
VIT-B/32 LAION
28.2
12.1
10.0
6.54
3.51
FARE
ConvNeXt-B ConvNeXt-B
61.6
55.3
48.5
35.7
43.4
LORE
72.2
56.2
49.1
38.3
47.2
J.3 In-domain Image Classification
Table 10 presents comparison of clean and adversarial accuracy across various CLiP-based vision backbones, all trained with 8, on the ImageNet- 100 dataset under the APGD attack.
100
J.4 Zero-shot Image Classification
Table presents the results of different settings for zero-shot image classification using the ViT-B/32 CLIP model, highlighting the superiority of our method over the FARE baseline. Additionally, for more challenging scenario, Table [2 reports model performance under high-intensity adversarial attacks, further demonstrating the resilience of our approach. These tables serve as the complete version of the results summarized in the main paper.
J.5
Out-of-Distribution Robustness
As shown in Figure [4 increasing the training perturbation strength leads to greater degradation in out-of-distribtiion (OOD) robustness across common corruptions in ImageNet-C. Despite this trend, models trained with LORE consistently exhibit better robustness compared to those trained with the FARE method, highlighting LORE's superior generalization under distributional shifts.
J.6 Ablation Sample-Specifee Margin m(x)
The choice of sample-specific tolerance margin, m(x) = pllde, (x) 1 is crucial design decision in the LORE framework.
Using a sample-invariant tolerance (Pixet), where the constraint fixed across all data points (d())(()) dh, (a:) < PPised,, offers an alternative but is challenging. It requires extensive, task- dependent and model-dependent hyperparameter tuning that severely limits its practical utility and generalization ability.
33

---

FARCY
LOSE
CLP
No 16
Awwiage
Jve t1y
Average
br phtress
50.3%
spc0l: 1690
Erghtress
speold re0t
retros tky
gass Mur
(b)
Figure 14: Robustness to common corruptions ImageNet-C an OOD evaluation for models trained with (a) = 4, and (b) increases, the degradation in OOD robustness also increases. Nevertheless, LORE consistently shows lower degradation compared to models trained with the FARE method.
we can observe, as the training perturbation strength
By contrast, the sample-specific margin provides an adaptive, scale-invariant tolerance tied to the pre- trained embedding's 's norm. This design choice reduces manual tuning, preserves semantic structure by relating the constraint geometrically to the cosine similarity of embeddings, and ultimately offers greater stability.
Experimental Comparison. To validate this choice, we conducted experiments using ViT-B/32 CLIP models fine-tuned for 5 epochs on ImageNet- 100. We compare the adaptive margin (LORE against several empirically chosen fixed tolerances (p 0.6, 0.8, 1.5). The superscript (e) denotes the training perturbation budget.
As shown in Table [3] fixed tolerances 0.6 and 1.5 lead to suboptimal performance, demonstrating that fixed margin either overly restrictive (degrading robustness) or overly loose (causing behavior similar to FARE). The sample-specific margin (LOREm(()) consistently achieves the best overall Average Robust Accuracy, validating its superior stability and efficacy across different training settings.
J.7 LORE in Supervised Adversarial Fine- Tunni-
Given an image-label pair (a, 3). TeCoA Mao et al. 2023 optimizes a contrastive loss to align adversarial image embeddings with text embeddings. The loss function is defined as:
exp(cos(2, (t+))/T)
s 0) max dEA
Σ log En exp(cos(2, (tx)))/T)
(25)
where z = Φ + 8) is the adversarial image embedding and (4,) is the text embedding. Here, (1:) corresponds to the text embedding of the i-th class, where each class label is inserted into fixed natural language template (e.g.. "This is photo of [class]").
We incorporated our proximity- based regularizer into TeCoA, forming combined method (TeCoA + 12), where l2 denotes the basedd embedding proximity constraint:
max min Fr.y~D [Tmmp(2 (x. 0) + (xx (10o(x) - (o)x)) ploo(x))
wEf BE
Distribution-Levad Constraint. We also explored a probability supervised setting, constraining the KI. divergence between the fine-tuned model's output distributions and those of the pretrained model. This approach requires only class text embeddings, not sample labels. Applied to TeCoA. it yiclds second regularized variant: TeCoA + KL.
variant of LORE for
34

---

Table 11: A comprehensive evaluation of clean and adversarial performance is conducted across various image classification datasets using the ViT-B/32 CLIP model. All models are trained on ImageNet and evaluated in a zero-shot setting across diverse benchmarks. Table demonstrates the increase (f) in performance of our method relative to the corresponding FARE models.
Zero-shot datasets
Vision encoder
ImageNet
CalTech
Cars
CIFARIO
CIFARIOO
DTD
EurOSAT
FGVC
Flowers
ImageNNtt-
ImageNee-SS
PCAM
OxfordPets
STL-10
Average
Eval.
Zero shoo-shot
CLIP
59.8
84.1
59.6
89.7
63.3
44.4
46.1
19.6
66.3
69.3
42.3
62.3
87.5
97.2
64.0
FARE
56.6
84.0
56.3
86.4
61.1
40.5
27.2
18.1
62.0
66.4
40.5
55.5
86.1
95.8
60.0
LORE!
57.4
84.4
55.9
88.5
64.5
40.1
29.9
16.7
61.3
67.2
41.5
53.8
86.9
96.3
60.5
10.5
FARE
52.9
82.2
49.7
76.3
51.1
36.4
18.4
15.7
53.3
60.4
35.9
48.2
82.7
93.0
54.1
LORE
55.7
83.0
51.0
83.4
59.7
37.2
23.0
15.9
54.5
63.4
39.3
51.2
84.3
94.5
57.0
12.9
FARE
78.1
36.5
35.8
28.8
10.6
36.1
49.3
50.0
71.8
44.7
LORE"
50.1
80.3
40.1
72.4
49.6
32.4
17.7
11.4
39.7
55.1
33.6
50.0
79.3
90.4
50.2
15.5
FARE
33.0
73.0
24.7
40.0
24.9
23.7
15.2
6.24
22.7
39.5
20.2
50.0
52.4
75.0
36.0
LORE"
42.6
75.3
28.7
61.5
35.8
26.1
16.4
8.25
25.8
45.2
26.2
50.0
69.3
84.2
42,5
16.5
FARE
27.6
69.1
17.0
34.2
20.2
20.6
15.0
4.68
16.5
34.1
16.8
50.0
37.6
67.3
31.0
LORE
74.6
24.9
35.9
24.5
7.14
22.8
43.0
24.4
50.0
67.4
41.2
110.3
FARE
23.2
66.0
12.8
31.3
17.5
18.8
15.0
13.7
30.2
14.4
50.0
27.6
61.9
28.0
LORE
40.5
74.3
23.8
64.8
38.8
24.3
16.2
6.75
22.0
41.9
22.9
50.0
66.4
84.2
41.2
113.2
CLIP
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.1
0.0
0.0
0.0
0.0
FARE
27.8
68.6
16.1
61.0
35.6
22.5
6.1
2.9
30.6
34.4
22.5
24.7
55.8
82.2
35.6
LORE
32,9
71.0
18.7
67.1
40.0
23.7
9.4
4.2
33.5
37.6
24.8
28.3
60.5
84,1
38.7
13.1
FARE
75.2
22.6
35.4
24.7
33.9
39.7
24.1
30.4
64.8
39,4
LORE
39.3
76.3
23.3
67.0
43,2
26.4
6.5
35.8
42.4
26.4
39.0
68.5
85.6
42.5
13.1
==
FARE
33.2
74.8
21.4
44,9
28.0
22.4
14.0
5.8
27.3
37.1
21.3
50.2
59.3
77.7
37.2
LORE
41.8
77.2
24.1
61.2
39.9
24.5
14.3
7.8
30,2
41.6
25.5
50.2
68.8
83.2
42,2
15.0
FARE
26.3
70.7
15.2
32.8
20.0
19.5
14.1
3.6
19.3
30.0
15.1
50.2
43.5
70.2
31,1
LORE"
36.2
74.4
18.9
52.2
30.6
20.8
15.4
22.3
34.2
22.1
50.2
60.5
78.1
37.4
16.3
FARE
66.9
16.7
17.7
15.5
25.3
12.5
50.2
30,9
27.3
LORE"
35.5
73.6
15.9
51.1
31.2
20.1
14.2
5.8
20.0
33.3
20.7
50.2
58.6
77.4
36.3
19.0
FAREÃO
19.0
65.4
8.20
26.3
14.3
16.7
14.8
3.0
13.1
22.0
11.0
50.2
23.7
57.2
25.1
LORE
33.8
71.2
14.3
51.7
30.1
19.5
12.9
4.2
18.9
31.5
19.4
50.2
53.9
76.7
35.0
19.9
CLIP
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.1
0.0
0.0
0.0
0.0
FARE
8.0
43.5
1.9
31.0
14.7
12.9
0.6
0.2
6.8
13.4
11.7
14.1
15.9
$4.9
17.0
LORE'
49.0
19.0
10.1
19.1
23.1
20.8
13.8
FARE
59.9
7.7
41.2
22.8
17.8
1.5
16.4
24.2
15.9
23.4
38.6
26.7
LORE3
24.0
63.3
8.6
47.2
27.2
18.2
10.6
1.7
18.5
26.0
18.4
28.0
44.4
73.1
29.6
12.9
2=0
FARE
24.1
65.5
10.4
36.0
21.6
18.8
12.3
2.7
17.9
27.7
15.8
50.0
44.4
68.8
30.1
LORE
32.6
69.5
12.4
50.8
29.6
20.9
13.0
3.3
21.6
32.3
20.0
50.1
55.9
76.1
35.0
14.9
FARES
64.6
16.7
50.2
33.3
26.5
LORE
30.1
68.3
10.7
44.0
25.6
18.5
13.8
3.7
16.8
27.6
17.0
50.2
49.7
71.2
32,1
15.6
FARE
17.4
62.7
6.4
24.5
13.7
15.4
13.1
1.5
11.2
19.7
10.7
50.2
24.0
56.2
23.8
LORE"
30,9
68.8
10.5
43.3
25.7
18.5
13.5
3.2
16.6
27.8
17.0
50.2
49.2
71.6
31.9
18.1
FARE
15.1
60.0
5.0
23.5
11.8
14.3
13.5
1.7
10.6
18.5
9.2
50.2
18.4
$2.5
22.2
LORE
29.7
66.8
8.8
38.8
24.1
17.9
12.4
14.9
27.0
16.2
50.2
45.5
69.1
30.3
18.1
CLIP
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.1
0.0
0.0
0.0
0.0
FARE
6.3
2.0
2.3
0.0
LORE"
0.7
9.7
0.0
3.5
3.1
4.0
0.0
0.0
0.2
3.8
2.8
2.7
0.0
9.4
3.0
FARE
3.2
27.5
0.5
12.3
7.0
7.7
4.3
0.0
2.4
6.8
5.1
15.8
3.0
30.1
9.4
LORE
5.7
31.1
0.7
13.0
8.2
9.7
0.8
0.0
3.1
8.3
6.5
18.2
7.2
33.5
10.8
11.4
c=40
FARE
10.7
46.3
1.5
19.7
11.8
11.9
10.2
6.4
11.4
45.2
16.2
46.1
18.2
LORE"
17.8
54.2
2.8
16.8
8.0
48.4
25.5
56.1
22.5
14.3
FARE
11.6
50.5
1.6
19.2
9.8
12.1
6.3
12.7
7.4
50.2
15.8
46.0
18.7
LORE"
19.2
57.0
3.5
26.2
16.4
13.9
12.7
1.0
8.9
16.9
10.5
50.2
26.9
57.0
23,2
14.5
FARE
10.9
50.0
1.5
18.3
9.2
11.4
11.8
0.7
6.3
11.9
6.5
50.2
12.4
44.3
18.0
LORE"
21.7
58.8
4.1
28.0
17.2
13.8
12.8
1.1
9.5
17.7
10.9
50.2
31.5
59.0
24.2
16.2
O
9.03
48.3
1.1
17.7
8.3
10.4
11.5
5.5
11.1
50.2
10.6
41.9
17.1
LORE
56.8
3.5
14,8
13.7
9.3
17.4
10.6
50.2
28.8
52.9
22.5
15.4
For classification using probability distributions induced by the contrastive model, we formulate the constrained optimization problem:
min En.y~D Lamp (I, y1 0)] s.t. (RI(P 1)) po x)) S p=m(x), where m(x) H (poo (: (x)
0
35

---

Table 12: Evaluation under high-intensity adversarial attacks. A comprehensive assessment of clean and adversarial performance conducted across various image classification datasets using the ViT-B/32 CLIP model. All models trained on ImageNet and evaluated in zero-shot setting across diverse benchmarks.
Zero-shot datasets
Vision enooder
ImageNer
CalTech
Cars
CIFARIO
CIFARION
DTD
EuroSAT
FGVC
Flowers
ImageNNet-R
ImageNec-SS
PCAM
OxfordPets
STL-110
Average
Eval.
Zero shot
FARE
5.5
28.5
0.1
11.4
6.2
7.7
10.4
0.0
2.7
5.8
3.5
50.2
3.6
30.4
12.3
0.0
LORE"
10.4
40.1
0.6
13.7
9.6
9.9
11.9
4.5
9.4
50.2
11.4
40.1
16.0
13.7
FARES
7.8
LORE
12.9
44.6
1.4
15.0
10.9
10.4
12.2
0.6
5.3
11.0
50.2
15.3
43.2
17.5
FARE-
5.3
31.4
0.0
13.7
5.4
7.3
11.8
0.0
3.2
6.1
3.6
50.2
4.2
28.5
12.7
LORE"
13.6
45.0
1.5
10.7
9.6
10.1
11.3
0.6
5.4
11.3
7.1
50.2
14.2
35.3
16.3
13.6
FARE
1.9
16.8
0.0
5.9
3.5
5.7
9.2
0.0
0.7
3.2
2.2
50.2
0.7
13.8
8.6
=
LORE"
4.8
24.3
0.2
6.7
5.0
7.4
8.1
0.0
2.0
5.5
4.0
50.2
3.6
22.3
10.7
12.1
FARE
19.1
0.0
3.8
5.4
10.0
0.0
1.4
3.2
50.2
0.9
16.2
9.3
LORE
30.4
6.2
8.0
3.8
50.2
12.3
13.0
FARE
19.5
0.0
4.0
5.2
1.6
3.3
50.2
1.2
9.4
LORE"
31,9
0.4
5.0
5.3
8.0
10.6
0.0
4.0
6.8
50.2
6.8
20.0
11.8
FARE
9.2
0.0
1.9
3.3
50.0
0.2
10.0
LORE"
1.7
13.9
0.0
2.8
2.7
4.8
0.1
0.0
0.7
3.2
2.2
50.1
0.4
9.5
6.9
FARE
0.8
10.5
0.0
3.9
2.2
3.6
8.0
0.0
0.5
1.5
1.1
50.1
0.5
6.5
6.8
II
LORE
3.2
18.9
0.1
3.7
3.3
5.6
2.9
0.0
1.6
4.2
3.0
50.1
2.1
13.8
8.4
11.6
FARE
11.5
0.0
2.3
3.4
0.6
1.8
50.1
0.6
7.6
LORE""
20.2
0.1
3.2
6.0
1.7
4.9
50.2
2.8
8.7
11.6
Table 13: Comparison of Adaptive Margin m(x) vs. Fixed Margins (ViT-B/32 ImageNet- 1000..
The Avg Robust Acc column the average of the three adversarial perturbation levels (E = 1, 2, 4).
Train
Method
Clean
E
2
4
Avg Robust Ace
LORE m(a)
0.1
68.81
51.51
33.71
9.93
31.72
LORE
0.6
70.12
24.32
18.65
0.56
14.51
2
LORE
0.8
69.65
45.65
29.45
5.65
26.92
LORE
1.5
63.45
41.23
25.47
3.64
23.45
FARE
62.23
40.12
23.15
3.40
22.22
LORE m(x)
66.91
46.99
35.71
17.58
33.43
LORE*
0.6
67.21
19.50
7.21
1.23
9.31
LORE
64.32
40.21
32.23
14.68
29.04
LORE
1.5
55.24
33.21
29.32
11.74
24.76
FARE
53.68
34.85
28.78
12.87
25.50
Building on this formulation, LORE's final objective function can be computed as before. The framework adjusts the proximity constraint by model confidence: low entropy (high confidence) tightens the constraint, while high entropy relaxes for greater adaptation flexibility.
The experimental setting involved fine-tuning ViT-B/32 CLIP models for 15 epochs ImageNet-100 using a learning rate of le 5. The proximity constraint hyperparameter used p = 0.1. The results are shown in Table
Results.
TeCoA consistently benefits from both LORE-based (l2) and KL regularization, achieving improved robustness while maintaining clean performance. Notably, for high adversarial training budgets (e.g.. Train è = 8), the TeCoA + la method yields massive increase in clean accuracy compared to baseline TeCoA. These results demonstrate that mixiimitt-bbasdd constraints constraints-whetherr -wether applied in embedding space (12) or probability space (KI.) effectively stabilize adversarial fine-tuning in supervised frameworks well.
36

---

Table 14: Extension to Supervised Fine- Tuning (ViT-B/32 CLIP on ImageNet-100) The Avg Robust Acc column is the average of the four adversarial perturbation levels (E = 1,2, 8).
Train
Method
Clean
4
Avg Robust Acc
TeCoA²
60.04 (ref)
49.11
35.94
14.73
0.50
25.07 (ref)
TeCoA² l2
75.97 (+15.93
53.24
39.12
18.12
1.23
27.93 (42.86 f
TeCoA² + KL
66.50 (+6.46 f)
53.04
39.53
16.94
0.60
27.53 (42.46 1)
TeCoA*
49.55 (ref)
44.20
38.21
19.80
1.03
25.81 (ref)
4
TeCoA* +
73.12 (+23.557
56.20
53.12
49.12
5.23
40.92 (+15.11 ft)
TeCoA* + KL.
56.01 (46.46 f)
48.62
42.03
22.18
1.25
28.52 (42.71 ft
TeCoA*
30.22 (ref)
24.72
19.09
7.41
2.35
13.39 (ref)
TeCoA* + 12
67.23 (437.01 t)
36.21
26.40
15.12
4.23
20.49 (47.10 f)
TeCoA KL
38.06 (47.84 f)
27.69
21.94
8.75
2.82
15.30 (+1.91 f)
J.8 Validation on Downstream Tasks (Segmentation)
To confirm LORE's generalizability beyond classification, we evaluated its effectiveness on semantic segmentation, critical VLM downstream task. We use the et al. [2024] to have fair comparison with our baselines.
benchmark from Kowalczuk
We assessed the robustness of the DINOv2 ViT-S/14 model the ADE20k segmentation dataset. The results in Table 15 show that the LORE-hardened model improves robustness against adversarial attacks in the embedding space while maintaining favorable clean accuracy trade-off.
Table 15: Segmentation Results on ADE20k (DINOv2 VIT-S/14). LORE and FARE denote models trained with an 8/255 perturbation budget. mloU is reported for clean data and under an adversarial attack in the embedding space.
Method
Clean mloU
Embed Attack mloU
Pretrained
0.42
0.23
LORE
0.43
0.35
FARE
0.23
0.15
K
Computation and Efficiency Analysis
In this section, we analyze computational aspects of LORE terms of training time, efficiency, and convergence behavior. While LORE introduces additional dual network and constraint enforcement mechanism, we find that its cost remains practical and comparable to FARE baselines.
K.1 Convergence Efficiency.
To compare the training efficiency of FARE and LORE, we measure the total training time (in minutes) required to reach specific robust accuracy (RAce) thresholds on the validation set. Table reports this comparison across various model backbones, including ViT-B/16, ViT-B/32, Co B. ViT-S/14, and ViT-B/14. LORE often reaches target RAce levels fewer training minutes than FARE, highlighting its superior optimization efficiency and stability.
K.2 Impact of X architecture on Training Time.
The results Table 17 show that LORE's current dual network design is not only significantly more efficient than using separate pretrained CLIP model dual network, but also achieves comparable runtime to simpler parameterizations (scalar linear forms). This demonstrates that LORE achieves computational efficiency without sacrificing expressive capacity.
37

---

Table 16: Training time (in minutes) required reach different robust accuracy (RAce) thresholds under PGD attack (a = 2/255). Lower is better. All models were trained using 4 NVIDIA H100 80GB GPUs.
Model
Method
10%
20
30 %0%
35%
Dataset
FARE
23
38
64
98
ViT-B/16
ImageNet-100
LORE
11
17
23
31
FARE
30
86
ViT-B/32
ImageNet
LORE
25
68
185
FARE
62
115
165
285
ViT-B/32
ImageNet-100
LORE
47
84
148
268
FARE
19
26
34
40
ConvNeXt-B
ImageNet-100
LORE
52
57
60
63
FARE
25
43
100
181
ViT-S/14
ImageNet
LORE
39
71
124
126
FARE
22
28
43
56
ViT-B/14
ImageNet
LORE
29
32
42
53
Table 17:
Training time (in seconds) for 50 iterations of LORE using 4xH100 GPUs across different architectures for Au- The Linear model uses a single-layer projection: nn Linear (self, embedding.size, 1). Lower is better.
Architecture
Training Time (s)
LORE (Current Dual Network) Scalar (input-independent)
551
533
Linear À (input-independent))
540
Pretrained CLIP
692
K.3
Impact of K on Training Time.
As described in Section LORE alternates between K primal updates and one dual update per batch. While Section [ anallzess the impact of K on final performance, here in Fig. [3] empirically examine how varying affects training time.
38

---

VIT-8/22 CanvNeXI-B
600
Training Time (s)
400
200
100
10
Figure 15: Figure: Training time (in seconds) for completing 30 training iterations of LORE under different values of K, using 8xH100 GPUs. Results are reported for two architectures: ViT-B/32 and ConvNeXt-B. Increasing K slightly raises the training time due more frequent primal updates, with consistent trends across both models.
39


Reviews:
[{"review_id": "Js8mZDELE8", "reviewer": "Reviewer_CYtr", "summary": "The paper proposes a novel unsupervised adversarial fine-tuning framework dubbed Lagrangian-Optimised Robust Embeddings (LORE). The authors aim to address the limitations of current unsupervised adversarial fine-tuning approaches: instability during early stages of fine-tuning and suboptimal trade-off between clear and adversarial performance. LORE maintains the performance on clean data during the fine-tuning process by enforcing embedding proximity constraints.  Moreover, LORE enables principled control over the clean-adversarial performance trade-off. The paper demonstrates the effectiveness of this approach on CLIP and DinoV2 models, compared with the SOTA FARE approach.", "questions": "Q1: Can LORE be extended to simultaneously fine-tune both image and text encoders in Visual Language Models like CLIP?\n\nQ2:  \"(2) LORE relies on a high-quality pretrained model, and its performance, especially on perceptual metrics, is sensitive to the fidelity of that model.\" Can you please provide more discussion on that matter? At what point does LORE stop being an effective approach?", "limitations": "The authors adequately addressed the limitations. However, I feel that more discussion on the following would improve the paper: \" (2) LORE relies on a high-quality pretrained model, and its performance, especially on perceptual metrics, is sensitive to the fidelity of that model.\"", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\nS1: LORE effectively addresses a crucial problem of instability during the early stages of adversarial fine-tuning in unsupervised adversarial fine-tuning. I find the significance of this result to be high, as it enables adversarial fine-tuning for a limited number of updates without collapsing the model’s performance. This makes the adversarial fine-tuning process more practical and user-friendly.\n\nS2: LORE effectively addresses an important problem of suboptimal trade-off between clear and adversarial performance, pushing the empirical Pareto front.\n\nS3: Strong experimental section. (However, see Weaknesses for ideas of additional improvements)\n\n### Weaknesses\n\nW1: Optimal trade-off claim.  The paper uses strong language to describe the performance of LORE. For example, “LORE yields a strictly better Pareto front, achieving optimal trade-offs between robustness and clean accuracy” (Figure 1). At the same time, the paper notes that: \"(1) A deeper theoretical analysis of the trade-offs in unsupervised adversarial finetuning is needed\" (L294-295).  While the empirical results indeed prove that the Pareto front is improved over the baseline, calling it “optimal” might be an overstatement. According to my understanding, there is no proof that LORE finds the global optimum of the robustness vs. accuracy trade-off.\n\nW2: Missing broader related work. I believe that the Related Works section misses some earlier attempts at ensuring adversarial robustness in the unsupervised fine-tuning setting. For example:\n\n[1] Kim et al., Adversarial Self-Supervised Contrastive Learning, NeurIPS 2020\n\n[2] Zhang et al., Adversarial Contrastive Learning for Unsupervised Representation Learning, NeurIPS 2021\n\n[3] Chiang et al., Robust Contrastive Learning for Unsupervised Visual Representation Learning, CVPR 2022\n\n\nW3: Limited downstream tasks. Dinov2 can produce an embedding usable in a variety of downstream tasks, e.g segmentation. Yet, the evaluation in the paper is limited to classification. I believe it would be valuable to asses the effectivness of robust models (e.g. DinoV2) obtained with LORE on more diverse downstream tasks, e.g. similarly to [4]. \n\n[4] Kowalczuk et al., Benchmarking Robust Self‑Supervised Learning Across Diverse Downstream Tasks, CVPR 2024 Workshop on Robust Representation Learning\n\n### Comments\nC1: While this is not a weakness per se, LORE targets instability during early stages of adversarial fine-tuning and suboptimal trade-off between clear and adversarial performance in the *unsupervised* adversarial fine-tuning. According to my understanding, this approach is not directly applicable to supervised models. Yet, the supervised setting also suffers from similar problems. This limits the (still high) significance of the method proposed in this paper. The paper's significance would be even greater if a LORE-like approach was applicable to all adversarial fine-tuning scenarios.", "was_revised": true, "final_justification": "**I clearly recommend accepting the paper.** LORE tackles an important challenge in adversarial fine-tuning: maintaining clean accuracy during the early stages of training. The authors submitted a strong paper and followed up with a rebuttal that effectively addressed my concerns. This includes promising results in supervised settings and on downstream tasks beyond classification, which also responds to concerns raised by other reviewers. In my view, the paper has clear practical value. For example, I see potential for LORE in applications such as model merging, where extended fine-tuning can compromise the integrity of merged models. LORE could enable the creation of robust task vectors while preserving clean data performance due to addressing the performance drop in the early stages of training.\n\n**Overall, I clearly recommend accepting the paper and kindly ask other Reviewers to consider increasing their scores.**", "rebuttal": "Thank you for your careful reading of our paper and for highlighting both its strengths and areas for improvement. We respond below to your main concerns and detail how we will strengthen the manuscript.\n\n> **Answer for W1**\n\nWe agree with your observation and have revised the caption of Figure 2 to remove the term “optimal,” which could misleadingly imply a proven global optimum that we do not formally establish. As shown in our experiments (Section 6), our goal is to present an approach that consistently outperforms the FARE baseline in practice, emphasizing empirical improvements without overstating theoretical guarantees.\n\n> **Answer for W2**\n\nWe sincerely appreciate your suggested references. The first paper you mentioned (Kim et al., NeurIPS 2020) is indeed relevant, and we will include it in the extended version of our Related Work section. However, we were unable to locate the last two references as cited. If you could clarify or provide more details, we would be happy to consider them as well.\n\n> **Answer for W3**\n\nThe authors acknowledge the importance of evaluating LORE's effectiveness on diverse downstream tasks beyond image classification, such as segmentation, for models like DINOv2. During the rebuttal, they conducted additional, though time-constrained and less comprehensive, experiments focusing on the ViT-S/14 model fine-tuned for 2 epochs on ImageNet. These new evaluations show that LORE-hardened models maintain strong performance on these tasks, demonstrating the general utility of LORE. The table shows the results of DINOv2 ViT-S/14 on the ADE20k dataset. We used EmbedAttack to evaluate robustness, following the approach in [4].\n\n| Method | Clean mIoU | EmbedAttack mIoU | \n|---|---|---|\n| Pretrained| 0.42 | 0.01 | \n| $\\text{LORE}^8$|  0.23 | 0.15 | \n| $\\text{FARE}^8$|  0.15 | 0.11 | \n\n\n> **Answer for C1**\n\nWe thank the reviewer for this insightful comment and fully agree that extending LORE to supervised adversarial fine-tuning would broaden its impact. While LORE was originally developed in an unsupervised setting, its core principle is potentially compatible with supervised adversarial training.\n\n### **Experimental Extension:**\n\n### **1. LORE in Supervised Adversarial Fine-Tuning**\n\nTo directly address the reviewer’s suggestion, we extended our work by applying LORE regularization to the supervised adversarial fine-tuning method TeCoA [1], which aligns image-text pairs using supervised losses. \n\nGiven an image-label pair $(x, y)$, TeCoA optimizes a contrastive loss to align adversarial image embeddings with text embeddings. The loss function is defined as:\n\n\\begin{equation} \nL_{sup}(x, y; \\theta) = \\max_{\\delta \\in \\Delta} - \\sum_i \\Bigg[y_i \\log \\frac{\\exp(\\cos(z, \\psi(t_i))/\\tau)}{\\sum_k \\exp(\\cos(z, \\psi(t_k))/\\tau)} \\Bigg], \n\\end{equation}\n\nwhere $z = \\phi_\\theta(x + \\delta)$ and $\\psi(t_i)$ are the image and text embeddings, respectively. Here, $\\psi(t_i)$ corresponds to the text embedding of the *i-th* class in the dataset, where each class label is inserted into a fixed natural language template, e.g., *\"This is a photo of {class}\"*. \n\n---\n\nWe incorporated our proximity-based regularizer into TeCoA, forming a combined method *(TeCoA + $l_2$)* as follows:\n\n$$\n\\max_{\\omega\\in\\Omega}\\min_{\\theta\\in\\Theta} E_{x,y \\sim D} \\left[ L_{\\text{sup}}(x,y;\\theta) + \\lambda_\\omega(x) \\big(\\|\\phi_\\theta(x)- \\phi_0(x)\\|_2^2 - \\rho  \\|\\phi_0 (x)\\|_2^2   \\big) \\right]\n$$\n\n### **2. Distribution-Level Constraint**\n\nWe also explored a probability-distribution variant of LORE for the supervised setting, constraining the KL divergence between the fine-tuned model’s output distributions and those of the pretrained model. This approach requires only class text embeddings, not sample labels. Applied to TeCoA, it yields a second regularized variant: TeCoA + KL.\n\nFor classification using probability distributions induced by the image classifier or a contrastive model (outputs logits of softmax of cosine similarities), for both current and original models, We formulate the constrained optimization problem:\n\n$$\n\\min_{\\theta \\in \\Theta} \\;\nE_{x,y \\sim D}\\big[L_{sup}(x,y;\\theta)\\big] \\quad \\text{s.t.} \\quad\nD_{\\text{KL}}(p_{\\theta_0}(\\cdot \\mid x)\\,\\|\\,p_\\theta(\\cdot \\mid x)) \\leq \\rho\\cdot m(x) , \\quad \\text{where} \\quad m(x) = H(p_{\\theta_0}(\\cdot \\mid x))\n$$\n\nAnd building on this formulation, LORE's final objective function can be computed as before. The framework adjusts the proximity constraint by model confidence: low entropy (high confidence) tightens the constraint, while high entropy relaxes it for greater adaptation flexibility. \n\n## Results\n\n**Experimental setting:** We fine-tuned ViT-B/32 CLIP models for 15 epochs on ImageNet-100 using a learning rate of 1e-5 and perturbation strength ε. The proximity constraint hyperparameter used ρ=0.1, consistent with previous LORE experiments.\n\n| Training ε | Method                 | Clean                  | ε=1   | ε=2   | ε=4   | ε=8   | Avg Robust Acc           |\n|------------|-----------------------|-----------------------|-------|-------|-------|-------|--------------------------|\n| 2          | **TeCoA**              | 60.04 (ref)                | 49.11 | 35.94 | 14.73 | 0.50  | 25.07    (ref)               |\n| 2          | **TeCoA + $l_2$**       | 75.97 (**+15.93 ↑**)  | 53.24 | 39.12 | 18.12 | 1.23  | 27.93 (**+2.86 ↑**)     |\n| 2          | **TeCoA + KL**     | 66.50 (**+6.46 ↑**)   | 53.04 | 39.53 | 16.94 | 0.60  | 27.53 (**+2.46 ↑**)     |\n| 4          | **TeCoA**             | 49.55  (ref)               | 44.20 | 38.21 | 19.80 | 1.03  | 25.81        (ref)           |\n| 4          | **TeCoA + $l_2$**       | 73.12 (**+23.57 ↑**)  | 56.20 | 53.12 | 49.12 | 5.23  | 40.92 (**+15.11 ↑**)    |\n| 4          | **TeCoA + KL**     | 56.01 (**+6.46 ↑**)   | 48.62 | 42.03 | 22.18 | 1.25  | 28.52 (**+2.71 ↑**)     |\n| 8          | **TeCoA**              | 30.22         (ref)        | 24.72 | 19.09 | 7.41  | 2.35  | 13.39         (ref)          |\n| 8          | **TeCoA + $l_2$**       | 67.23 (**+37.01 ↑**)  | 36.21 | 26.40 | 15.12 | 4.23  | 20.49 (**+7.10 ↑**)     |\n| 8          | **TeCoA + KL**  | 38.06 (**+7.84 ↑**)   | 27.69 | 21.94 | 8.75  | 2.82  | 15.30 (**+1.91 ↑**)     |\n\nAs shown above, TeCoA consistently benefits from both LORE and KL regularization, achieving improved robustness while maintaining clean performance. These results demonstrate that LORE is not limited to unsupervised fine-tuning and that proximity-based constraints—whether applied in embedding space or probability space—can effectively stabilize adversarial fine-tuning in supervised frameworks as well.\n\n> **Answer for Q1**\n\nOur primary focus in this work was improving robustness on the vision side, but your suggestion opens an important direction: extending LORE to jointly fine-tune both image and text encoders. Prior findings (e.g., TeCoA [1]) indicate that such joint tuning can cause training instability, especially in zero-shot settings where cross-modal alignment may drift and generalization degrades. Similarly, FARE [2] emphasizes the need to preserve the original embedding characteristics of both clean and adversarial examples in VLM pipelines (e.g., CLIP in LLaVA). Simultaneous fine-tuning risks keeping modalities aligned with each other while drifting away from their pretrained distributions, harming robustness and semantic fidelity. Nonetheless, it could offer benefits in scenarios such as image-only attacks, where updating the text encoder may help realign embeddings with perturbed image features, potentially mitigating attack effects.\n\n## On Extending LORE:\n\n**In Unsupervised Settings:**\n\nExtending LORE to jointly fine-tune both encoders in a fully unsupervised setting is non-trivial. Without access to text supervision (e.g., class labels or prompts), it's unclear how to meaningfully update the text encoder. In this case, LORE would not have the necessary guidance to preserve alignment or constrain both encoders effectively.\n\n**In Supervised Settings:**\n\nIn contrast, supervised settings offer a viable path forward. Since textual labels and their embeddings are available, the text encoder can be updated effectively. For instance, in our proposed extensions to TeCoA (discussed in response to Comment C1), applying LORE’s proximity-based regularization can help stabilize joint fine-tuning. The constraints act as stabilizers, keeping both encoders anchored to their pretrained representations while allowing controlled adaptation.\n\nMoreover, if the threat model includes perturbations to both modalities, then fine-tuning the text encoder becomes necessary, and LORE-style regularization could be critical for maintaining dual robustness.\n\n> **Answer for Q2**\n\nBoth LORE and FARE's effectiveness hinges on the quality of their frozen pretrained reference model, which acts as an anchor. LORE will still converge and improve robustness, as it helps control the nominal-adversarial robustness trade-off by constraining model deviation; however, its performance ceiling (both clean accuracy and robustness) is inherently limited by the reference model's quality. There's no sharp cutoff; LORE's benefits gradually lessen with lower-quality pretrained models. Empirical evidence from Tables 1 and 2 consistently supports this: stronger pretrained models (e.g., higher clean accuracy, larger architectures like ViT-B/16 over ViT-B/32, or DINOv2 ViT-B/14 over ViT-S/14) consistently lead to better nominal and robust results when LORE is applied.\n\n--- \n[1] Mao, C., Geng, S., Yang, J., Wang, X. E., and Vondrick, C. Understanding zero-shot adversarial robustness for large-scale models. In ICLR, 2023.\n\n[2] Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. ICML, 2024.\n\n[4] Kowalczuk et al., Benchmarking Robust Self‑Supervised Learning Across Diverse Downstream Tasks, CVPR 2024"}, {"review_id": "HaKM59StBi", "reviewer": "Reviewer_TnAL", "summary": "In this paper, the authors propose a Lagrangian-based optimization method to train a robust visual encoder, i.e., CLIP and DINOv2. Compared to the previous method, their method is more stable during the training process and achieves a better tradeoff between clean and adversarial accuracy, which is very important for adversarial robustness. Their experiments show that their method achieves higher adversarial accuracy on the image classification problem while the clean accuracy does not drop significantly.", "questions": "In your method, you use a sample-specific tolerance margin, which is related to the norm of the original feature embedding. What would the results be if we used a sample-invariant tolerance?", "limitations": "Yes, they discuss their limitations in the last section.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros:\n* In Section 4, they show that simply training the CLIP using an adversarial loss $L_{FARE}$ and a fixed-weighted regularizer could not solve the problem, which provides a good motivation for their method.\n* The paper is well-written with clear logic and results.\n* Their method is effective, achieving better robustness under adversarial attack while having minimal degradation on the clean accuracy.\n* They validate the efficiency of their method on various dataset sizes and adversarial attack budgets, indicating the robustness of their method over general settings.\n\nCons:\n* In the experiment part, the authors use the image classification task as the measurement to verify the robustness. Since the CLIP representation has been used in various applications such as detection and VLM, could the authors validate the robustness on those tasks?\n* There are many other related methods proposed to improve the robustness of visual encoders [1,2], so I suggest you compare your method with more baselines rather than only comparing with FARE.\n* The novelty of this work is limited. There has already been the unconstrained version of the adversarial optimization problem. And the difference between the proposed loss function Eq. (6) and the regularized loss function Eq. (3) only lies in the trainable weight $\\lambda(x)$. \n* In Line 130, the embedding space is denoted as $\\mathcal{Y}$, while $y$ is used to represent the label of a data point $x$. It would be better to distinguish these two.\n\n[1]: Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models.\n[2]: Improving robustness of CLIP by adversarial training enhanced by brain activity.", "was_revised": true, "final_justification": "After the discussion with the authors, I felt most of my concerns were addressed. They added some results on the segmentation task to show that the method can improve the robustness of CLIP on many downstream tasks. They highlighted their novelty compared to prior work. Although I still think the innovation of using a trainable $\\lambda$ is somewhat limited, I raised my rate given the improved performance and their detailed response.", "rebuttal": "We appreciate that Reviewer TnAL appreciates the superior trade-offs and stability of our method, and finds it important for adversarial robustness.\n\n# Response to Questions\n\n> **In your method, you use a sample-specific tolerance margin, which is related to the norm of the original feature embedding. What would the results be if we used a sample-invariant tolerance?**\n\nThank you for the insightful question. Using a sample-invariant tolerance instead of a sample-specific margin indeed offers an alternative, but it is challenging because it requires extensive, model-dependent hyperparameter tuning that may not generalize. Therefore, the authors chose a sample-specific margin using $m(x)=\\|\\phi_{\\theta_0}(x)\\|^2_2$, which provides an adaptive, scale-invariant tolerance tied to the pre-trained embedding's norm, reducing manual tuning and preserving semantic structure. additionally this relates the constraints geometrically to the cosine similarity of embeddings.\n\n## Experimental Comparison\n\nTo validate the design choice between sample-specific and sample-invariant tolerance settings, experiments were conducted using ViT-B/32 CLIP models. These models were fine-tuned for 5 epochs on ImageNet-100. For sample-invariant settings, $\\rho=0.8$ was empirically chosen as the best performing fixed tolerance among tested candidates. The baseline for comparison was a prior LORE setup with $m(x)=\\|\\phi_{\\theta_0}(x)\\|^2_2$ and $\\rho=0.1$, which represents the sample-specific adaptive tolerance.\n\nThe table below presents the results:\n\n| Method                   | ρ   | Clean                   | $\\epsilon=1$ | $\\epsilon=2$ | $\\epsilon=4$ | Avg Robust Acc            |\n|--------------------------|-----|-------------------------|--------------|--------------|--------------|---------------------------|\n| LORE$^2$ + $m(x)$        | 0.1 | 68.81 (ref)             | 51.51        | 33.71        | 9.93         | 31.72 (ref)               |\n| LORE$^2$                 | 0.6 | 70.12 (+1.31 ↑)         | 24.32        | 18.65        | 0.56         | 14.51 (−17.21 ↓)          |\n| LORE$^2$                 | 0.8 | 69.65 (+0.84 ↑)         | 45.65        | 29.45        | 5.65         | 26.92 (−4.80 ↓)           |\n| LORE$^2$                 | 1.5 | 63.45 (−5.36 ↓)         | 41.23        | 25.47        | 3.64         | 23.45 (−8.27 ↓)           |\n| FARE$^2$                 | —   | 62.23 (−6.58 ↓)         | 40.12        | 23.15        | 3.40         | 22.22 (−9.50 ↓)           |\n|---|---|---|---|---|---|---|\n| LORE$^4$ + $m(x)$        | 0.1 | 66.91 (ref)             | 46.99        | 35.71        | 17.58        | 33.43 (ref)               |\n| LORE$^4$                 | 0.6 | 67.21 (+0.30 ↑)         | 19.50        | 7.21         | 1.23         | 9.31 (−24.12 ↓)           |\n| LORE$^4$                 | 0.8 | 64.32 (−2.59 ↓)         | 40.21        | 32.23        | 14.68        | 29.04 (−4.39 ↓)           |\n| LORE$^4$                 | 1.5 | 55.24 (−11.67 ↓)        | 33.21        | 29.32        | 11.74        | 24.76 (−8.67 ↓)           |\n| FARE$^4$                 | —   | 53.68 (−13.23 ↓)        | 34.85        | 28.78        | 12.87        | 25.50 (−7.93 ↓)           |\n\n\nThe results show that a fixed tolerance of 0.6 is overly restrictive, degrading robustness, while 1.5 is too loose, making LORE behave like FARE. Although fixed values can be tuned per setting, finding a universal optimum is difficult. In contrast, the sample-specific margin ($\\text{LORE}^2$+$m(x)$, $\\text{LORE}^4$+$m(x)$, $\\rho=0.1$) adapts to embedding scale, consistently improving robustness under various perturbations while maintaining clean accuracy and offering greater stability across training $\\epsilon$ values and model types.\n\n# Response to the Strengths And Weaknesses:\n\nWe appreciate your feedback on Section 4 effectively motivating our method and your recognition of its ability to achieve robust results with minimal clean accuracy degradation across diverse settings.\n\n> **In the experiment part, the authors use the image classification task as the measurement to verify the robustness. Since the CLIP representation has been used in various applications such as detection and VLM, could the authors validate the robustness on those tasks?**\n\nThis is an insightful point that has also been raised by another reviewer, and we fully agree on its importance. While our initial evaluation focused on the foundational aspect of visual encoder robustness through image classification, zero-shot capabilities, and out-of-distribution generalization (relevant to VLM applications), we acknowledge that the utility of models like DINOv2 and CLIP extends to various downstream tasks, such as segmentation.\n\nDuring the rebuttal period, we conducted additional experiments to assess LORE's effectiveness on these more diverse downstream tasks for robust models. We found that LORE-hardened models maintain their strong performance even when evaluated on tasks beyond classification. However, due to time constraints, these evaluations are not as comprehensive as our previous experiments and focused specifically on the ViT-S/14 model, which was fine-tuned for 2 epochs on ImageNet for these new task-specific assessments. The results of these evaluations are presented in the table below, further demonstrating the generalizability and practical utility of LORE. This required dedicated experimental setups for each task, involving specialized datasets and metrics, which we are now able to include. The table shows the results of DINOv2 ViT-S/14 on the ADE20k dataset.\n\n| Method | Clean mIoU | EmbedAttack mIoU | \n|---|---|---|\n| Pretrained| 0.42 | 0.01 | \n| $\\text{LORE}^8$|  0.23 | 0.15 | \n| $\\text{FARE}^8$|  0.15 | 0.11 | \n\n\n> **There are many other related methods proposed to improve the robustness of visual encoders [1,2], so I suggest you compare your method with more baselines rather than only comparing with FARE.**\n\nWe appreciate your suggestion. Reference [1] is the original paper introducing the FARE method, which we have used as our primary benchmark and discussed in detail throughout our paper. Regarding [2], unfortunately, we could not include it in our comparison as the paper is not publicly available (there isn't on arXiv and also it isn't accessible through other academic sources).\n\nIt is also worth noting that TeCoA [3] was compared directly in the FARE paper, where FARE demonstrated competitive or superior performance over TeCoA. Since our method, LORE, consistently outperforms FARE across multiple metrics and setups, this indirectly highlights its strength relative to TeCoA as well. Due to space constraints, we did not replicate TeCoA in our paper, but we acknowledge it as a relevant supervised baseline.\n\n[3] Mao, C., Geng, S., Yang, J., Wang, X. E., and Vondrick, C. Understanding zero-shot adversarial robustness for large-scale models. In ICLR, 2023.\n\n> **The novelty of this work is limited. There has already been the unconstrained version of the adversarial optimization problem. And the difference between the proposed loss function Eq. (6) and the regularized loss function Eq. (3) only lies in the trainable weight $\\lambda(x)$**\n\nThank you for the comment. We respectfully disagree with the notion that the novelty of our work is limited. Our approach addresses a critical limitation in standard unsupervised adversarial fine-tuning frameworks, particularly in how they handle the trade-off between robustness and nominal (clean) performance.\n\nIn the FARE method, the regularization coefficient $\\lambda$ is effectively set to 0, meaning there is no mechanism to explicitly control the trade-off between clean and adversarial performance, and no such regularization schemes are introduced in the paper. While Eq. (3) introduces a constant $\\lambda$ as a regularizer, we show in Section 4 that using a fixed $\\lambda$ leads to suboptimal performance and instability.\n\nIn contrast, our method proposes a sample-dependent and adaptive $\\lambda$, modeled as a trainable dual network (Eq. 6), which offers several key advantages:\n\n- Formulating the problem as a constrained optimization problem, not just an ad hoc regularization of the original objective.\n- It provides principled control over the robustness–accuracy trade-off via constrained optimization (Section 6.1).\n- Stabilizing the network during training, whenever needed throughout training, only activated when the model is losing its nominal performance (Figure 10, 11. Appendix H)\n- In Figure 1b, we empirically show that by adaptively adjusting $\\lambda$ (the dual function), LORE achieves **a strictly better Pareto frontier** than naively regularized losses with **any** fixed $\\lambda$\n- As demonstrated in Section G.1, sample-based regularization performs significantly better than input-independent formulations.\n- It improves Out-of-Distribution Robustness, Embedding Interpretability, Image Classification accuracy and by dynamically adjusting the regularization strength based on each input sample (Section 6.2, 6.3). \n\nDespite the structural simplicity of our approach, we emphasize that LORE achieves strong empirical performance across multiple backbones and datasets, as shown in Section 6 and Appendix J. \n\nAdditionally, we provide theoretical justification for LORE in:\n\n- Section 6.1, which presents a formal analysis of the controllability of the trade-off via the constraint threshold $\\rho$, and\n- Appendix B, where we analyze the impact of dual variable parameterization on suboptimality bounds.\n\nOverall, we believe our formulation offers a practical and theoretically grounded improvement over previous methods, resolving a previously overlooked issue in unsupervised adversarial fine-tuning.\n\n> **In Line 130, the embedding space is denoted as ‘$\\mathcal{Y}$’, while ‘$y$’ is used to represent the label of a data point ‘$x$’. It would be better to distinguish these two.**\n\nThanks for the suggestion. We will address it in the final version."}, {"review_id": "Ly9iWl9V4P", "reviewer": "Reviewer_Jf2T", "summary": "This paper proposes Lagrangian-Optimized Robust Embeddings (LORE), a novel unsupervised adversarial fine-tuning framework, which enhances the robustness of visual encoders while preserving nominal performance without relying on heuristic loss weighting. Extensive experiments show that LORE significantly improves zero-shot adversarial robustness with minimal degradation in clean data accuracy and outperforms FARE consistently. However, the paper still needs further refinement and revision. Further experimental validation and analysis are recommended to be added.", "questions": "Some problems are:\n1、In this paper, FARE is considered as a baseline for evaluating the proposed approach. Hence, it is important to highlight the contributions of the proposed method LORE compared with FARE in the introduction.\n\n2、In Section 2 Related Work, there are recent studies that highlight the vulnerability of VLMs to adversarial attacks. However, the corresponding description of the relevant references is just too limited to fit the regular structure of a paper.\n\n3、For Figure 1(a), the meaning of varying `\\rho` is not mentioned in the paper. Further explanation is necessary to be added to the manuscript.\n\n4、To empirically validate the approach, the majority of our experiments were conducted and all models were trained for 2 epochs on ImageNet or 5 epochs on ImageNet-100. Does this signify that the structure of the model is very simple and therefore the training process is comparatively fast?\n\n5、A comprehensive evaluation of clean and adversarial performance is conducted across various image classification datasets in Table 3. But there are experimental results of ‘0’ and no experimental results for `\\epsilon`=8. Further experimental validations are necessary to be added to the manuscript.", "limitations": "Mentioned in the paper", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Pros: Extensive experiments and Interesting Topic.\n\nCons:  Further validation and analysis are recommended to be added.", "was_revised": true, "final_justification": "I have read the authors' response. The authors have addressed some of my concerns. However, a major revision is required. In particular, more experiments should be given. Therefore, I will keep my original rating.", "rebuttal": "# Response to Reviewer Jf2T\n\nWe appreciate that Reviewer Jf2T finds our method novel, our experimental results significant, and the topic interesting.\n\n# Response to the questions\n\n> Answer for Q1\n\nThank you for this valuable suggestion. While we have already cited the FARE paper in the introduction (Line 41), we agree that a clearer comparison between LORE and FARE at the outset would better contextualize our contribution. In the revised version, we will explicitly highlight the key distinctions and motivations for proposing LORE in contrast to FARE within the introduction.\n\n> Answer for Q2\n\nIn the initial version of the paper, due to space limitations, we kept the Related Work section concise and focused on maintaining a coherent narrative rather than elaborating on each individual reference. Additionally, we would like to clarify that our primary focus is on improving the robustness of visual encoders, rather than VLMs as a whole. While we acknowledge the growing body of work on adversarial robustness in VLMs, we have deliberately centered our discussion on methods most relevant to image encoder fine-tuning.\n\nThat said, we appreciate the suggestion and in the camera-ready version, where we are allowed an additional page, we plan to expand this section and provide deeper analysis and categorization of relevant literature, particularly those highlighting VLM vulnerabilities and defenses.\n\n> Answer for Q3\n\nThank you for your comment. To clarify, Figure 1(a) is not related to varying the hyperparameter $\\rho$, $\\rho=0.1$ was used for this experiment. If the comment is instead referring to Figure 1(b), then as explained in Section 6.1, $\\rho$ defines the threshold for the proximity constraint (see Eq. 5) and directly controls the trade-off between nominal (clean) accuracy and adversarial robustness. Varying $\\rho$ adjusts the allowed deviation from the reference encoder, enabling a principled way to navigate this trade-off.\n\nWe included Figure 1(b) in Section 4 to motivate the LORE framework by showcasing its empirical advantage over naïve regularization methods, even before formally introducing the full formulation. Additionally, we explicitly reference and explain the meaning of varying $\\rho$ in Lines 229–231, where we describe how LORE achieves controllability through this parameter. The theoretical and empirical roles of $\\rho$ are then discussed in full detail in Section 6.1 and are further supported by our robustness suboptimality analysis.\n\nWe hope this clears up the confusion, and we are happy to provide further clarification if needed.\n\n> Answer for Q4\n\nThe models we fine-tune, such as CLIP ViT-B/32, DINOv2, and ConvNeXt-B, are large, pre-trained foundation models, not simple architectures. The relatively low number of fine-tuning epochs (2 for ImageNet and 5 for ImageNet-100) is not due to model simplicity; it signifies the efficiency of this approach for fine-tuning on large-scale datasets.\n\nThe number of epochs was selected based on training convergence behavior. Specifically, for the large-scale ImageNet dataset, 2 epochs were sufficient to reach strong convergence, **consistent with the FARE paper**, and yielded robust performance in both in-domain and zero-shot evaluations. Similarly, given the smaller size and diversity of ImageNet-100, we found that 5 epochs were adequate. Additionally, we provide training curves for several experiments in Appendix G.2 (e.g., Figure 10), where the model's convergence can be clearly observed.\n\n> Answer for Q5\n\nThank you for your comment. The '0' values for the original CLIP model (rows labeled \"CLIP\") under adversarial attacks in Table 3 are expected and reflect the severe vulnerability of the base model. This underscores the motivation for robust fine-tuning methods like LORE, as CLIP without adversarial training exhibits negligible robustness. As shown in the table, LORE consistently outperforms FARE on average across different settings, further validating its effectiveness. Regarding other '0' values in Table 3 (e.g., under evaluation at $\\epsilon=4$), these correspond to models that were trained with lower perturbation strengths (e.g., $\\epsilon=1,2$ for FARE1 and LORE2). As expected, models trained with smaller perturbations typically do not generalize well to stronger attacks—highlighting a common limitation of fixed-$\\epsilon$ training. \n\nAs for high-intensity perturbations (e.g., $\\epsilon=8$), we have indeed conducted those evaluations. The corresponding results are reported in Tables 9 and 10 of Appendix J.4. Due to space constraints in the main manuscript, these extended results are included in the appendix, as noted in Line 188 of the Results section.\n\nRegarding further experimental validations, in addition to the results presented in Section 6, we conducted additional evaluations in **Appendix J**, including Square Attack, a black-box adversarial attack, and evaluation under Gaussian noise corruption.\n\n# Further experimental validation\n\n## 1. Validating our choice of sample-specific margins.\n\nTo validate the design choice between sample-specific and sample-invariant tolerance settings, experiments were conducted using ViT-B/32 CLIP models. These models were fine-tuned for 5 epochs on ImageNet-100. For sample-invariant settings, $\\rho=0.8$ was empirically chosen as the best-performing fixed tolerance among tested candidates. The baseline for comparison was a prior LORE setup with $m(x)=\\|\\phi_{\\theta_0}(x)\\|^2_2$ and $\\rho=0.1$, which represents the sample-specific adaptive tolerance.\n\nThe table below presents the results:\n\n| Method                   | ρ   | Clean                   | $\\epsilon=1$ | $\\epsilon=2$ | $\\epsilon=4$ | Avg Robust Acc            |\n|--------------------------|-----|-------------------------|--------------|--------------|--------------|---------------------------|\n| LORE$^2$ + $m(x)$        | 0.1 | 68.81 (ref)             | 51.51        | 33.71        | 9.93         | 31.72 (ref)               |\n| LORE$^2$                 | 0.6 | 70.12 (+1.31 ↑)         | 24.32        | 18.65        | 0.56         | 14.51 (−17.21 ↓)          |\n| LORE$^2$                 | 0.8 | 69.65 (+0.84 ↑)         | 45.65        | 29.45        | 5.65         | 26.92 (−4.80 ↓)           |\n| LORE$^2$                 | 1.5 | 63.45 (−5.36 ↓)         | 41.23        | 25.47        | 3.64         | 23.45 (−8.27 ↓)           |\n| FARE$^2$                 | —   | 62.23 (−6.58 ↓)         | 40.12        | 23.15        | 3.40         | 22.22 (−9.50 ↓)           |\n|---|---|---|---|---|---|---|\n| LORE$^4$ + $m(x)$        | 0.1 | 66.91 (ref)             | 46.99        | 35.71        | 17.58        | 33.43 (ref)               |\n| LORE$^4$                 | 0.6 | 67.21 (+0.30 ↑)         | 19.50        | 7.21         | 1.23         | 9.31 (−24.12 ↓)           |\n| LORE$^4$                 | 0.8 | 64.32 (−2.59 ↓)         | 40.21        | 32.23        | 14.68        | 29.04 (−4.39 ↓)           |\n| LORE$^4$                 | 1.5 | 55.24 (−11.67 ↓)        | 33.21        | 29.32        | 11.74        | 24.76 (−8.67 ↓)           |\n| FARE$^4$                 | —   | 53.68 (−13.23 ↓)        | 34.85        | 28.78        | 12.87        | 25.50 (−7.93 ↓)           |\n\nFixed tolerances show limitations: 0.6 is too restrictive, hurting robustness, while 1.5 is too permissive, mimicking FARE. Finding a universally optimal fixed tolerance is difficult. The sample-specific margin (e.g., LORE$^2$+$m(x)$ and LORE$^4$+$m(x)$ with $\\rho=0.1$) is superior, adapting to embedding scale, consistently offering better robustness across various perturbations and training conditions while maintaining clean accuracy and stability.\n\n## 2. Extentions to supervised finetuning\n\nWe also demonstrate that LORE's proximity constraints can be extended to supervised adversarial fine-tuning, applying it to TeCoA [1]. We introduce two variants: TeCoA + $l_2$, which constrains the $l_2$ distance between fine-tuned and pretrained image embeddings, and TeCoA + KL, which uses KL divergence to constrain output probability distributions. Experiments with ViT-B/32 CLIP models on ImageNet-100 demonstrate that both regularized methods consistently enhance robustness and clean accuracy compared to baseline TeCoA, across various perturbation strengths. This proves LORE's broad applicability and the stabilizing effect of proximity-based constraints in both embedding and probability spaces for supervised adversarial fine-tuning. For more details, refer to our comment to Reviewer CYtr.\n\n\n---\n\n[1] Mao, C., Geng, S., Yang, J., Wang, X. E., and Vondrick, C. Understanding zero-shot adversarial robustness for large-scale models. In ICLR, 2023."}]

Author rebuttal:

Meta-review: This paper proposes LORE, a principled adversarial fine-tuning framework using sample-dependent constrained optimization. It addresses early instability and robustness–accuracy trade-offs with strong empirical results across classification, segmentation, and supervised settings. 
Reviewers raised major concerns around novelty, downstream evaluation, and baseline coverage. 
Authors added segmentation results, supervised extensions (TeCoA+LORE), and clarified theoretical and empirical distinctions from prior work like FARE. 
Given the practical use-case and clarity of the paper, I recommend acceptance.
Decision: Accept (poster)