Title: LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders
Venue: NeurIPS 2025
Accepted: yes
Keywords: Adversarial finetuning

Adversarial Robustness

Unsupervised Learning

Visual Encoders

Constrained Optimization
Abstract: Visual encoders have become fundamental components in modern computer vision pipelines. However, ensuring robustness against adversarial perturbations remains a critical challenge. Recent efforts have explored both supervised and unsupervised adversarial fine-tuning strategies. We identify two key limitations in these approaches: (i) they often suffer from instability, especially during the early stages of fine-tuning, resulting in suboptimal convergence and degraded performance on clean data, and (ii) they exhibit a suboptimal trade-off between robustness and clean data accuracy, hindering the simultaneous optimization of both objectives. To overcome these challenges, we propose **L**agrangian-**O**ptimized **R**obust **E**mbeddings (LORE), a novel unsupervised adversarial fine-tuning framework. LORE utilizes constrained optimization, which offers a principled approach to balancing competing goals, such as improving robustness while preserving nominal performance. By enforcing embedding-space proximity constraints, LORE effectively maintains clean data performance throughout adversarial fine-tuning. Extensive experiments show that LORE stabilizes training and significantly improves zero-shot adversarial robustness with minimal degradation in clean data accuracy. Furthermore, we demonstrate the effectiveness of the adversarially fine-tuned image encoder in out-of-distribution generalization and enhancing the interpretability of image embeddings. The code is available on [GitHub](https://github.com/Theborna/LORE-Lagrangian-Optimized-Robust-Embeddings).

LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders

Abstract

Visual encoders have become fundamental components in modern computer vision pipelines. However, ensuring robustness against adversarial perturbations remains a critical challenge. Recent efforts have explored both supervised and unsupervised adversarial fine-tuning strategies. We identify two key limitations in these approaches: (i) they often suffer from instability, especially during the early stages of fine-tuning, resulting in suboptimal convergence and degraded performance on clean data, and (ii) they exhibit a suboptimal trade-off between robustness and clean data accuracy, hindering the simultaneous optimization of both objectives. To overcome these challenges, we propose L agrangianO ptimized R obust E mbeddings (LORE), a novel unsupervised adversarial fine-tuning framework. LORE utilizes constrained optimization, which offers a principled approach to balancing competing goals, such as improving robustness while preserving nominal performance. By enforcing embedding-space proximity constraints, LORE effectively maintains clean data performance throughout adversarial fine-tuning. Extensive experiments show that LORE stabilizes training and significantly improves zero-shot adversarial robustness with minimal degradation in clean data accuracy. Furthermore, we demonstrate the effectiveness of the adversarially fine-tuned image encoder in out-of-distribution generalization and enhancing the interpretability of image embeddings. The code is available on GitHub.

1 Introduction

In recent years, embeddings from foundation Vision-Language Models (VLMs), such as CLIP [Radford et al., 2021], have transformed downstream tasks, including classification [Conde and Turgutlu, 2021], object detection [Zhong et al., 2021], segmentation [Xu et al., 2022], and image generation [Saharia et al., 2022]. These models, trained on large-scale web data, semantically align inputs from different modalities into a joint embedding space. enabling remarkable zeroshot generalization capabilities like zero-shot image classification, where virtually any class can be encoded via its textual description.[Radford et al., 2021, Qian and Hu, 2024].

However, visual encoders, such as CLIP models, exhibit significant vulnerabilities. Adversarial attacks [Goodfellow et al., 2015] and backdoor vulnerabilities [Bai et al., 2024, Liang et al., 2024] can lead to erroneous and misleading embeddings. These challenges highlight the need for robust alignment methods in large, pre-trained visual encoders. To address this, adversarial fine-tuning plays a crucial role in enhancing the robustness of deep neural networks against adversarial examples

[Pang et al., 2020, Kurakin et al., 2017, Madry et al., 2019]. Moreover, maintaining accuracy and correct embeddings is vital for downstream applications, as it ensures the model's versatility and reliability without compromising clean data performance, i.e., its nominal performance [Tsipras et al., 2019, Dobriban et al., 2022]. While recent works have proposed effective methods to enhance the robustness of visual encoders [Schlarmann et al., 2024, Mao et al., 2023], they often struggle to manage the trade-off between robustness and nominal performance effectively. Therefore, principled approaches are required to explicitly balance this trade-off [Zhang et al., 2019a].

In this work, we focus on the vision encoder and introduce L agrangianO ptimized R obust E mbeddings (LORE), a novel unsupervised adversarial fine-tuning framework that effectively balances robustness and nominal performance through a novel constrained optimization framework. While we primarily target CLIP's image encoder, due to its widespread adoption and strong zero-shot capabilities, we also demonstrate the applicability of LORE to other foundation models such as DINOv2 [Oquab et al., 2024]. LORE enforces proximity to a reference model in the embedding space by leveraging the Lagrangian dual method [Bertsekas, 1997], enabling robustness improvements without compromising semantic fidelity on clean data.

Our main contributions are as follows: 1) We propose LORE, a novel unsupervised adversarial fine-tuning framework that achieves a superior empirical balance on the trade-off between robustness and nominal performance, with a hyperparameter ( ρ ) enabling principled control over this tradeoff. 2) We demonstrate the effectiveness of LORE in image classification, significantly improving zero-shot adversarial robustness over CLIP while maintaining minimal degradation in clean accuracy. 3) We analyze the behavior of the adversarially fine-tuned LORE-CLIP encoder in terms of outof-distribution generalization and show that it improves the interpretability of image embeddings, highlighting its advantages over other unsupervised adversarial fine-tuning baselines.

2 Related Work

Adversarial Robustness: Attacks and Defenses. Adversarial attacks exploit neural network vulnerabilities by crafting imperceptible perturbations, leading to misclassifications [Goodfellow et al., 2015, Szegedy et al., 2014]. Defenses like adversarial training improve robustness but struggle with generalization [Madry et al., 2019]. Balancing robustness and nominal performance remains challenging [Tsipras et al., 2019, Zhang et al., 2019b], necessitating more versatile defenses, especially in multimodal settings. A principled approach to managing the trade-off is Constrained Optimization: [Robey et al., 2021] pioneered using a Lagrangian duality framework to enforce explicit constraints, avoiding heuristic loss weighting.

Robustness in Foundation Encoders. Recent studies have highlighted the vulnerability of foundation visual encoders, e.g. CLIP and DINOv2, to adversarial attacks, demonstrating that embeddings can be disrupted [Zhang et al., 2024]. To address this, supervised defenses like TeCoA [Mao et al., 2023] and MMCoA [Zhou et al., 2024] improve robustness by aligning adversarial and clean embeddings, often relying on auxiliary information like text, using simplistic text descriptions, or attention maps,[Yu et al., 2024], assuming normalized embeddings. PMG-AFT [Wang et al., 2024] extends this approach by minimizing the distance between adversarial features and those of the pre-trained model. In the label-free setting, Unsupervised Adversarial Fine-Tuning seeks to achieve robustness by modifying contrastive losses, leading to methods like Robust Contrastive Learning (RoCL) [Kim et al., 2020] and the decoupled approach of DeACL [Zhang et al., 2022]. Among unsupervised fine-tuning strategies for the CLIP vision encoder, FARE [Schlarmann et al., 2024] stands out as a direct baseline. However, FARE and similar methods often lack explicit mechanisms to balance robustness and task performance, resulting in a suboptimal trade-off. Due to its unsupervised nature, we consider FARE a baseline for evaluating our proposed approach.

3 Background

Adversarial Attacks. Adversarial attacks aim to deceive machine learning models by adding imperceptible perturbations to inputs. A prominent example is the Projected Gradient Descent (PGD) attack, which iteratively maximizes a loss function J (e.g., cross-entropy) to craft adversarial examples. Starting from an input x , PGD updates:

(a)

0

15

30

45

Robust Accuracy (%)

48

56

64

72

Clean Accuracy (%)

=0.0

=0.1

=0.2

=0.5

=1.0

=0.01

=0.03

=0.07

=0.1

=0.3

Naive Reg.

LORE

(b)

CleanAccuracy

c=10-c=8-g=6

===10 (LORE)

g=4-c=2

60

50

40

30

20

10

Iteration

0

0

200

400

600

800

1k

1.2k

1.4k

where y is the true label, η is the step size, and ε bounds the ℓ ∞ -norm of the perturbation. The projection Proj B ( x,ε ) ensures x ′ stays within the allowed perturbation set. This procedure reveals the susceptibility of models like CLIP to adversarial manipulations.

Unsupervised Adversarial Fine-Tuning. FARE [Schlarmann et al., 2024] proposed an unsupervised method for fine-tuning CLIP vision encoder ϕ . FARE's goal is to minimize the Euclidean distance between clean and perturbed embeddings under worst-case perturbation conditions:

where ϕ θ 0 refers to the original (frozen) CLIP encoder, and δ represents the adversarial perturbation. Unlike methods like TeCOA and MMCoA, FARE does not require text inputs or labels, relying solely on the image encoder. This loss function aims to (i) enhance the robustness of image embeddings generated by ϕ θ , and (ii) ensure that the image encoder ϕ θ remains close to the original encoder ϕ θ 0 to maintain performance on clean data, preserving nominal performance.

4 Preliminary Experiments

To motivate our approach, we conduct a series of preliminary experiments that reveal critical limitations in adversarial fine-tuning with the loss defined in Eq. (2). In particular, we examine two key challenges: (i) instability during early fine-tuning, and (ii) poor robustness-accuracy trade-offs under naive regularization.

(i) Fig. 1a depicts the clean data accuracy of a CLIP image encoder during adversarial fine-tuning using the loss function defined in (2). The results indicate substantial declines in clean data accuracy, particularly at elevated perturbation strengths (e.g., ε = 8 , 10 ) 2 . Additionally, there is notable instability during the early stages of training. This instability hampers optimal convergence and diminishes performance on clean data. Consequently, the model may converge to local minima, losing significant pretrained knowledge, despite achieving robustness.

(ii) As a straightforward approach to mitigate the degradation of clean data accuracy, one may introduce a naive regularization term to the loss (2), aiming to minimize the following objective:

where λ serves as a hyperparameter to regulate the strength of the regularization. We assessed the trade-off between clean and robust accuracy by plotting the Pareto frontier across various regularization strengths, as illustrated in Fig. 1b. The blue curve represents naive regularization (with

Algorithm 1 Lagrangian-Optimized Robust Em- beddings (LORE)

θ

t

θ

0

θ

t

≈

θ

0

ϕ

θ

0

(

x

)

ϕ

θ t

(

x

)

ρm

(

x

)

ϕ

θ t

(

x

+

δ

)

d

Input

Encoder

Encoder

ndui

varying λ ), revealing a steep trade-off, with robustness gains costing a considerable expense to clean data accuracy. Conversely, the proposed method, LORE (depicted by orange stars, with varying ρ ), demonstrates a superior Pareto front, consistently attaining higher robust accuracy at comparable or enhanced levels of clean data accuracy. Moreover, LORE maintains stable clean data accuracy throughout training (Fig. 1a), even with large perturbation strengths.

5 LORE: Lagrangian-Optimized Robust Embeddings

Our findings in Section 4 highlight the need for a principled method that both stabilizes adversarial fine-tuning and offers a favorable robustness-accuracy trade-off. Constrained learning offers a structured approach to balancing competing objectives like robustness and nominal accuracy. The constrained optimization problem for adversarial robustness in classification, as explored in Robey et al. [2021], can be formulated as:

where ℓ is the loss function, ρ ≥ 0 controls the nominal performance level, f θ is the classifier parameterized by θ , and ∆ denotes the set of possible perturbations. The presence of infinitely many constraints indexed by the input space makes this a semi-infinite learning problem, which allows the application of tools from semi-infinite optimization theory.

Extending the optimization framework of (4) to an unsupervised setting, we propose enforcing proximity to a reference model ϕ θ 0 (e.g., a pre-trained model) in the embedding space R k , where k denotes the embedding dimension, thereby eliminating the dependence on y . Our formulation is:

Here, d : R k × R k → R + is a divergence metric, and m : X → R + is a sample-specific tolerance margin, independent of θ , used to modulate the constraint across inputs, defining a per-sample margin within which deviations are considered acceptable. For example, when using Euclidean distance, setting m ( x ) = ∥ ϕ θ 0 ( x ) ∥ 2 ensures the constraint becomes scale-invariant with respect to the original embedding magnitude.

Solving the Constrained Problem. Constrained optimization in deep learning presents significant challenges due to the high-dimensional and non-convex nature of neural networks. Therefore,

following [Robey et al., 2021, Chamon et al., 2020, Chamon and Ribeiro, 2021], we leverage Lagrangian duality to obtain approximate solutions for (5) that generalize effectively with respect to both adversarial and nominal performance. Specifically, we solve the dual optimization problem:

where λ ω : X → R + is the dual network parameterized by ω , with outputs made non-negative via Softplus activation, which ensures valid Lagrange multipliers and avoids the sparse gradients of ReLU. This network simplifies the original optimization over all functions λ : X → R + . By jointly optimizing θ and ω , our method achieves two key objectives: (i) enhancing the robustness of the model's embeddings against adversarial examples through the worst-case perturbation δ ; and (ii) maintaining proximity to the reference model's embeddings on clean data, i.e., ϕ θ 0 ( x ) , to preserve nominal performance. Note that this adaptive framework fundamentally differs from regularization, and is automatically enforced if embeddings diverge, The theoretical gaps and limitations introduced by our formulation and parameterization are detailed in Appendix B.

Practical Implementation. For computational efficiency, we implement the dual network as λ ω ( x ) = F ω ( ϕ θ 0 ( x )) , where F ω is a lightweight two-layer MLP that takes the reference model's embeddings as input. This leverages readily available semantic information from ϕ θ 0 ( x ) for inputdependent constraint weighting with no extra embedding computation. In Appendix G, we provide a detailed analysis of the λ ω architecture and its impact on performance. Specifically, we empirically demonstrate that modeling the dual variable as an input-dependent network, i.e., λ ω ( x ) , significantly outperforms the input-independent variant λ ω . We instantiate the divergence metric as the squared ℓ 2 distance, d ( z 1 , z 2 ) = ∥ z 1 -z 2 ∥ 2 2 , and set m ( x ) = ∥ ϕ θ 0 ( x ) ∥ 2 2 to enforce scale-invariant constraints. Substituting these choices into our general framework yields the following optimization problem:

The adversarial perturbation δ is approximated using projected gradient descent (PGD) within an ℓ ∞ -bounded set. Problem (7) employs a primal-dual framework to balance robustness and accuracy dynamically. The primal step optimize ϕ θ for robustness while maintaining proximity to ϕ θ 0 on clean data. The dual step updates parameters ω to strengthen constraints where d ( ϕ θ ( x ) , ϕ θ 0 ( x )) exceeds ρm ( x ) , preventing overly conservative solutions. This approach is crucial for vision-language models, ensuring semantic alignment via embedding proximity, and eliminates heuristic loss weighting by treating robustness and accuracy as competing objectives within a Lagrangian duality framework.

Intuitively, our scale-invariant constraint ∥ ϕ θ ( x ) -ϕ θ 0 ( x ) ∥ 2 2 ≤ ρ ∥ ϕ θ 0 ( x ) ∥ 2 2 preserves angular relationships. As detailed in Appendix F, this bounds the deviation in cosine similarity S C ( u, v ) = u T v/ ( ∥ u ∥∥ v ∥ ) , ensuring | S C ( ϕ θ 0 ( x ) , v ) -S C ( ϕ θ ( x ) , v ) | ≤ 2 √ ρ for any vector v ∈ R n . This preserves image-text alignment in vision-language models during adversarial fine-tuning.

Optimization. We optimize the LORE objective using an alternating training procedure. For each batch x sampled from the dataset D , we first generate adversarial examples δ . Then, we perform K steps of gradient descent to update the encoder parameters θ . After the K primal updates, we take a single gradient step to update the dual network parameters ω , using the clean loss L clean (as defined in Algorithm 1) to guide the adjustment of λ ω ( x ) . This alternating update strategy enables the model to dynamically balance robustness and clean performance, with K acting as a hyperparameter that controls the frequency of primal updates. The full procedure is summarized in Algorithm 1. Although non-convex convergence guarantees are challenging, this alternating optimization is empirically stable and effective in our experiments. Ablation studies on the choice of K and constraint threshold ρ are provided in Appendix I.

6 Main Results

Although LORE and FARE optimize the identical adversarial loss, LORE's proximity constraints serve as a non-trivial, sample-based regularizer that steers training away from poor local minima, yielding consistently stronger robustness without sacrificing clean accuracy. We now present a suite of experiments to substantiate this claim.

We begin by demonstrating how LORE enables controlled trade-offs between robustness and nominal performance. Next, we evaluate zero-shot image classification under both clean and adversarial

10

2

10

1

10 0

Rho (

)

70.0

70.5

71.0

71.5

72.0

Clean Accuracy

Clean Accuracy

Robust Accuracy

10

2

10

1

10 0

Rho (

)

0.955

0.960

0.965

0.970

0.975

0.980

0.985

0.990

cos

(

(

x

),

0

(

x

)

)

Cosine Similarity Comparison

cos

(

(

x

),

0

(

x

)

)

cos(

(

x

+ ),

(

x

))

10

2

10

1

10 0

Rho (

)

2.5

3.0

3.5

4.0

4.5

5.0

(

x

+ )

(

x

)

2

L2 Distance Comparison

(

x

+ )

(

x

)

2

(

x

)

0

(

x

)

2

7.5

10.0

12.5

15.0

17.5

20.0

22.5

25.0

Robust Accuracy

0.84

0.86

0.88

0.90

0.92

0.94

cos(

(

x

+ ),

(

x

))

0.125

0.150

0.175

0.200

0.225

0.250

0.275

0.300

(

x

)

0

(

x

)

2

settings. We then analyze LORE's benefits for embedding interpretability and out-of-distribution robustness. Further results and ablations are provided in Appendix J.

Experimental Setup. To empirically validate our approach, we conduct the majority of our experiments using the ViT-B/32 CLIP model. In the second part of our evaluation (Section 6.3), we further assess the effectiveness of LORE on alternative architectures such as ConvNeXt-B CLIP and DINOv2. We use both ImageNet and ImageNet-100, a curated subset of ImageNet, as training datasets throughout our experiments. Additional details for each experiment, including figures and tables, are provided in Appendix C. Throughout the results, superscripts denote the ℓ ∞ perturbation bound ε used during adversarial fine-tuning; for instance, LORE 2 refers to a model trained with ε = 2 / 255 . We consistently compare our method with FARE, which serves as the unconstrained counterpart to LORE and also utilizes unsupervised adversarial fine-tuning. All models were trained for 2 epochs on ImageNet or 5 epochs on ImageNet-100. Experiments were conducted using 8 NVIDIA HGX H100 80GB GPUs.

6.1 Controlling the Robustness-Accuracy Trade-off

The parameter ρ directly governs the allowed clean embedding deviation, offering a more precise control knob for the robustness-accuracy trade-off than heuristic weightings. While such trade-offs have been extensively studied in supervised adversarial robustness [Zhang et al., 2019a, Xiao et al., 2024, Javanmard et al., 2020], they remain underexplored in unsupervised adversarial fine-tuning.

We apply our constrained optimization framework to probe robustness-accuracy trade-offs. Using a proximity constraint, we restrict the hypothesis space from the full family of encoders H to the fidelity-constrained subset H ρ = { ϕ ∈ H | d ( ϕ ( x ) , ϕ θ 0 ( x )) ≤ ρm ( x ) for a.e. x ∈ X } . Intuitively, H ρ is a neighborhood of the clean encoder ϕ θ 0 : for small ρ , it's a tight ball around ϕ θ 0 , and as ρ grows it expands until it recovers the full class H .

We then define the pointwise adversarial loss as ℓ adv ( ϕ, x ; ϕ θ 0 ) ≜ max δ ∈ ∆ d ( ϕ ( x + δ ) , ϕ θ 0 ( x ) ) , which induces the optimization

Constraining the search space stabilizes training by avoiding extreme encoder shifts. However, it does so at the cost of theoretical optimality-since restricting H may exclude the global minimizer. In practice, the constrained formulation can nonetheless outperform the unconstrained one by steering clear of poor local minima.

Assumption 6.1. Assume d ( · , · ) is a nonnegative metric on the embedding space. Moreover, for any fixed reference embedding c , the map u ↦→ d ( u, c ) is L d -Lipschitz in its first argument, i.e., ∣ ∣ d ( u 1 , c ) -d ( u 2 , c ) ∣ ∣ ≤ L d ∥ u 1 -u 2 ∥ for some norm ∥ · ∥ .

Theorem 6.2 (Robustness Suboptimality Bounds) . Let:

Since H ρ ⊂ H , we have R ρ ≥ R . Moreover, the suboptimality gap satisfies:

Average

zoom blur

speckle noise

spatter

snow

shot noise

saturate

pixelate

motion blur

jpeg compression

impulse noise

glass blur

gaussian noise

gaussian blur

frost

fog

elastic transform

defocus blur

contrast

brightness

No Shift

20%

25%

30%

35%

40%

45%

50%

55%

60%

FARE²

LORE²

CLIP

50.3%

47.4%

41.2%

59.8%

58.6%

55.6%

(a)

1

2

4

6

8

10

Training & Evaluation Epsilon (

)

0.16

0.18

0.20

0.22

0.24

0.26

0.28

0.30

Average Cosine Similarity

CLIP

FARE

LORE

(b)

Here ϕ ∗ and ϕ ∗ ρ are the minimizers in H and H ρ , respectively; L ∗ ρ and L ′ are their associated Lipschitz constants; and adversarial perturbations are ℓ ∞ -bounded by ε in R k . (Proof in Appendix A.)

The bound shows that the excess loss R ρ -R grows with both the attack radius ε and the encoder shift ∥ ϕ ∗ ρ -ϕ ∗ ∥ . While we employ squared ℓ 2 geometry for its optimization convenience, this choice only affects the numerical constants-absorbed into the embedding norms and local Lipschitz terms.

Intuitively, increasing ρ relaxes the proximity constraint, granting the model more flexibility (and thus greater robustness) at the expense of nominal accuracy, whereas smaller ρ keeps the model tightly aligned with the pre-trained encoder, favoring clean performance.

We empirically evaluate LORE's control over the robustness-accuracy frontier by training ten models with varying ρ . Figure 1b shows the resulting Pareto frontier compared to additive-regularization baselines (Eq. 2), and Figure 3 reports clean-data accuracy and adversarial success rate on the test set. These experiments confirm both the practical tightness of our theoretical bound and the superior trade-off achieved by our constrained-optimization framework.

6.2 Out-of-Distribution Robustness and Embedding Interpretability

To assess how unsupervised adversarial fine-tuning affects out-of-distribution (OOD) robustness, we evaluate a ViT-B/32 CLIP model fine-tuned with LORE on ImageNet, using the ImageNetC[Hendrycks and Dietterich, 2019], which applies common corruptions to simulate distribution shifts. Since adversarial fine-tuning often reduces clean accuracy, it can lead to degraded OOD performance. As shown in Fig. 4a, FARE 2 suffers a substantial drop under corruptions, whereas LORE 2 maintains much stronger generalization. In fact, for certain perturbations (e.g., jpeg compression ), LORE 2 even outperforms the original pretrained CLIP model, underscoring its robustness to distribution shift.

Adversarial training is also known to enhance embedding interpretability and cross-modal alignment [Croce et al., 2025]. To measure this effect in the CLIP vision-language space, we generate 150 natural-language templates using GPT-4 (e.g., 'This is a photo of {}') to capture class semantics, then compute the average cosine similarity between each clean image embedding and its corresponding text embedding. Figure 4b shows that both FARE 2 and LORE 2 improve over the pretrained baseline, but LORE 2 consistently achieves higher alignment across all training and evaluation ε settings, demonstrating superior semantic representation.

6.3 Image Classification

Zero-shot Image Classification. We evaluate adversarial robustness of the ViT-B/32 CLIP vision encoder on 13 zero-shot benchmarks from CLIP-benchmark 3 , all originally trained on ImageNet. For each dataset, we compare LORE against FARE under both clean and adversarial conditions.

Method

Backbone

Clean

ε

= 1

ε

= 2

ε

= 4

ε

= 8

FARE

2

ViT-B/16

70.40

53.0

34.9

8.8

0.06

LORE

2

ViT-B/16

74.7

62.3

47.7

20.8

0.74

FARE

4

ViT-B/16

58.1

47.7

37.1

19.0

2.22

LORE

4

ViT-B/16

71.5

62.3

53.3

34.7

9.06

FARE

2

ViT-B/32 LAION

65.4

41.0

19.0

2.02

0.02

LORE

2

ViT-B/32 LAION

70.2

51.8

31.4

7.26

0.04

FARE

4

ViT-B/32 LAION

52.7

36.7

23.4

6.72

0.20

LORE

4

ViT-B/32 LAION

68.4

44.7

29.6

10.7

0.62

FARE

2

ConvNeXt-B

74.2

61.6

46.1

16.7

0.22

LORE

2

ConvNeXt-B

75.6

64.9

52.4

25.6

1.04

FARE

4

ConvNeXt-B

70.6

61.6

52.3

32.7

6.48

LORE

4

ConvNeXt-B

73.5

66.0

58.1

40.3

10.4

Method

Backbone

Clean

ε

= 1

ε

= 2

ε

= 4

ε

= 8

FARE

4

ViT-S/14

69.2

60.7

51.2

30.7

2.91

LORE

4

ViT-S/14

77.3

60.8

50.0

30.3

5.8

FARE

8

ViT-S/14

55.1

48.9

42.7

30.0

8.13

LORE

8

ViT-S/14

75.1

55.9

48.8

36.8

13.7

FARE

4

ViT-B/14

78.3

71.9

64.1

44.0

6.51

LORE

4

ViT-B/14

80.2

73.5

67.1

49.6

11.2

FARE

8

ViT-B/14

69.4

63.8

57.8

44.1

16.0

LORE

8

ViT-B/14

80.5

65.0

59.7

48.5

21.8

1

2

4

6

8

10

Training & Evaluation Epsilon (

)

30%

40%

50%

60%

70%

Clean Accuracy (%)

61.0%

57.0%

50.0%

43.0%

41.0%

41.0%

60.0%

54.0%

45.0%

36.0%

31.0%

28.0%

LORE

FARE

1

2

4

6

8

10

Training & Evaluation Epsilon (

)

10%

15%

20%

25%

30%

35%

40%

45%

Robust Accuracy (%)

39.0%

30.0%

22.0%

16.0%

12.0%

9.0%

36.0%

27.0%

18.0%

12.0%

9.0%

7.0%

LORE

FARE

Robustness is measured using the first two AutoAttack methods [Croce and Hein, 2020]: APGD with cross-entropy loss and APGD with targeted DLR loss, each run for 100 iterations under an ℓ ∞ constraint. As shown in Table 11, the base CLIP model has negligible adversarial robustness, whereas LORE consistently outperforms FARE across all settings, improving adversarial accuracy without sacrificing clean performance.

In-domain Image Classification. We further assess LORE on in-domain tasks across various vision architectures. Table 1 reports clean and adversarial accuracy on ImageNet-100 for several CLIP-style backbones using APGD. We also fine-tune the DINOv2 visual encoder with a fixed classification head on the full ImageNet dataset. Results in Table 2 show that LORE enhances robustness not only for CLIP-style models but also for other foundation models such as DINOv2 [Oquab et al., 2024], demonstrating its broad applicability across visual encoder architectures 4 .

Robustness at High Adversarial Intensity. A key challenge in improving robustness against large adversarial perturbations is the instability introduced by high training perturbation budgets ( ε ). Training with large ε values often causes a sharp drop in nominal performance and a loss of alignment with the original model's semantics, making naive adversarial fine-tuning impractical in highε regimes. As illustrated in Fig. 1a and further supported by higher evaluation ε in Fig. 5, this degradation is particularly evident in the early stages of training. Appendix D provides further insight, showing that adversarial fine-tuning using the loss in Eq. (2) rapidly disrupts the proximity between clean inputs and their reference embeddings. This degradation intensifies with increasing ε , resulting in a more pronounced collapse in clean data accuracy. In contrast, Fig. 6 compares LORE and FARE, showing that LORE explicitly enforces constraint satisfaction over time, thereby preserving proximity between clean embeddings and their pre-trained references, while FARE exhibits a progressive increase in deviation during training. This constraint-guided optimization plays a critical role in mitigating early instability and enables LORE to achieve robust and stable learning even under high adversarial intensity.

Distance

0.0

0.1

0.2

0.3

0.4

0.5

Training Step

20

25

30

35

40

45

50

Distance

0.0

0.1

0.2

0.3

0.4

0.5

Training Step

20

25

30

35

40

45

6.4 Analysis: Additional Discussions and Ablation Studies

For further analysis, we present additional discussions and ablations through the following Q&As.

Q: Why use a sample-specific tolerance margin m ( x ) instead of a fixed one?

A: The core of the LORE framework relies on a samplespecific tolerance margin, m ( x ) = ρ ∥ ϕ θ 0 ( x ) ∥ 2 , which dynamically adjusts the allowable perturbation based on the feature magnitude of each input sample.

Table 4 compares LORE with the adaptive margin (LORE m ( x ) , ρ = 0 . 1 ) against LORE with several fixed margins ( ρ = 0 . 6 , 0 . 8 , and 1 . 5 ) as well as the FARE baseline. Experiments were conducted using ViT-B/32 CLIP models fine-tuned for 5 epochs on ImageNet-100.

The results show that fixed margins are either overly restrictive , with ρ = 0 . 6 severely degrading ε = 4 robustness, or overly loose , with ρ = 1 . 5 leading to performance similar to FARE. In contrast, the adaptive margin LORE m ( x ) consistently achieves the best robust accuracy, validating our design choice.

Further discussion and results are in Appendix J.6.

Table 4: Ablation on Adaptive Margin m ( x ) . Adversarial Accuracy is evaluated at the same ε as training.

Q: Can LORE's proximity principle be extended to supervised adversarial fine-tuning?

A: Wefound that LORE's core principle of using proximity constraints to maintain clean performance during adversarial fine-tuning, is transferable and highly effective in the supervised setting. This demonstrates the method's broader applicability beyond its original unsupervised design.

We applied LORE's embedding-space proximity regularization ( ℓ 2 ) to TeCoA [Mao et al., 2023], creating the combined method TeCoA + ℓ 2 . We also explored a variant constraining the KL divergence of the model's output distribution (TeCoA + KL).

Table 5 presents the results, showing that TeCoA + ℓ 2 consistently achieves the highest clean accuracy, with massive gains at high adversarial training budgets. Experiments were performed using ViT-B/32 CLIP models finetuned for 5 epochs on ImageNet-100. This validates that proximity-based constraints, implemented via LORE's Lagrangian framework, are a generally effective mechanism for stabilizing and improving both clean and robust performance in diverse adversarial fine-tuning scenarios.

Further discussion and results are in Appendix J.7.

7 Conclusion, Limitations and Future work

Summary. We proposed LORE, an unsupervised adversarial fine-tuning framework that enhances the robustness of visual encoders while preserving nominal performance without relying on heuristic loss weighting. Extensive experiments show that LORE consistently outperforms FARE, particularly under stronger attacks, achieving superior robustness with better clean data accuracy. LORE's robust visual encoders improve reliability in critical applications, fostering trust in AI. Maintaining high, clean data accuracy ensures effective performance in standard operational environments.

Limitations and future work. While LORE is effective, it has several limitations that suggest directions for future work. (1) A deeper theoretical analysis of the trade-offs in unsupervised adversarial finetuning is needed; our constrained framework provides a useful starting point. (2) The effectiveness and performance ceiling of LORE, which relies on the pretrained model as a fixed anchor ( ϕ θ 0 ), are inherently limited by the quality and fidelity of that frozen reference model. (3) Our use of a neural network to model Lagrange multipliers is heuristic; better parameterizations could improve efficiency and reduce duality gaps. (4) While we adopt Lagrangian duality with manageable gaps, alternative constrained optimization techniques may offer stronger guarantees. Future work could also explore supervised LORE variants.

Table 5: Extension to Supervised FineTuning. Adversarial Accuracy is evaluated at the same ε as training.

References

Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: a query-efficient black-box adversarial attack via random search, 2020. URL https: //arxiv.org/abs/1912.00049 .

Jiawang Bai, Kuofeng Gao, Shaobo Min, Shu-Tao Xia, Zhifeng Li, and Wei Liu. BadCLIP: TriggerAware Prompt Learning for Backdoor Attacks on CLIP, March 2024. URL http://arxiv.org/ abs/2311.16194 . arXiv:2311.16194 [cs].

P. Bertsekas. Nonlinear programming. Journal of the Operational Research Society , 48(3): 334-334, Mar 1997. ISSN 1476-9360. doi: 10.1057/palgrave.jors.2600425. URL https: //doi.org/10.1057/palgrave.jors.2600425 .

Luiz F. O. Chamon and Alejandro Ribeiro. Probably approximately correct constrained learning, 2021. URL https://arxiv.org/abs/2006.05487 .

Luiz F. O. Chamon, Santiago Paternain, Miguel Calvo-Fullana, and Alejandro Ribeiro. The empirical duality gap of constrained statistical learning. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 8374-8378, 2020. URL https: //api.semanticscholar.org/CorpusID:211096932 .

Marcos V. Conde and Kerem Turgutlu. CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classification. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages 3951-3955, June 2021. doi: 10.1109/CVPRW53098.2021.00444. URL http://arxiv.org/abs/2204.14244 . arXiv:2204.14244 [cs].

Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 2206-2216. PMLR, 13-18 Jul 2020. URL https://proceedings. mlr.press/v119/croce20b.html .

Francesco Croce, Christian Schlarmann, Naman Deep Singh, and Matthias Hein. Adversarially robust clip models can induce better (robust) perceptual metrics. In SaTML , 2025.

Edgar Dobriban, Hamed Hassani, David Hong, and Alexander Robey. Provable tradeoffs in adversarially robust classification, 2022. URL https://arxiv.org/abs/2006.05161 .

Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2015. URL https://arxiv.org/abs/1412.6572 .

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations, 2019. URL https://arxiv.org/abs/1903.12261 .

Adel Javanmard, Mahdi Soltanolkotabi, and Hamed Hassani. Precise tradeoffs in adversarial training for linear regression, 2020. URL https://arxiv.org/abs/2002.10477 .

Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial self-supervised contrastive learning, 2020. URL https://arxiv.org/abs/2006.07589 .

Antoni Kowalczuk, Jan Dubi´ nski, Atiyeh Ashari Ghomi, Yi Sui, George Stein, Jiapeng Wu, Jesse C. Cresswell, Franziska Boenisch, and Adam Dziedzic. Benchmarking robust self-supervised learning across diverse downstream tasks, 2024. URL https://arxiv.org/abs/2407.12588 .

Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale, 2017. URL https://arxiv.org/abs/1611.01236 .

Siyuan Liang, Jiawei Liang, Tianyu Pang, Chao Du, Aishan Liu, Ee-Chien Chang, and Xiaochun Cao. Revisiting backdoor attacks against large vision-language models, 2024. URL https: //arxiv.org/abs/2406.18844 .

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, 2019. URL https://arxiv.org/ abs/1706.06083 .

Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models, 2023. URL https://arxiv.org/abs/2212.070 16 .

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2024. URL https://arxiv.org/abs/2304.07193 .

Tianyu Pang, Kun Xu, Yinpeng Dong, Chao Du, Ning Chen, and Jun Zhu. Rethinking softmax crossentropy loss for adversarial robustness, 2020. URL https://arxiv.org/abs/1905.10626 .

Qi Qian and Juhua Hu. Online zero-shot classification with clip, 2024. URL https://arxiv.org/ abs/2408.13320 .

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021. URL http://arxiv.org/abs/2103.00020 . arXiv:2103.00020 [cs].

Alexander Robey, Luiz F. O. Chamon, George J. Pappas, Hamed Hassani, and Alejandro Ribeiro. Adversarial robustness with semi-infinite constrained learning, 2021. URL https://arxiv.or g/abs/2110.15767 .

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding, 2022. URL https://arxiv.org/abs/22 05.11487 .

Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. ICML , 2024.

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks, 2014. URL https://arxiv.org/ab s/1312.6199 .

Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy, 2019. URL https://arxiv.org/abs/1805.12152 .

Sibo Wang, Jie Zhang, Zheng Yuan, and Shiguang Shan. Pre-trained model guided fine-tuning for zero-shot adversarial robustness. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 24502-24511, 2024. doi: 10.1109/CVPR52733.2024.02313.

Jiancong Xiao, Ruoyu Sun, Qi Long, and Weijie J. Su. Bridging the gap: Rademacher complexity in robust and standard generalization, 2024. URL https://arxiv.org/abs/2406.05372 .

Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model, 2022. URL https://arxiv.org/abs/2112.14757 .

Lu Yu, Haiyang Zhang, and Changsheng Xu. Text-guided attention is all you need for zero-shot robustness in vision-language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems , volume 37, pages 96424-96448. Curran Associates, Inc., 2024. URL https://proceedings.neurips.cc /paper_files/paper/2024/file/aec2dfc4f5e19acf05c15587c889dbc4-Paper-Confe rence.pdf .

Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Axi Niu, Jiu Feng, Chang D. Yoo, and In So Kweon. Decoupled adversarial contrastive learning for self-supervised adversarial robustness, 2022. URL https://arxiv.org/abs/2207.10899 .

Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 7472-7482. PMLR, 09-15 Jun 2019a. URL https://proceedings.mlr.press/v97/zhang19p.html .

Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy, 2019b. URL https: //arxiv.org/abs/1901.08573 .

Tingwei Zhang, Rishi Jha, Eugene Bagdasaryan, and Vitaly Shmatikov. Adversarial illusions in multi-modal embeddings, 2024. URL https://arxiv.org/abs/2308.11804 .

Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. RegionCLIP: Region-based Language-Image Pretraining, December 2021. URL http://arxiv.org/abs/2112.09106 . arXiv:2112.09106 [cs].

Wanqi Zhou, Shuanghao Bai, Danilo P. Mandic, Qibin Zhao, and Badong Chen. Revisiting the adversarial robustness of vision language models: a multimodal perspective, 2024. URL https: //arxiv.org/abs/2404.19287 .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately reflect the paper's contributions and scope. LORE, a novel unsupervised adversarial fine-tuning framework, enhances robustness while maintaining model fidelity, as supported by the experimental results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: Our approach entails certain limitations. The limitations of the work are discussed in the conclusion section 7 of the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper provides the full set of assumptions and complete proofs for its theoretical results, specifically in Section 6, where the Lagrangian Multiplier Optimization is derived, and in the Appendix, which contains further details and proofs for the propositions. While the paper includes theoretical results, it is not a heavily theoretical paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper provides sufficient details to reproduce the main experimental results. Section 6 includes comprehensive information on the datasets used, implementation specifics, and evaluation metrics. Furthermore, the authors have stated that the code will be made publicly available, ensuring full reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The authors state in the paper that the code will be made publicly available. This, combined with the detailed descriptions of the datasets, experimental setup, and evaluation metrics, ensures that the main experimental results can be faithfully reproduced.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/pu blic/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper provides sufficient training and test details in Section 6 and the Appendix. It describes the datasets used, data splits, hyperparameter settings, the method used for hyperparameter selection, and the optimizers used for training.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: While the paper does not report error bars, the statistical significance of the experiments is ensured through evaluation across multiple diverse datasets and settings, using standard benchmark datasets. Furthermore, retrainining the model with multiple random seeds for each experiment is computationally expensive

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In Section 6, we provide comprehensive details on the computational resources used and the duration of the training process.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics. The paper focuses on improving the robustness of visual encoders against adversarial perturbations, which is a relevant concern for responsible AI development.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses the potential positive and negative societal impacts of the work performed in the conclusion section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not involve the release of data or models that have a high risk of misuse, such as pretrained language models, image generators, or scraped datasets. The research focuses on improving the robustness of visual encoders against adversarial perturbations.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper properly credits the creators or original owners of assets used. The datasets and models used are standard benchmarks in the field, and appropriate citations are provided.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper introduces a novel unsupervised adversarial fine-tuning framework (LORE), but it does not introduce new data, code, or models that require separate documentation. The core contribution is the algorithm itself.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing experiments or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve research with human subjects. The research focuses on improving the robustness of visual encoders against adversarial perturbations.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The paper does not describe the use of LLMs as part of its core methodology, and LLMs were not involved in the design, implementation, or evaluation of this framework.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Part I

Appendix

Table of Contents

A Proof of Theorem 6.2

We define the following quantities:

And

Since H ρ ⊂ H , we have R ρ ≥ R . The sub-optimality gap is bounded by:

Proof. Let δ ∗ 1 = arg max δ ∈ ∆ d ( ϕ 1 ( x + δ ) , ϕ 0 ( x )) denote the perturbation that maximizes the adversarial loss for model ϕ 1 relative to the reference model ϕ 0 . For any two models ϕ 1 , ϕ 2 , we aim to bound the difference in their adversarial losses, considering assumption 6.1:

Alternatively, we set the Lipschitz constant to 1 for brevity; however, it is worth noting that in general, this constant appears explicitly in the bound and should be carried through the analysis. Note that without loss of generality, we have assumed that max δ ∈ ∆ d ( ϕ 1 ( x + δ ) , ϕ 0 ( x )) -max δ ∈ ∆ d ( ϕ 2 ( x + δ ) , ϕ 0 ( x )) ≥ 0 . If that is not the case, then we have to replace δ ∗ 1 with δ ∗ 2 = arg max δ ∈ ∆ d ( ϕ 2 ( x + δ ) , ϕ 0 ( x )) throughout the proof. We now decompose this difference using the triangle inequality:

Assuming that the function ϕ 1 -ϕ 2 is Lipschitz continuous with constant L , we obtain:

Assuming δ ∗ 1 ∈ R k with ∥ δ ∗ 1 ∥ 2 ≤ ε √ k -which corresponds to assuming that d is Lipschitz with respect to the Euclidean norm, and that ε is bounded in the ℓ ∞ norm-we have:

We now extend this pointwise bound to the expected adversarial loss:

where ∥ ϕ 1 -ϕ 2 ∥ denotes the expected difference in their outputs over the input distribution.

Similarly, setting ε = 0 , the adversarial loss reduces to the clean loss, yielding a corresponding bound:

Finally, applying this to the case ϕ 1 = ϕ ∗ ρ and ϕ 2 = ϕ ∗ , we obtain the desired bound on the deviation in adversarial loss due to constraining the hypothesis space.

B The Parametrization Gap

In this paper, we solved our constrained optimization problem, by optiming its dual problem, and using neural networks for the lagrangian multipliers. particularly solving:

Instead of the original constrained optimization problem. in this section we are interested in deriving some bounds on the duality gap, following the proof from Robey et al. [2021].

Proof.

Assumption B.1. For all g ∈ conv ( H ) , there exists ˜ θ ∈ Θ such that

where η > 0 is a sufficiently small constant.

First, ignoring the parametrization of λ , and assuming λ can be any function from Λ = { λ : X → R + } , we consider the Lagrangian:

Now consider the original problem:

If the function class H parametrized by θ were convex, this would be a convex program. Since by definition, ϕ 0 ∈ H = { ϕ θ : θ ∈ Θ } , there exists θ ∈ Θ such that d ( ϕ θ ( x ) , ϕ 0 ( x )) = 0 < ρm ( x ) for all x ∈ X . Thus, Slater's condition is satisfied.

Therefore, if H were convex, we would have strong duality, i.e., p ∗ = d ∗ . However, for most typical neural networks, H is non-convex. By weak duality, we always have: d ∗ ≤ p ∗ . To derive a lower bound, consider the following problem for some positive constant η > 0 :

This is now a convex program. Since ϕ 0 itself satisfies d ( ϕ 0 ( x ) , ϕ 0 ( x )) = 0 < ρm ( x ) -η , Slater's condition is again satisfied. Hence, strong duality holds, and the Lagrangian becomes:

Thus,

Assuming the infimum and supremum are attained at g ∗ and ˜ λ ∗ , we have:

Using the relation between ˜ L and L , we obtain:

Since g ∗ is strictly feasible for the relaxed constraint, the complementary slackness condition implies ˜ λ ∗ ( x ) = 0 for almost every x . Therefore:

Using the suboptimality bound from Lemma A, we have: √

Since g ∗ is feasible in the relaxed problem with the stricter constraint d ( g ∗ ( x ) , ϕ 0 ( x )) ≤ ρm ( x ) -η, for some ˜ θ ∗ ∈ Θ approximating g ∗ such that ∥ g ∗ -ϕ ˜ θ ∗ ∥ < η, the function ϕ ˜ θ ∗ is feasible in the original problem, because for almost every x ∈ X

ℓ clean ( ϕ ˜ θ ∗ ( x ) , ϕ 0 ( x )) ≤ ℓ clean ( g ( x ) , ϕ 0 ( x )) + ∥ g -ϕ ˜ θ ∗ ∥ ≤ ρm ( x ) -η + ∥ g -ϕ ˜ θ ∗ ∥ < ρm ( x ) Since p ∗ is the minimum over all feasible θ ∈ Θ , it follows that p ∗ ≤ ℓ adv ( ϕ ˜ θ ∗ , ϕ 0 ) .

Therefore, since θ ∗ is the optimal solution to the original problem: √

Finally, we note that although we have treated the dual space as the full infinite-dimensional set Λ = { λ : X → [0 , ∞ ) } , in this work we have restricted λ to a finite-dimensional family { λ ω } ω ∈ Ω ⊂ Λ that uniformly approximates its elements. Concretely, if for every λ ∈ Λ there exists ω ∈ Ω with ∥ λ -λ ω ∥ L 1 ( D ) ≤ ξ , then replacing

only incurs an arbitrarily small error O ( ξ ) . All weak-duality arguments carry over immediately, and under Slater's condition the resulting strong-duality statement remains valid up to this negligible approximation.

One viewpoint is that limiting the expressivity of λ ω through parametrization effectively relaxes the constraints, as the network cannot fully ensure the constraints are met. As an extreme case, consider when λ ω ( x ) = ω for all x ∈ X . In this case:

Then the (relaxed) Lagrangian becomes

Optimizing first over ϕ (so that E [max δ d ] is fixed) and then taking the supremum over ω ≥ 0 forces

otherwise L ( ϕ, ω ) → + ∞ as ω → + ∞ . In other words, the constantλ relaxation exactly enforces

Thus, by limiting the expressivity of λ , we move from the original per-sample constraint

to the weaker but still meaningful average constraint

C Additional Experimental Details

In this appendix, we provide further experimental details beyond those given in the main text. Experiments were conducted using 8 NVIDIA HGX H100 80GB GPUs.

Training hyperparameters. We report below the training settings used across all experiments. Unless otherwise noted, all models were trained using AdamW with a weight decay of 1 × 10 -4 , a cosine learning rate scheduler, and adversarial training with PGD (10 iterations, step size ε/ 4 ) under an ℓ ∞ constraint. Each λ network used the 2-layer linear_mlp architecture, with a hidden dimension of 512, and was optimized via K = 5 inner primal updates with learning rate 5 × 10 -4 . More experimental details are provided in Table 6. Additional information about all figures and tables in the paper is summarized in Table 7.

D Constraint Satisfaction in LORE

To further understand the behavior of constraint enforcement under varying adversarial budgets, we visualize the distribution of distances between clean embeddings and their corresponding pretrained references throughout training for ε = 1 , 2 , and 4 in Figure 7. As we can observe, larger perturbation strengths lead to greater deviation from the pre-trained reference in the early stages of training. This early-stage divergence results in a more pronounced initial drop in the model's nominal performance. However, LORE is able to effectively regulate this deviation through its constraint-aware mechanism, gradually aligning the clean embeddings back within the ρ threshold. This demonstrates the robustness and practicality of LORE in preserving clean performance even under severe adversarial training regimes.

In contrast, due to the lack of such constraint regulation in FARE, the distance between clean embeddings and pre-trained references cannot be reliably controlled. As a result, FARE experiences a catastrophic initial drop in nominal accuracy, particularly under larger perturbation budgets. This failure to maintain embedding fidelity further underscores the importance of the dual network in LORE for stabilizing the training process and preserving clean accuracy.

To better illustrate this behavior, all subfigures in Figure 7 show the distance distributions beginning from the 20th training iteration onward. These comparisons clearly highlight the contrast between LORE's effective enforcement of the proximity constraint and FARE's limited capability to manage deviation across increasing adversarial strengths.

E Generalization Gap in Adversarial Training

Theorem E.1 (Generalization Gap in Adversarial Training) . It is well known that the generalization gap for a given loss function is upper bounded by complexity measures, giving rise to theoretical justifications of the bias-variance trade-off. Assuming bounded norm embeddings, i.e., ∥ ϕ ( x ) ∥ 2 ≤ K for all x, ϕ , we can see that the uniform loss bound B satisfies:

Therefore, with probability at least 1 -2 δ ,

where L H = { ℓ ( ϕ, · ) | ϕ ∈ H} is the loss class induced by hypothesis class H , R n ( L H ) is the empirical Rademacher complexity of L H , and sup ϕ,x | ℓ ( ϕ, x ) | ≤ B .

In adversarial training, the loss class L H becomes extremely complex due to the inner max δ operation, leading to large Rademacher complexity R n ( L H ) . This explains why adversarial training requires significantly more samples for generalization compared to standard training.

When we restrict to simpler hypothesis classes H ρ ⊂ H org (through techniques like Lipschitz constraints or norm bounds), the Rademacher complexity decreases, potentially improving generalization.

Distance

0.0

0.1

0.2

0.3

0.4

0.5

Training Step

20

25

30

35

40

45

50

(a) LORE,

ε

= 1

Distance

0.0

0.1

0.2

0.3

0.4

0.5

Training Step

20

25

30

35

40

45

50

(b) LORE,

ε

= 2

Distance

0.0

0.1

0.2

0.3

0.4

0.5

Training Step

20

25

30

35

40

45

50

(c) LORE,

ε

= 4

Distance

0.0

0.1

0.2

0.3

0.4

0.5

Training Step

20

25

30

35

40

45

(d) FARE,

ε

= 1

Distance

0.0

0.1

0.2

0.3

0.4

0.5

Training Step

20

25

30

35

40

45

ε

= 2

Distance

0.0

0.1

0.2

0.3

0.4

0.5

Training Step

20

25

30

35

40

45

(f) FARE,

ε

= 4

However, the bounds we derived are notoriously crude; they fail to capture important phenomena like double descent and often dramatically overestimate the actual generalization gap in practice.

F Deviation Between Cosine Similarities

Assumption F.1. For each input x , let

Proposition F.2. Under the above assumption, for any nonzero v ∈ R n ,

implies a uniform bound on the change in cosine similarity to any fixed vector v ∈ R n .

Proof. Write and suppose

We show that enforcing

so

Now decompose

so by the triangle inequality,

Since ∣ ∣ ∥ ˆ u ∥ - ∥ u ∥ ∣ ∣ ≤ ∥ u -ˆ u ∥ and ∥ ˆ u ∥ ≤ ∥ u ∥ + ∥ u -ˆ u ∥ , one shows easily

Hence as claimed.

Remark. In vision-language models one may take v = ψ ( t ) , the text embedding of prompt t , so the same bound guarantees | S C ( ϕ org ( x ) , ψ ( t )) -S C ( ϕ θ ( x ) , ψ ( t )) | ≤ 2 √ ρ .)

G Impact of Dual Network on Model Performance

G.1 Comparison of Alternative Architectures for the Dual Function

In Figure 8, we compare the clean and robust accuracy achieved by two different architectures used for the dual function: a simple scalar value and a network-based (sample-dependent) function, as adopted in the current LORE setting. While both configurations perform comparably in terms of clean accuracy, the network-based dual function generalizes substantially better on adversarial examples, leading to consistently higher robust accuracy throughout training. This highlights the importance of a flexible, sample-adaptive mechanism in effectively enforcing robustness constraints during adversarial fine-tuning.

(a)

(b)

Clean Accuracy

Robust Accuracy

ScalarNetwork

ScalarNetwork

60

122

15

50

10

40

30

20

Iteration

step

200

400

600

800

1k

1.2k

1.4k

200

400

600

800

1k

1.2k

1.4k

G.2 Effect of the Dual Network λ ω ( x ) on Clean Accuracy

Figure 9 illustrates the impact of the dual network λ ω ( x ) on model performance. As shown, at the initial steps, higher values of λ ω ( x ) help maintain the model's nominal performance, ensuring it performs well on clean data. In contrast, for FARE, due to the absence of such a proximity constraint during the early iterations, the model, lacking robustness, passes through suboptimal states, leading to a significant drop in nominal performance.

To further support this observation, we present comprehensive results in Figure 10 and Figure 11, showcasing the behavior of the dual network and its impact across different architectures. In Figure 10, experiments on DINOv2 models (base and small) demonstrate that LORE consistently achieves higher clean accuracy compared to FARE, especially in the early stages of training, while adaptively modulating λ ω ( x ) to control constraint satisfaction. Similarly, Figure 11 reports the performance of ViT-B/16 and ConvNeXt-B models, confirming the effectiveness and generality of the dual network across various architectures and perturbation strengths. These results highlight that LORE's constraint-aware mechanism is stable, avoiding the sharp degradation commonly observed in FARE adversarial fine-tuning.

(a)

(b)

(c)

CleanAccuracyoverIterations

RobustAccuracy overIterations

FARE  LORE

FARE  LORE

60

50

40

10

30

20

10

Iteration

Iteration

500

1k

1.5k

2k

500

1k

1.5k

2k

Average of the Dual Network Output

1.5

0.5

SSMS

Iteration

100

200

300

H Revisiting Embedding Models

CLIP [Radford et al., 2021]. A major part of our experiments builds upon CLIP, which consists of an image encoder ϕ and a text encoder ψ that map images and text descriptions into a shared embedding space. For zero-shot classification, textual descriptions are typically formatted as "This is a photo of a [CLS]' , where [CLS] represents class labels. The probability of assigning an image x to a class ˆ y is computed via a softmax over cosine similarities:

where τ is a temperature parameter, and K denotes the number of classes.

DINOv2 [Oquab et al., 2024]. In addition to CLIP, we incorporate DINOv2, a powerful selfsupervised visual transformer-based encoder, into our exploration of embedding models. While

(a) DINOv2-base, Clean accuarcay

(

ε

= 16)

(b) DINOv2-base, Dual Network Output

(

ε

= 16)

(c) DINOv2-base, Clean accuarcay

(

ε

= 8)

(d) DINOv2-base, Dual Network Output

(

ε

= 8)

(e) DINOv2-small, Clean accuarcay

(

ε

= 16)

(f) DINOv2-small, Dual Network Output

(

ε

= 16)

(g) DINOv2-small, Clean accuarcay

(

ε

= 8)

(h) DINOv2-small, Dual Network Output

(

ε

= 8)

CleanAccuracy

FARE

LORE

80

14

12

60

10

8

40

6

20

寸

2

D

Iteration

Q

LMM

teration

1k

2k

3k

4k

1k

2k

3k

4k

5k

CleanAccuracy

FARE

LORE

80

14

12

60

10

8

40

6

4

20

Iteration

2

Iterafibn

0

1k

2k

3k

4k

1k

2k

3k

4k

5k

CleanAccuracy

Average of the Dual NetworkOutput

FARE

LORE

80

10

60

00

40

6

4

20

2

Iteration

Iteration

0

0

0

1k

2k

3k

4k

0

100

200

300

400

CleanAccuracy

Average of the Dual NetworkOutput

LOREFARE

80

14

12

60

10

8

40

6

20

4

2

Iteration

Iteration

0

1k

2k

3k

4k

1k

2k

3k

4k

5k

CLIP provides a joint image-text embedding space, DINOv2 focuses solely on visual representation learning. This complementary perspective allows us to compare and leverage both multimodal and unimodal embedding paradigms.

(a) ViT-B/16, Clean accuarcay

(

ε

= 8)

(b) ViT-B/16, Dual Network Output

(

ε

= 8)

(c) ConvNeXt-B, Clean accuarcay

(

ε

= 4)

(d) ConvNeXt-B, Dual Network Output

(

ε

= 4)

CleanAccuracy

LOREFARE

14

60

12

10

40

8

6

20

4

°2

Iteration

Iteration

0

0

100

200

300

400

500

600

700

200

400

600

800

1k

1.2k

1.4k

CleanAccuracy

Averageof theDual Network Output

LOREFARE

80

14

70

12

60

10

50

8

40

6

30

4

20

2

Iteration

Iteration

0

0

200

400

600

800

1k

1.2k

200

400

600

800

1k

1.2k

DINOv2 learns visual features by minimizing a cross-view prediction loss between student and teacher networks. Given N image views, the loss is computed as:

where q ic and p ic are the teacher and student probabilities for class c and view i , respectively, and C is the number of output dimensions.

Our broader work centers around embedding models, with a primary emphasis on CLIP, while also investigating the capabilities and representations of models like DINOv2. In general, we found that the DINOv2 model has much richer image embeddings than CLIP and can achieve much higher robustness over the same perturbation and dataset.

I Impact of K on Model Performance

LORE alternates between K steps of updating the primal encoder and one step of updating the dual network. In this section, we study the effect of the hyperparameter K on final performance. As shown in Table 8, increasing K leads to improved clean accuracy (Acc) and robust accuracy (RAcc), particularly when moving from K = 1 to K = 3 or 5 . This demonstrates that more frequent primal updates between dual updates help stabilize training and improve performance. Based on this observation, we choose K = 5 as the default in our final LORE implementation. For further discussion on how K impacts computational cost and training time, see Appendix K.

J Additional Experimental Results

In this section, we present additional experiments to further validate the robustness and generalization capabilities of our proposed method. These evaluations span multiple settings, including blackbox adversarial attacks (e.g., Square Attack), Gaussian noise corruption, in-domain and zero-shot classification, and out-of-distribution (OOD) robustness. By comparing against the FARE baseline

across diverse conditions and datasets, we demonstrate that LORE consistently achieves superior performance, particularly under challenging threat models and distributional shifts.

J.1 Square Attack Evaluation

In this section, we evaluate the robustness of our fine-tuning approach against a black-box adversarial attack known as the Square Attack [Andriushchenko et al., 2020]. Unlike gradient-based methods, Square Attack operates without access to model gradients and perturbs the input using a queryefficient, score-based strategy. This makes it a strong candidate for evaluating real-world robustness where white-box access is not feasible. We conduct experiments on LORE 4 , which is adversarially fine-tuned on ImageNet, to assess how well the model generalizes to such black-box settings. The results, summarized in Table 9, show that our method consistently outperforms the baseline under this challenging threat model.

J.2 Evaluation Under Gaussian Noise Corruption

To further assess the robustness of our method, we evaluate the performance of LORE 4 and FARE 4 under varying levels of Gaussian noise corruption. We use the ViT-B/32 CLIP model, with both methods fine-tuned on ImageNet. As shown in Figure 13, LORE 4 consistently maintains higher accuracy than FARE 4 across a wide range of noise strengths ( σ ), especially in low to moderate noise settings. The right subplot illustrates the accuracy gap, highlighting LORE 4 's robustness advantage up to σ = 40 , beyond which the performance of both models converges as the corruption becomes extreme. This evaluation further supports the generalization capabilities of our method in the presence of unseen corruptions. Figure 12 provides a visual illustration of how a single image degrades under increasing levels of Gaussian noise.

Noise: 0

Noise: 32

Noise: 64

Noise: 128

0

1

2

4

8

16

32

40

48

56 64

80

96 112128

Gaussian noise strength (

)

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

LORE

4

FARE

4

0

1

2

4

8

16

32

40

48

56 64

80

96 112128

Gaussian noise strength (

)

0.0%

2.0%

4.0%

6.0%

8.0%

J.3 In-domain Image Classification

Table 10 presents a comparison of clean and adversarial accuracy across various CLIP-based vision backbones, all trained with ε = 8 , on the ImageNet-100 dataset under the APGD attack.

J.4 Zero-shot Image Classification

Table 11 presents the results of different settings for zero-shot image classification using the ViT-B/32 CLIP model, highlighting the superiority of our method over the FARE baseline. Additionally, for a more challenging scenario, Table 12 reports model performance under high-intensity adversarial attacks, further demonstrating the resilience of our approach. These tables serve as the complete version of the results summarized in the main paper.

J.5 Out-of-Distribution Robustness

As shown in Figure 14, increasing the training perturbation strength leads to greater degradation in out-of-distribution (OOD) robustness across common corruptions in ImageNet-C. Despite this trend, models trained with LORE consistently exhibit better robustness compared to those trained with the FARE method, highlighting LORE's superior generalization under distributional shifts.

J.6 Ablation on Sample-Specific Margin m ( x )

The choice of a sample-specific tolerance margin, m ( x ) = ρ ∥ ϕ θ 0 ( x ) ∥ 2 , is a crucial design decision in the LORE framework.

Using a sample-invariant tolerance ( ρ fixed ), where the constraint is fixed across all data points ( d ( ϕ θ ( x ) , ϕ θ 0 ( x )) ≤ ρ fixed), offers an alternative but is challenging. It requires extensive, taskdependent and model-dependent hyperparameter tuning that severely limits its practical utility and generalization ability.

Average

speckle noise

spatter

snow

shot noise

saturate

pixelate

motion blur

jpeg compression

impulse noise

glass blur

frost

fog

elastic transform

defocus blur

contrast

brightness

No Shift

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

55%

60%

FARE

LORE

CLIP

50.3%

38.5%

30.5%

59.8%

29.8%

21.6%

(a)

Average

speckle noise

spatter

snow

shot noise

saturate

pixelate

motion blur

jpeg compression

impulse noise

glass blur

frost

fog

elastic transform

defocus blur

contrast

brightness

No Shift

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

55%

60%

FARE

LORE

CLIP

50.3%

30.8%

17.0%

59.8%

23.9%

17.4%

(b)

By contrast, the sample-specific margin provides an adaptive, scale-invariant tolerance tied to the pretrained embedding's norm. This design choice reduces manual tuning, preserves semantic structure by relating the constraint geometrically to the cosine similarity of embeddings, and ultimately offers greater stability.

Experimental Comparison. To validate this choice, we conducted experiments using ViT-B/32 CLIP models fine-tuned for 5 epochs on ImageNet-100. We compare the adaptive margin (LORE m ( x ) ) against several empirically chosen fixed tolerances ( ρ = 0 . 6 , 0 . 8 , 1 . 5 ). The superscript ( ε ) denotes the training perturbation budget.

As shown in Table 13, fixed tolerances of 0 . 6 and 1 . 5 lead to suboptimal performance, demonstrating that a fixed margin is either overly restrictive (degrading robustness) or overly loose (causing behavior similar to FARE). The sample-specific margin (LORE m ( x ) ) consistently achieves the best overall Average Robust Accuracy, validating its superior stability and efficacy across different training settings.

J.7 LORE in Supervised Adversarial Fine-Tuning

Given an image-label pair ( x, y ) , TeCoA [Mao et al., 2023] optimizes a contrastive loss to align adversarial image embeddings with text embeddings. The loss function is defined as:

where z = ϕ θ ( x + δ ) is the adversarial image embedding and ψ ( t i ) is the text embedding. Here, ψ ( t i ) corresponds to the text embedding of the i -th class, where each class label is inserted into a fixed natural language template (e.g., "This is a photo of {class}").

We incorporated our proximity-based regularizer into TeCoA, forming a combined method (TeCoA + l 2 ), where l 2 denotes the ℓ 2 -based embedding proximity constraint:

Distribution-Level Constraint. We also explored a probability-distribution variant of LORE for the supervised setting, constraining the KL divergence between the fine-tuned model's output distributions and those of the pretrained model. This approach requires only class text embeddings, not sample labels. Applied to TeCoA, it yields a second regularized variant: TeCoA + KL.

For classification using probability distributions induced by the contrastive model, we formulate the constrained optimization problem:

Building on this formulation, LORE's final objective function can be computed as before. The framework adjusts the proximity constraint by model confidence: low entropy (high confidence) tightens the constraint, while high entropy relaxes it for greater adaptation flexibility.

Results. The experimental setting involved fine-tuning ViT-B/32 CLIP models for 15 epochs on ImageNet-100 using a learning rate of 1 e -5 . The proximity constraint hyperparameter used ρ = 0 . 1 . The results are shown in Table 14.

TeCoA consistently benefits from both LORE-based ( l 2 ) and KL regularization, achieving improved robustness while maintaining clean performance. Notably, for high adversarial training budgets (e.g., Train ε = 8 ), the TeCoA + l 2 method yields a massive increase in clean accuracy compared to the baseline TeCoA. These results demonstrate that proximity-based constraints-whether applied in embedding space ( l 2 ) or probability space (KL)-can effectively stabilize adversarial fine-tuning in supervised frameworks as well.

J.8 Validation on Downstream Tasks (Segmentation)

To confirm LORE's generalizability beyond classification, we evaluated its effectiveness on semantic segmentation, a critical VLM downstream task. We use the open-source benchmark from Kowalczuk et al. [2024] to have a fair comparison with our baselines.

We assessed the robustness of the DINOv2 ViT-S/14 model on the ADE20k segmentation dataset. The results in Table 15 show that the LORE-hardened model improves robustness against adversarial attacks in the embedding space while maintaining a favorable clean accuracy trade-off.

K Computation and Efficiency Analysis

In this section, we analyze the computational aspects of LORE in terms of training time, efficiency, and convergence behavior. While LORE introduces an additional dual network and constraint enforcement mechanism, we find that its cost remains practical and comparable to FARE baselines.

K.1 Convergence Efficiency.

To compare the training efficiency of FARE and LORE, we measure the total training time (in minutes) required to reach specific robust accuracy (RAcc) thresholds on the validation set. Table 16 reports this comparison across various model backbones, including ViT-B/16, ViT-B/32, ConvNeXtB, ViT-S/14, and ViT-B/14. LORE often reaches target RAcc levels in fewer training minutes than FARE, highlighting its superior optimization efficiency and stability.

K.2 Impact of λ ω architecture on Training Time.

The results in Table 17 show that LORE's current dual network design is not only significantly more efficient than using a separate pretrained CLIP model as a dual network, but also achieves comparable runtime to simpler parameterizations (scalar or linear forms). This demonstrates that LORE achieves computational efficiency without sacrificing expressive capacity.

K.3 Impact of K on Training Time.

As described in Section 5, LORE alternates between K primal updates and one dual update per batch. While Section I analyzes the impact of K on final performance, here in Fig. 15, we empirically examine how varying K affects training time.

700

ViT-B/32

ConvNeXt-B

600

500

Time

400

Training

300

200

100

0

+

2

3

4

5

10

K

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Lagrangian-Optimized Robust Embeddings (LORE), a novel unsupervised adversarial fine-tuning framework, which enhances the robustness of visual encoders while preserving nominal performance without relying on heuristic loss weighting. Extensive experiments show that LORE significantly improves zero-shot adversarial robustness with minimal degradation in clean data accuracy and outperforms FARE consistently. However, the paper still needs further refinement and revision. Further experimental validation and analysis are recommended to be added.\n\nSTRENGTHS AND WEAKNESSES:\nPros: Extensive experiments and Interesting Topic.\n\nCons:  Further validation and analysis are recommended to be added.\n\nQUESTIONS:\nSome problems are:\n1、In this paper, FARE is considered as a baseline for evaluating the proposed approach. Hence, it is important to highlight the contributions of the proposed method LORE compared with FARE in the introduction.\n\n2、In Section 2 Related Work, there are recent studies that highlight the vulnerability of VLMs to adversarial attacks. However, the corresponding description of the relevant references is just too limited to fit the regular structure of a paper.\n\n3、For Figure 1(a), the meaning of varying `\\rho` is not mentioned in the paper. Further explanation is necessary to be added to the manuscript.\n\n4、To empirically validate the approach, the majority of our experiments were conducted and all models were trained for 2 epochs on ImageNet or 5 epochs on ImageNet-100. Does this signify that the structure of the model is very simple and therefore the training process is comparatively fast?\n\n5、A comprehensive evaluation of clean and adversarial performance is conducted across various image classification datasets in Table 3. But there are experimental results of ‘0’ and no experimental results for `\\epsilon`=8. Further experimental validations are necessary to be added to the manuscript.\n\nLIMITATIONS:\nMentioned in the paper", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a Lagrangian-based optimization method to train a robust visual encoder, i.e., CLIP and DINOv2. Compared to the previous method, their method is more stable during the training process and achieves a better tradeoff between clean and adversarial accuracy, which is very important for adversarial robustness. Their experiments show that their method achieves higher adversarial accuracy on the image classification problem while the clean accuracy does not drop significantly.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n* In Section 4, they show that simply training the CLIP using an adversarial loss $L_{FARE}$ and a fixed-weighted regularizer could not solve the problem, which provides a good motivation for their method.\n* The paper is well-written with clear logic and results.\n* Their method is effective, achieving better robustness under adversarial attack while having minimal degradation on the clean accuracy.\n* They validate the efficiency of their method on various dataset sizes and adversarial attack budgets, indicating the robustness of their method over general settings.\n\nCons:\n* In the experiment part, the authors use the image classification task as the measurement to verify the robustness. Since the CLIP representation has been used in various applications such as detection and VLM, could the authors validate the robustness on those tasks?\n* There are many other related methods proposed to improve the robustness of visual encoders [1,2], so I suggest you compare your method with more baselines rather than only comparing with FARE.\n* The novelty of this work is limited. There has already been the unconstrained version of the adversarial optimization problem. And the difference between the proposed loss function Eq. (6) and the regularized loss function Eq. (3) only lies in the trainable weight $\\lambda(x)$. \n* In Line 130, the embedding space is denoted as $\\mathcal{Y}$, while $y$ is used to represent the label of a data point $x$. It would be better to distinguish these two.\n\n[1]: Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models.\n[2]: Improving robustness of CLIP by adversarial training enhanced by brain activity.\n\nQUESTIONS:\nIn your method, you use a sample-specific tolerance margin, which is related to the norm of the original feature embedding. What would the results be if we used a sample-invariant tolerance?\n\nLIMITATIONS:\nYes, they discuss their limitations in the last section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel unsupervised adversarial fine-tuning framework dubbed Lagrangian-Optimised Robust Embeddings (LORE). The authors aim to address the limitations of current unsupervised adversarial fine-tuning approaches: instability during early stages of fine-tuning and suboptimal trade-off between clear and adversarial performance. LORE maintains the performance on clean data during the fine-tuning process by enforcing embedding proximity constraints.  Moreover, LORE enables principled control over the clean-adversarial performance trade-off. The paper demonstrates the effectiveness of this approach on CLIP and DinoV2 models, compared with the SOTA FARE approach.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\nS1: LORE effectively addresses a crucial problem of instability during the early stages of adversarial fine-tuning in unsupervised adversarial fine-tuning. I find the significance of this result to be high, as it enables adversarial fine-tuning for a limited number of updates without collapsing the model’s performance. This makes the adversarial fine-tuning process more practical and user-friendly.\n\nS2: LORE effectively addresses an important problem of suboptimal trade-off between clear and adversarial performance, pushing the empirical Pareto front.\n\nS3: Strong experimental section. (However, see Weaknesses for ideas of additional improvements)\n\n### Weaknesses\n\nW1: Optimal trade-off claim.  The paper uses strong language to describe the performance of LORE. For example, “LORE yields a strictly better Pareto front, achieving optimal trade-offs between robustness and clean accuracy” (Figure 1). At the same time, the paper notes that: \"(1) A deeper theoretical analysis of the trade-offs in unsupervised adversarial finetuning is needed\" (L294-295).  While the empirical results indeed prove that the Pareto front is improved over the baseline, calling it “optimal” might be an overstatement. According to my understanding, there is no proof that LORE finds the global optimum of the robustness vs. accuracy trade-off.\n\nW2: Missing broader related work. I believe that the Related Works section misses some earlier attempts at ensuring adversarial robustness in the unsupervised fine-tuning setting. For example:\n\n[1] Kim et al., Adversarial Self-Supervised Contrastive Learning, NeurIPS 2020\n\n[2] Zhang et al., Adversarial Contrastive Learning for Unsupervised Representation Learning, NeurIPS 2021\n\n[3] Chiang et al., Robust Contrastive Learning for Unsupervised Visual Representation Learning, CVPR 2022\n\n\nW3: Limited downstream tasks. Dinov2 can produce an embedding usable in a variety of downstream tasks, e.g segmentation. Yet, the evaluation in the paper is limited to classification. I believe it would be valuable to asses the effectivness of robust models (e.g. DinoV2) obtained with LORE on more diverse downstream tasks, e.g. similarly to [4]. \n\n[4] Kowalczuk et al., Benchmarking Robust Self‑Supervised Learning Across Diverse Downstream Tasks, CVPR 2024 Workshop on Robust Representation Learning\n\n### Comments\nC1: While this is not a weakness per se, LORE targets instability during early stages of adversarial fine-tuning and suboptimal trade-off between clear and adversarial performance in the *unsupervised* adversarial fine-tuning. According to my understanding, this approach is not directly applicable to supervised models. Yet, the supervised setting also suffers from similar problems. This limits the (still high) significance of the method proposed in this paper. The paper's significance would be even greater if a LORE-like approach was applicable to all adversarial fine-tuning scenarios.\n\nQUESTIONS:\nQ1: Can LORE be extended to simultaneously fine-tune both image and text encoders in Visual Language Models like CLIP?\n\nQ2:  \"(2) LORE relies on a high-quality pretrained model, and its performance, especially on perceptual metrics, is sensitive to the fidelity of that model.\" Can you please provide more discussion on that matter? At what point does LORE stop being an effective approach?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations. However, I feel that more discussion on the following would improve the paper: \" (2) LORE relies on a high-quality pretrained model, and its performance, especially on perceptual metrics, is sensitive to the fidelity of that model.\"", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}