Flatness Guided Test-Time Adaptation for Vision-Language Models
Authors: Aodi Li
Liansheng Zhuang
Xiao Long
Houqiang Li
Shafei Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Vision-Language Models
Generalization
Loss landscape
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Test-time adaptation (TTA) of Vision-Language Models (VLMs) has emerged as a technique for tackling distribution shifts during the test time. 
Recent research indicates that the test-time adaptation is intrinsically linked to the model's training history.
However, existing TTA methods, such as Test-time Prompt Tuning, often design adaptation strategies in isolation from the models' training characteristics, which degrade their performance. 
This paper argues that the flatness acquired via sharpness-aware training is an efficient clue for the test-time adaptation of VLMs. 
Built on this insight, this paper proposes a novel Flatness-Guided Adaptation framework (FGA) for VLMs to cohesively unify training and test-time procedures. Its core idea is to leverage the alignment between the training minimum and test loss flat regions to guide the adaptation process.
Specifically, our FGA consists of a prompt-tuning stage and a test-time adaptation stage. In the tuning stage, a Sharpness-Aware Prompt Tuning method is utilized to identify the training flat minimum, offering a geometric clue of flatness for subsequent adaptation. In the test stage, a Sharpness-based Test Sample Selection approach is proposed to ensure the alignment of flat minima between the training and each augmented test sample's loss landscape. 
In comparison to existing TTA methods, our FGA avoids the expensive prompt parameter updates during test time, and substantially reduces the computation overhead.
Extensive experiments on both domain generalization and cross-dataset benchmarks demonstrate that our FGA achieves superior performance over prevalent TTA methods. 
Notably, when employing a ViT-B/16 image encoder, FGA even outperforms TPT+CoOp by an average of 4.88\% across all four ImageNet out-of-domain variants.

Published as a conference paper at ICLR 2026
FLATNESS GUIDED TEST-TIME ADAPTATION FOR LISIONLLNNUAGE MODELS
Aodi Li', Liansheng Zhuang!.200 Xiao Long', Houqiang Li & Shafei Wang?
School of Cyber Science and Technology, University of Science and Technology of China 2 National Engineering Laboratory for Brain-Inspired Intelligence Technology and Applications, University of Science and Technology of China *Pngg Cheng Laboratory, Shenzhen, China aodi li (mma 1. uste. edu. cn, 1szhuang@ustc. cettuaangustc.edu...oo cn
ABSTRACT
Test-time adaptation (TTA) of Vision-Language Models (VLMs) has emerged a technique for tackling distribution shifts during the test time. Recent research indicates that the test-time adaptation is intrinsically linked the model's s train- ing history. However, existing TTA methods, such as Test-time Prompt Tuning, often design adaptation strategies in isolation from the models' training charac- teristics, which degrade their performance. This paper argues that the flatness ac- quired via sharpness-aa aware training is an efficient clue for the test-time adaptation of VLMs. Built on this insight, this paper proposes novel Flatness-Guided Adap- tation framework (FGA) for VLMs to cohesively unify training and test-time pro- cedures. Its core idea is to leverage the alignment between the training minimum and test loss flat regions to guide the adaptation process. Specifically, our FGA consists of prompt-tuning stage and test-time adaptation stage. In the tuning stage, a Sharpness-Awar ing flat minimum, offering geometric clue flatness for subsequent adaptation. In the test stage, a Sharpness-based Test Sample Selection approach is proposed to ensure the alignment of flat minima between the training and each augmented sample's 's loss landscape. In comparison to existing TTA methods, our FGA avoids the expensive prompt parameter updates during test time, and substantially re- duces the computation overhead. Extensive experiments on both domain general- ization and cross dataset benchmarks demonstrate that our FGA achieves superior performance over prevalent TTA methods. Notably, when employing ViT-B/16 image encoder, FGA even outperforms TPT+CoOp by an average of 4.88% across all four ImageNet out-of out-of-domain variants.
Prompt Tuning method is utilized to identify the train-
INTRODUCTION
Recent advancements in vision-language pretraining, such as CLIP (Radford et al.] 2021), have generated new opportunities for developing foundational models in vision tasks (Jia et al. 2021 Yang et al. 2022). These models, trained on extensive collections of image-text pairs, can learn and represent diverse range of visual concepts. By means well-designed prompts, they can be applied to downstream tasks in a zero-shot manner without requiring task-specific data (Li et al.) 2022
Ramesh et al. 2022 Patashnik et al. 202T). Consequently, various prompt tuning methods (Zhou et al. 2022b are proposed to directly learn prompts using training data from downstream tasks. Though these methods find better prompts compared to hand-crafted ones, the learned prompts are limited to the training distribution and may have limited generalization beyond that.
To address this issue, several studies (Chnn et al. 2022 Boudiaf et al. 2022 Wang et al. 2020 have attempted to develop test-time adaptation (TTA) methods, which aim to rapidly adjust pre-trained models to unlabeled test data streams during inference. Among various TTA strategies, methods based on Test-Time Prompt Tuning (TPT) (Shu al. (022)). which optimize a set of learnable prompts via entropy minimization on augmented test views, have demonstrated promising perfor- mance and gained significant attraction (Shu et al. 2022 Feng et al. 2023 Yoon et al. 2024). Re- cent research indicates that the test-time adaptation is intrinsccally influenced by the model's 's training history (Goyal al. 2022). However, most existing TTA methods, including TPT-hased methods,

---

Published as a conference paper at ICLR 2026
0.2
0.1
5:54
LosS
Landscups
800
Alxard Lasttaeeeappe
9.04
002
0.89
(a) Traditional TTA Methods
(b) Our FGA
Figure 1: Comparison of conventional TTA methods and Flatness-Guided Adaptation (FGA). (a) Traditional TTA methods treat the test landscape as statie, aiming to optimize parameters to achieve the test flat minimum (*), using the training minimum (*) as an initialization.
(b) Our FGA keeps parameters unchanged during testing. Instead, it adjusts the test landscape to a position where the training minimum (+) is already very close to the minimum (+) of the adapted test landscape.
design adaptation strategies in isolation, treating the test phase as a standalone optimization problem disconnected from the model's training history. This isolation from the training phase may fail to exploit valuable geometric and representational properties inherent in the pre-trained model, leading to suboptimal test-time adaptation.
To improve model generalization, seeking flat minima within the training loss landscape has emerged as an effective training strategy over the past few years (Foret al., 2020 Kwon et al.] 2 Kim et al., 2022). It is widely observed that parameters residing in flat minima tend to gen- eralize better to dus----ditto data (Cha et al., 202T Zhang et al., 2024b Li et al. 2025 Zou et al. 2024) than those sharp ones. Nonetheless, conventional TTA methods often ignore the influ- ence of sharpness-awart training on the test-time adaptation. While Sharpness-Ave Aware Minimization (SAM) (Foret et al., 2020) seeks flat regions during training, its principle is rarely extended to guide test-time adaptation in a unified framework. This disconnection leads to computationally expensive test-time optimizations (e.g., backpropagation in TPT (Shu et al., 2022)) that are agnostic to loss geometric structure and often yield suboptimal generalization. This paper argues that flatness is not merely a desirable property during training but a powerful clue that can dictate test-time adaptation.
Inspired by this insight, this paper proposes a novel Flatness- Guided Adaptation (FGA) framework, which cohesively unifies training and test-time procedures from the perspective of loss landscape geometry. It mainly leverages the alignment between the training minimum and test loss flat re- gions to guide the adaptation process (see Figure D. Specifically,, our FGA framework consists of two synergistic stages: (1) In the prompt tuning stage, a Sharpness-Awar Prompt Tuming (SAPT) method is utilized to fine-tune the prompts on the downstream training dataset, aiming at seeking the training flat minimum. Since flatter minima generally indicate better model generalization than sharper ones (Keskar et al., 2016 Dziugaite & Roy 2017| Jing et al.) 2019, Foret et al.) 2020),
the minimization of sharpness not only improves model generalization but also provides test-time criterion to measure the alignment of flat minima within the training and test loss landscapes. (2) In the test-time stage, FGA leverages the geometric clue of flatness acquired via SAPT. For a given test sample, a Sharpness-based Test Sample Selection (STSS) method is proposed to intelligently select its augmented views based on the sharpness score of their loss landscapes around the train- ing flat minimum. This ensures that the final prediction is derived from a test-time loss landscape whose flat minima align with those identified during training. During this process, loss landscapes are efficiently altered through data augmentations. In comparison with existing TTA methods, our FGA avoids the expensive prompt parameter updates during test time, eliminating the computational overhead of adaptation and offering more plausible adaptation strategy. Theoretical analysis sug- gests that using the sharpness- based metric will help distinguish the proximity of test samples to the training distribution. The closer an augmented sample is to the training distribution, the smaller its sharpness-ber based score is likely to be. Since models tend to generate more reliable results for data closer to the training distribution, FGA significantly improves the generalization ability of vision- language models. Extensive experiments on domain generalization (Hendrycks et al., 2021b) and cross-dataset (Zhou et al. 2022a) benchmarks demonstrate the superior performance of FGA over prevailing TTA methods.
2

---

Published as a conference paper at ICLR 2026
Our main contributions can be summarized as follows:
A novel Flatness-Guided Adaptation (FGA) framework is proposed to cohesively unity training and test-time procedures for ussionllanuun models. By ensuring the alignment of the model's training flat minimum with flat regions in test loss landscapes, it significantly enhances the gen- eralization capabilities of VLMS under distribution shifts.
Theoretical analysis is presented to offer a clearer insight into how sample selection test time improves the reliability of predictions.
Extensive experiments domain generalization and cross-dataset benchmarks demonstrate the superior performance of FGA over other prevalent TTA methods, while significantly eliminating the computational overhead.
RELATED WORK
Test-time adaptation (TTA) of vision-language models. Vision-language models like CLIP have shown strong performance in various tasks. To enhance CLIP's transfer learning for downstream classification tasks, methods like text prompt learners (e.g., CoOp (Zhou et al. 2022b and Co- CoOp (Zhou et al. 2022a)) and visual adapters (e.g., Tip-Adapter (Zhang et al.) 2022)) have been proposed. However, these methods struggle with distribution misalignment between pre-training and test data. Test-time adaptation (TTA) methods address this by adjusting models during testing, with two main streams (Abdul Samadh et al., 2024): the first modifies the training process using a self- supervised proxy task, such as image rotation prediction, and uses it to guide test-time optimization (c.g., Test-Time Training (Sun et al. 2020) and TTT++ (Liu et al. (202)); the second adapts models without altering the training process (e.g.. TPT (Shu et al. (022). which uses entropy minimiza- tion to learn adaptive parameters during testing). DiffTPT (Feng et al. 2023) introduces a diffusion model to generate diverse augmentations for further improvements. PromptAlign (Abdul Samadh et al. 2024) adds an explicit term to align the learned distributions with that of test data. Meanwhile, online methods like TDA KKarmanov al. 2024) and DPE (Zhang et al., 2024a) use a key-value cache or prototype set to adapt progressively to test data. They benefit from information aggregated during testing but are unsuitable for single test-sample scenarios, unlike TPT-based methods. This paper proposes novel of loss landscapes to enhance CLIP's generalization and inference efficiency in single sample adaptation scenarios. By avoiding backpropagation and parameter updates during testing, FGA sig- nificantly reduces computational overhead while achieving robust out-of-domain performance.
Sestness-Guideds Adaptation (FGA) framework that leverages the geometry
Generalization from a loss landscape view. In recent years, optimization techniques aimed at flat minima loss landscapes have surged to improve the generalization of deep models (Keskar al.) 2016 Dziugaite & Roy 2017 Jiang al. 2019). Among them, SAM (Foret et al., 2020). which focuses on finding parameters located in regions of the loss landscape with consistently low loss values, has gained significant attention for its effectiveness and scalability. To seek flatter minima, numerous SAM variants, such as ASAM (Kwon et al. (021) and FisherSAM (Kim et al.) 2022), have already been developed over the past few years. This concept of flat minima has also been extended to improve the out-of-domain generalization of deep models (Zou et al.. 2024 Cha et al., 2021 Li al. 2025). However, most of them focus the training stage. SAR (Niu et 2023) and SoTTA (Gong et al. 2023). two online test-time adaptation (TTA) methods, both utilize sharpness- aware minimization test time to improve robustness by seeking flat minima. Yet, they operate solely during testing without accounting for training-testing sharpness interactions. In contrast, our FGA applies sharpness-awand minimization during training to establish flatness as a criterion for subsequent alignment, then at test time adapts by adjusting loss landscapes through augmentation selection (without updating model parameters), and preserving the pre-trained flat minimum's opti- mality on adapted test loss landscapes.
METHODOL.OGY
3.1 PRELIMINARIES
Contrastive Language-Image Pre-training. CLIP (Radford et al.) 2021) primarily comprises Text Encoder E, and an Image Encoder E,.. The Image Encoder available in two architectures:
3

---

Published as a conference paper at ICLR 2026
based ResNet (He et al., 2016) and the other using the popular Vision Transformer (ViT) (Dosovitskiy al. 2020). This encoder transforms an input image æ into its feature rep- resentation, i.e., e, (ar). For classifica- tion task with K classes, the k-th class label is formatted into the textual prompt "a photo of a [k-th class]", which is then tokenized into sequence 4 (fsos, f2,.2....... LL. ck2 (eos). Here, Isos and (EOs represent the embeddings of the start and end tokens, while (, .,,,....... .....,,, corresponds to the phrase "a photo of a", and the token C₁ denotes the specific description of the k-th class. The text encoder of CLIP, de- signed as a Transformer architecture, processes these tokens to generate text features: Ct,k (Ix). During the pre-training stage, CLIP trained the WIT dataset (Radford et al.] 2021) through contrastive learning approach. In this setup, each image paired with its corresponding text sentence as a positive sam- ple, while all other image-text combinations are treated as negative samples. The goal of the contrastive learning objective to enhance the cosine similarity of positive pairs while reduc- ing that of negative pairs. In the classification stage, all classes in the dataset converted to text, and the cosine similarity between im- age embeddings and text embeddings is com- puted to determine the probability of an image belonging each category:
(0)
cat
Text Engader
Lauirali prompee
dog
Sharpness- aware Loss
Image Encoder
Training
task
call
Text Encoder
dog
prompe
Text Features
Sharpeess-
Sharpnese-
Text Features
Seloct
Encoder
Test N "'ugmmniations
Ouiputs
Figure 2: Overview of our Flatness-Guided Adaptation (FGA). consists of two synergistic Prompt Tun- ing: optimizes the model parameters to reduce the loss value and sharpness, enabling stable and effective adaptations during test time without di- rect access to training data. (b) Sharpness-based Test Sample Selection: It introduces a selection mechanism to identify augmented test samples that ensure the training flat minimum aligns with those in their loss landscapes, enabling more con- fident predictions.
mechanisms: (a) Sharpness-awan
exp (sim (er, e4)
(yhk x))
K Σラ= exp (sim (er. e,) e,)
where T is the temperature of the softmax.
Prompt tuning. Prompt tuning has emerged popular tuning method for Transformer-based models in downstream tasks. This approach does not modify the model parameters; rather, changes the input to the model. making it highly efficient. Specifically, instead of using the template "a photo of a [class]", it replaces the tokens associated with the hand-crafted prompts ("a photo of a²)) with learnable parameters p (Pi...., which are then updated based the dataset used for downstream tasks.
Test-time prompt tuning. Prompts optimized downstream training data are prone to overfitting and often generalize poorly under distribution shifts. To mitigate this, Test-Time Prompt Tuning (TPT) (Shu et al. 2022) fine-tunes a specific prompt for each individual test sample. During testing, multiple augmented views of the test samples are generated. Then, predictions with entropy below a predetermined threshold are kept, while others are discarded using confidence filter. The averaged entropy of selected predictions is then used as a loss function to update the prompts.
3.2 PLATNESS-GUIDER ADAPTATION
Our proposed Platness-Guider Adaptation (FGA) framework fundamentally offers unified, loss landscape-centric methodology that seamlessly bridges the training and test phases. This approach leverages the flatness universal guiding principle to enhance both generalization during training and robust adaptation during inference under distribution shifts. As illustrated in Figure 2 FGA integrates two complementary mechanisms that operate in concert: sharpness-awee aware prompt tuning (SAPT) during training and sharpness-guide guided test sample selection (STSS) during inference. This section will elaborate the key idea and technical details of the framework.
4

---

Published as a conference paper at ICLR 2026
3.2.1 SHARPNESS-AWAr PROMPT TUNING
FGA mainly exploits the alignment between training and test flat minima for the efficient adapta- tion of VLMs. However, a major challenge in achieving this alignment arises from inherent data constraints. Test samples remain unknown during training, and test time, training data becomes inaccessible due to storage limitations and privacy requirements. To overcome this, FGA focuses on the intrinsic properties of the training minimum, with the goal that the test loss landscape ex- hibits these same desirable characteristics: (1) The training minimum should correspond to a low loss value, indicating that the model effectively learning from the data; (2) The training minimum may display implicit biases, such as reduced sharpness, which are often beneficial for improved generalization.
Traditional training methods for optimizing the prompts, such CoOp (Zhou et al. 2022b), typi- cally use cross-entrope loss to fine-tune the prompt p:
(cE(D)= = Σ log F. (v)((x)))
where Pp(yi (y. (x)) represents the predictive probability that æ; belongs to the its true label class. While standard SGD tends to find flat minima, methods such as SAM (Foret et al. 2020) can enhance this implicit bias through explicit perturbation-awant optimization. To enable more precise alignment during testing, adopt Charpness-awan Prompt Tuning (SAPT) during training, which jointly minimizes both the loss and its "sharpness":
(SAPT()) = (CE(P) + max [(CE(P + (ce(p))
(3)
|el|sso
The first and second terms above represent the loss value and loss sharpness, with À acting as a hyperparameter to balance them. Similar to previous studies (Foret al. 2020 Kwon al. 2021),
sharpness is defined as the sensitivity of the training loss to small perturhations (with a norm less than p) added the prompts p. Since the perturbation strength p small enough, we can apply Taylor expansion to approximately solve for the optimal perturbation e":
VpfcE(p) ( ([Vppcr(p))
E" = arg max LE (p- - -) - (pF(p) Ro arg max é T VVpCC((p) [<|lSp
(4)
||c|SS
During training via stochastic gradient descent, the contribution from Vpe* can disregarded due to the minor perturbation strength p.
In this way, SAPT not only yields robust prompts that enhance generalization but also provides the sharpness measure additional information for adaptation during testing.
3.2.2 SHARPNESS- -AARPE TEST SAMPLE SELECTION
Through sharpness-aware prompt tuning, the prompts are positioned at a flat minimum within the training loss landscape. To avoid computationally expensive gradient descent during inference, keep the pre-trained prompt fixed and instead adapt the test loss landscapes such that the well-trained prompt from the downstream training dataset (i.e.. the training flat minimum) coincides with the flat minimum in the adapted test landscape, as illustrated in Figure
To achieve this alignment, we propose a Sharpness-based Test Sample Selection (STSS) method. STSS utilizes data augmentations to create multiple test loss landscapes for each sample. By select- ing augmented samples that align the training minimum with flat minima in their respective loss landscapes, we ensure the training minimum remain optimal. Given that such alignment typically corresponds to small loss values and reduced loss sharpness in these test landscapes, STSS intro- duces a sharpness-basee based score as a metric. To mitigate the computational burden of backpropagation in calculating sharpness, we redefine it as the maximum variation in the loss resulting from R ran- dom perturbations:
fsrss( (p) tSRG (p) +
ESRG (p+ p'
Cr fSRc (p)
(5)
m........... N
||r,
Here, EsTss represents the sharpness-based score used to select the most reliable augmented test samples, and εsng denotes a surrogate loss function when test labels are unavailable, such as en- tropy (Wang et al. 2020 Goyal et al. 2022). The perturbation direction is expressed by e, <,/||er]],
5

---

Published as a conference paper at ICLR 2026
where Ep is drawn from the standard normal distribution N. The quantity p' controls the magnitude of perturbations during testing, To obtain ÉSRG (p+ p' ||
we first obtain text features for each category (er, through the forward pass of the text encoder:
(e..,, = E. RRRR))
(6)
where 1g,r denotes the input token related to the k-th category with the r-th pertuation. Notably, the additional computational cost of this step is minimal, text features only need to be computed once per test category. Then, the surrogate loss for perturbed prompts is:
tSRG (p
d'
Er
(yn (x) log (ux)z),
|6
=
with probabilities derived from cosine similarity:
exp (sim (er,x.r e;) T)
P. (y+|)))
(8)
Σ/= exp (sim (et,j,r e,) T)
Finally, the final prediction aggregates votes from the top s augmented samples with the lowest sharpness brrseedss-bbeed scores, which more reliable predictions according the theoretical analysis in the next section.
4 THEORETICAL ANALYSIS
This section provides theoretical explanation of how our method improves test-time classification. Let's begin with the following problem: During training. the model learns from data sampled in- dependently and identically from distribution S; During testing, however, data is drawn from two distinguish between these test
distinct distributions Ti and T2. Then, the question is: How distributions and determine on which one the model will perform more reliably?
To address this, first derive upper bound for the generalization error, which quantifies the model's 's performance on unseen data from T and Tz. We will then explore how, when the test dis- tributions are sufficiently distinguishable, FGA can effectively distinguish between them. This is crucial because, as we will show, when the test distribution closely resembles the training distribu- tion. the generalization error bound decreases, leading to more accurate predictions.
Theorem (Generalization Bound) Consider vector-valued function class F (ff()) : X
RK] } and bounded loss function { RR x y + [O. Define the worst-case loss l within p-radius perturbation as
e (ff((r),)) max {(fo+e(x)..))
NellaSS
Assume S is j-Lippchitz continuous with respect to f. Let S and T denote the training and test distributions, respectively. Denote S. {æ yı }== bution S. Then, with probability at least ô, the following generalization bound holds:
i.i.d. training samples drawn from distri-
M dS; 8 + Rm M
log(1/8)
ET (o
(10)
2n
where (wr, VT) is random pair following the distribution T; 82 (fo) = : : le (fo(a))) v.)
denotes the empirical loss over Sni d(S; T) is the distribution discrepancy measure defined in Ap- pendix A; and (F,, represents the Rademacher complexity of F, on S.
In the following, we will show that when the two test distributions are sufficiently distinguish- ablc-compared with the tightness of the above upper bound can effectively differentiate be- tween them. To proceed with this analysis, we first introduce the concepts of bound tightness and distribution separability.
Definition 2 (3-tightness) Given random variable X and confidence level 5 (0, 1), let be an upper bound such that P(X < a)) 2 1-66 If there exists an oracle upper bound at satisfying P(X S a²) = 5, then is said be B-tight with 3 la a²\

---

Published as a conference paper at ICLR 2026
Table I: Results on datasets with natural distribution shifts. We report top-1 accuracy (%) for each method across five datasets, using the CLIP-VIT-B/1C backbone. We highlight the best results in bold and underline the second-best results. The abbreviation "IN" means the ImageNet dataset. "I" denotes results reproduced by adapting the method to the single-sample setting.
Algorithm
IN
IN-A
IN-V2
IN-R
IN-Sketch
Avg.
OOD Avg.
CLIP- VLT--V Radford et al. 2021
68.34
49.89
61.88
77.65
48.24
61.20
59,42
CoOp (hoou et al. 2022b
71.51
49.71
64.20
75.21
47.99
61.72
59.28
CoCoOp Zhtuu
20
71.02
50.63
64.07
76.
48.75
62.13
59.91
Tip-Adapler Zhang ct al. 2022
70.75
51,04
63.41
77.76
48.88
62.37
60.27
TPT (Shu et al. 2022
69.70
53.67
64.30
73,90
46,40
61.59
59.57
DIF Feng el ale 2025
70.30
55.68
65.10
75.00
46.80
62.58
60.64
C-TPT Yoor 2 [0:4
52.90
63.40
78.00
48,50
60.70
ZERO (Farina a. 2024
69.06
61.35
64.13
77.28
48.29
64.02
62.76
MTA Zanzlla BCn Aved
2024
69.29
57.41
63.61
76.92
48.58
63.16
61.63
PromptAlign (Abdul Saaaad al. 2024
59.37
65.29
79.33
50,23
63.56
TDA (Karmanoy el al. 2024
69.51
60.11
64.67
80.24
50.54
65.01
63.89
DPE Zhang
207248
71.91
59.63
65.44
80,40
52.26
65.93
64.43
TPT Shu et al 20A Cop
73.30
56.88
66.60
73,80
49,40
64,00
61.67
Dir Feng et m. 2023
+CoOp
75.00
58.09
66.80
73,90
49,50
64.66
62.07
C-TPT Yoon al. 20:4
CoOp
72.90
52.73
65.61
76.46
48.63
63.27
60.86
ZERO
P044]
CCOpp
73.61
63.17
66.82
77.71
48.52
65.97
64.06
MTA
Zanella Ben Ayed
2024
CCoOp
73.99
59.29
66.97
78.20
49.96
65.68
63.61
SAR!
NNu el. 2023
#CoOp
73.03
55.35
65.89
77.09
48.65
64.00
61.75
SAPT+CoOp
70.79
51.04
64.41
77.66
49.31
62.64
60.61
STSS+CoOp
73.99
64.00
67.11
77.92
49,36
66.48
64.60
FGA (Ours)
74.01
65.90
67.23
81.24
51.81
68.04
66.55
Definition 3 (y-separabilitt) Let Tı and To be two test distributions and S the training distribution. Denote by S): the discrepancy between a distribution and S. Then Tr and Tz are called 71- separable if d(Ti: - d(T2: >
Theorem Consider vector-valued function class J, (fe X +RK), where the parame- ters O lie in a set θ such that the loss function is bounded within MM). Let f =
and y (y1,...,ye .K) be probability distributions over a finite set f K}. with fu,y, 2 7 > 0 for all i. Denote by Hff) the entropy and by H(y, f) the cross entropy. Assume the worst-case cross entropy Hy f) p-Lipschitz continuous with respect to f over θ. Given training distribution S and two "-separable test distributions Ti and Tz, assume d(S, Ti) < d(S, T2). Define the quan- tile function Q5(5) f for the entropy loss of So on To such that PP (H < E[HP] + (,))) = 1- and let (()) Sup(Q1(8), Q2(5)} be the supremum quantile. Then, with probability at least 1 8:
M (S; Q(8/2) Hg. + (F. M
log (2/8)
H)(fo(TT))) S
2n
+ loog 1 Eslly fz(())l1 " Eslly -f fg(x))1
(11)
1
Here, the notation θ is defined Ø = 0 + arg argmaxics max (H(y, fo+e(x)),, H (f)))()))) Furthermore, if this bound is Be-tight for Tj and Tz with B, < ",, then there exists a threshold & such that:
P ((H < <) (11 (fe(a,)) <t)
(12)
This inequality indicates that a test distribution further from the training distribution tends to exhibit a higher sharpness score. By comparing sharpness scores across test distributions, we can identify which one is closer to the training distribution, thus yielding more reliable predictions. Notably, the tunable parameter p controls the tightness the upper bound, facilitating a precise differentiation between test distributions and improving the model performance. It is important to note that in the theoretical analysis presented in this section, we do not distinguish between P and p' (which are utilized to calculate the sharpness of the training and test loss landscapes, respectively). However, in practical implementation, we may opt to use different values for p and ø' for better performance. Due to space limitations, detailed proofs and further discussions are provided in the appendices.

---

Published as a conference paper at ICLR 2026
Table 2: Cross-dataset generalization from ImageNet to fine-grained classification datasets. During the prompt tuning stage, the prompts are tuned on ImageNet with 16-shot training data per category, using ViT-B/16 image encoder.
Method
[Caltech101
Pets
Cars Flowers 102
Aireraft SUN397
DTD Eurosal Food101
UCF101]
Avg.
CLIP-VIT-B11 (Radford et al. 2021
93.35
88.25 65.48
67.44
23.67
62.59
44.27
42.01
83,65
65.13
663.58
CoOp (Zoou et
2022h
93.70
89.14 64.51
68.71
18.47
41.92
85,30
66.55
63.88
CoCoOp Zhou m
20220
94.43
90.14 65,32
71.88
22.94
67.36
45.73
39.23
83,97
68.44
64,94
TPT Shu et al. 2022
94.16
87.79 66.87
68.98
24.78
65.50
47.75
42.44
84.67
68.04
65.
Di
2023
92.49
67.01
70.10
65.74
47.00
87.23
62.67
64,92
C-TPT
Yoon ot al
1024
93:60
88.20 65.80
69.80
24.00
64.80
46.00
43.20
83,70
65.70
64,48
ZERO
Farmaa al.
2024
93.66
87.75 68.04
67.68
25.21
65.03
46.12
34.33
86.53
67.77
64.21
PromptAlig (Abilul Samadh et al.
2024
94.01
90.76 68.50
72.39
24.80
67.54
47.24
47.86
86.65
69.47
66.92
TDA
(Karmanov et all. 2074
94,24
88.63 67.28
71.42
23.91
67.54
47,40
58.00
86.
70.66
67.53
TPT+C8Op (Zhou
2022b
93.75
88.93 67.06
68.25
25.89
66.40
47.15
48.78
83.82
66.53
65.66
+MT (KliaLaaa (Kliattak at 2025
93.59
90.72 66.50
72.37
24.70
67.54
45.87
47,80
86.64
69.19
66.50
ZERO (Farina at
Cop
93.85
67.23
19.14
43.62
33.53
82.67
66.61
62.46
ZERO (Farna ot a.. 2024
MMPLL
94.48
90.60 68.58
71.62
26.25
68.20
45.86
42.17
86.77
69.87
66.44
FGA (Ours)
96.96
91.28 68.93
72.11
26.97
69.29
49.76
47.58
84,95
68.17
667.60
5
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Datasets. We conduct two types of experiments evaluate the model's robustness to natural dis- tribution shifts and its cross-dataset generalization capabilities, following previous research such as TPT (hhu et al. 2022). To assess the model's robustness to natural distribution shifts, we ap- ply prompt tuning on the ImageNet (Deng et al. 2009) dataset, and evaluate its performance on four ImageNet variants: ImageNet-A (Hendrycks et al. 2021c). ImageNet- V2 (Recht et al., 2019), (Wang al., 2019), which is also known as the domain generalization task. In addition, we perform cross-dataset evaluations for im- age classification across 10 datasets, each from distinct domain with different classes: including Caltech101 (Fei-Fei et al., 2004),, OxfordPets (Parkhi et al.] [2012). StanfordCars (Krause et al. 2013), Flower102 (Nilsback & Zisserman 2008), Aircraft (Maji et al. 2013), SUN397 (Xiao al.
ImageNet-R (Hendrycks et al., (021a) and ImageNet-
2010 DTD (Cimpoi et al. 2014). Food101 (Bossard et al., 2014). UCFI01 (Soomro (012) and
Eurosat (Helber et al. 2019). y. In this experiment, ImageNet serves as the source dataset, while the
remaining fine-grained datasets are used as target datasets for evaluation.
Implementation details. Our experiments are based on pretrained CLIP (Radford et al. 2021 mod- els. specifically CL.P- (using ResNet50 image encoder) and CLIP- VTT-B/1G (using Vision Transformer image encoder). Due to space limits, we focus on reporting the experimental results of CLIP-VTT-B/16 deferring those of Ceip-ResNet50 to the appendix. In the prompt tun- ing stage, our experiments are built on the CoOp (Zhou al. 2022b) framework. The prompts are trained in 16-shot manner the ImageNet dataset. We set the number of prompts to 4 and utilize the SGD optimizer, with a learning rate of 0.002. For cross-dataset and domain generalization tasks, the prompts were trained for and 50 epochs, with batch sizes of 4 and 32, respectively. The key hyperparameters p are determined through a grid search, with the values ranging from [0.05, 0. 0.3. 0.5, 0.7]. During testing, existing TPT-based methods usually leverage the input image along with its 63 augmented views. To ensure fair comparison, we consistently employ the same data augmentation strategy as TPT in all experiments. To avoid tuning hyperparameters on test data, we just set x= = 1 and p' = 0.5 for all experiments. Please refer to the Appendix for more discussions about other hyperparameters.
5.2 MAIN RESULTS
Robustness to natural distribution shifts. We first compare the proposed FGA with prevalent TTA techniques on ImageNet and its variant out-of-domain (OOD) datasets. The results, presented in Ta- ble I
highlight the superior performance of FGA across several ImageNet-based OOD datasets. No- tably, even the ablated version of our method, STSS+CoOp. exhibits strong performance, surpassing all previous approaches with an OOD average of 64.60% and an overall average of 66.48%. We at- tribute this robustness to the fact that standard SGD training already imbues models with an implicit bias toward flatter minima. By explicitly enhancing this geometric property through SAPT, the full
8

---

Published as a conference paper at ICLR 2026
0.790
5
0.785
5m6
10
Accuracy (1)
0.780
0.775
5526
0.770
--- 5-30
0.785
0,760
0.2
0.4
0.6
0.8
1.0
(a) Case
(b) Case
Figure 3: The influence of the key hyper- parameter p' on the test accuracy.
Figure 4: 2D Visualization loss landscapes associ- ated with different augmented test samples.
FGA algorithm achieves a substantial eaap in generalization performance. Specifically, when com- pared to TPT+CoOp, FGA shows an average accuracy improvement of 4.88% (61.67% - 66.55%) on the OOD variants. Furthermore, our FGA also consistently surpasses other powerful TTA meth- ods (e.g., DifTTPT, C-TPT, ZERO, MTA, and SAR) when they are combined with CoOp. It is crit- ical to note that while online TTA methods like SAR benefit from aggregating information across test data stream, FGA operates in a more challenging single-sample adaptation setting. For fair comparison, we have adapted SAR to the single-sample adaptation setting. These superior results of FGA strongly demonstrate its effectiveness in enhancing CLIP's out-of-domain generalization across diverse datasets.
Cross-dataset generalization. We also observe superior performance of the FGA in evaluating cross-dataset generalization from ImageNet to various fine-grained classification benchmarks. Based on the comprehensive results presented in Table 2, the proposed FGA method demonstrates supe- rior overall performance, achieving the highest average accuracy of 67.60% and attaining top-tier results 6 out of 10 datasets, including notably strong performance on Caltech101 (96. 99%... Furthermore, FGA (67 60%%) achieves an average accuracy improvement of 1.94% over the power- ful baseline TPT+CoOp (65.66%). It also exhibits superior performance over the combinations of TTA methods (like TPT and ZERO) and different tuning methods (CoOp and MaPLe). Please note that due the significant difference in TTA settings, is not imperative to expect single-sample TTA methods to surpass online TTA methods like TDA. Overall, these superior results further validate its effectiveness in adapting to diverse datasets during testing. It particularly valuable for VLMs like CLIP, as it enables models recognize more fine-grained categories in image classification without the need for additional training.
Runtime and memory efficiency. To evaluate the practical deployment advantages of FGA, quantify its computational overhead in terms of inference time per test image and peak GPU memory consumption. All experiments conducted on single NVIDIA Tesla V100 GPU under consistent settings. Specifically, FGA requires only 0.07s per image from ImageNet, which is 23.86x faster than DifTTPT (1.67s) and 8.86x faster than TPT (0.62s). In terms of memory usage, FGA consumes merely 4.14 GB, which is 4.67x lower than that of TPT (19.33GB). These results demonstrate that FGA achieves highly competitive test-time adaptation performance while maintaining notably low runtime and memory overhead, making it suitable for real-time and resource-constrained applica- tions.
5.3 ABLATION STUDY
Main components analysis. Our ablation study on the domain generalization benchmark (using
CLIP-ViT-BA architecture, shown in Table validates the necessity of each FGA component: (1) Sharpness-aware prompt tnning (SAPT) enhances generalization, boosting CoOp's ssss average accuracy by 0.92%
on ImageNet and OOD datasets; (2) Test-time sharpness selection (STSS) drives major gains, with CoOp+STSS outperforming CoOp by 4.76% 11.72%-~66.448
(3) SAPT synergistically enhances STSS, where full FGA (Co0p+SAPT+STI) gain over CoOp+SAPT (62.64%-66 (8.004) (exceeding standalone STSS improvements of 4.76%). This confirms that flatter minima from SAPT intrinsically improve test-time sample selection.
achieves 5.40%

---

Published as a conference paper at ICLR 2026
Ablative analysis on key parameter p'. Theoretical analysis (Section 4) establishes p and p' as key generalization controllers, playing significant roles during the training and testing stages, re- spectively. Since p's role has been well explored in previous research (Foret et al.] 2020), we focus on p' for test-time adaptation: as mentioned earlier, it governs distribution distinguishabiliyy. with proper values enhancing prediction reliability through sensitive discrimination. Empirical valida- tion on ImageNet-R (Figure 3) shows non-monotonic accuracy dependence on p'-initially rising then falling. It is because extreme values (p' measures and degrade performance. Crucially, P = 0 degenerates to entropy maximization, and its comparison with non-zero cases also demonstrates sharpness's 's necessity. Notably, all experiments fix p = 0.5 without test-data tuning, and this analysis solely aims to demonstrate the control ef- fect of p' on generalization. Additionally, sample retention follows a similar trend: accuracy peaks then declines with increased retention. This reflects the probabilistic correlation: lower sharpness typically means greater proximity to the training distribution, meaning performance degrades when retaining excessively high-sharpness samples.
p' > 0) may yield uninformative sharpness
5.4
VISUALIZATIONN OF LOss LANDSCAPES
To intuitively validate FGA's effectiveness, we visualize the test data's 's loss surface using a 2D tech- nique (Li al. 2018) Figure 4 revealing how sample selection enhances prediction reliability. The visualization demonstrates critical relationships: when parameters reside in flat minima (Figure top), augmented samples maintain semantic integrity and enable reliable predictions. Conversely, parameters outside flat minima (Figure 4 bottom) yield distorted semantic representations that de- grade generalization. This contrast clearly illustrates that FGA can help filter out unreliable test samples to mitigate their adverse effects, thereby strengthening the generalization ability of VLMs.
6 CONCLUSION
This paper demonstrates that flatness operates not just beneficial training characteristic but as a key geometric clue for test-time adaptation. This understanding motivates the proposal of a novel framework, Flatness- Haltnes--Guided Adaptation (FGA), which utilizes the principle of loss landscape flat- ness as a unified guide to improve both training and test generalization against distribution shifts. Different from previous TTA methods that often fine-tune prompts per sample, it directs adaptation by leveraging the geometric relationship between training minima and test-time loss landscapes. Specifically, it first identifies flat minima during prompt tuning and then ensures the alignment across training and test landscapes via a selective mechanism. Comprehensive experiments and theoretical analysis confirm FGA's effectiveness and superior performance. We anticipate this work will ad- vance the understanding of loss landscapes and inspire future TTA technologies.
Future work. Our proposed FGA is grounded in a general analysis of loss landscape geometry, foundational concept that is broadly applicable across diverse model architectures and learning paradigms. This foundation makes FGA a flexible component that could be integrated into modern visual-language models, advanced prompt tuning methods, new types of test-time adaptation objectives to potentially enhance their performance. In this work, we intentionally followed the experimental setup introduced in the TPT paper. This choice allows controlled and fair comparison with prior TTA methods, helping us to clearly demonstrate the contribution FGA itself. The extension of FGA to other experimental configurations would require extensive engineering efforts to conduct large-scale experiments across diverse settings, and thus remains our future work.
ACKNOWLEDGEMENTS
This work was financially supported by the National Natural Science Foundation China (NSFC) under Grant N2.020000 The authors gratefully acknowledge the NSFC for its critical support of this research.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Jameel Abdul Samadh, Mohammad Hanan Gani, Noor Hussein, Muhammad Uzair Khattak, Muhammad Muzammal Naseer, Fahad Shahbaz Khan, and Salman H Khan. Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization. Advances in Neural Information Processing Systems, 36, 2024.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 1eat-100-minnng discriminative compo- nents with random forests. In Computer ECcoon---VVV-- 2014: 13th European conference, zurich, Switzerland, September 6- 12, 2014, proceedings, part VI 13, pp. 446-461. Springer, 2014.
Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free online test- time adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8344 8353, 2022.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Infor- mation Processing Systems, 34:22405-22416 2021.
Dian Chen, Dequan Wang. Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 295-305, 2022.
Mircea Cimpoi. Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference computer vision and paltern recognition, pp. 3606-3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 218-2482255 255. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data, arXiv preprint arXiv: /703. /008, 2017.
Matteo Farina, Gianni Franchi, Giovanni Iacca, Massimiliano Mancini, and Elisa Ricci. Frustrat- ingly easy test-time adaptation of vision-language models. arXiv preprint arXiv:2405.18330, 2024.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178-178. IEEE, 2004.
Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmentation with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 2704-2714 14,, 2023.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
Taesik Gong, Yewon Kim, Taeckyung Lee, Sor Chottananurak, and Sung-Ju Lee. Sotta: Robust test-time adaptation on noisy data streams. Advances Neural Information Processing Systems, 36:14070-14093, 2023.
Sachin Goyal, Mingjie Sun, Aditi Raghunathan, and J Zico Kolter. Test time adaptation via conjugate lseudo-labels. Advances in Neural Information Processing Systems, 35:6204-6218 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
11

---

Published as a conference paper at ICLR 2026
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical generalization. In Proceedings of the IEEE/CVF international con- ference on computer vision, pp. 8340-83400 03400,839,,0 2021a.
analysis of
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international con- ference on computer vision, pp. 8340-8344 8009,,33,400 2021b.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pp. 15262-15271, 2021c
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference machine learning, pp. 4904-4916. PMLR, 2021.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint arXiv: 1912.018, 2019.
Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14162-14171, 2024.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv 2016.
preprint
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Maple: Multi- prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 19113- 2023.
Minyoung Kim, Da Li, Shell X Hu, and Timothy Hospedales. Fisher sam: Information geometry and sharpness aware minimisation. In International Conference on Machine Learning, pp. 11 148 11161. PMLR, 2022.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops. pp. 554-561, 2013.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. In International confer- ence on machine learning, pp. 5905-5914. PMLR, 2021.
Aodi Li, Liansheng Zhuang, Xiao Long, Minghong Yao, and Shafei Wang. Seeking consistent flat minima for better domain generalization via refining loss landscapes. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pp. 15349-15359, June 2025.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. Advances in neural information processing systems, 31, 2018.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Li- juan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-ing -mage pre- traiirin In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10965-10975, 2022.
12

---

Published as a conference paper at ICLR 2026
Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexan- dre Alahi. Ttt++: When does self-supervised test-time training fail or thrive? Advances Neural Information Processing Systems, 34:21888-218222 2021.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-graines visual classification of aircraft. arXiv preprint arXiv:/306.5155 2013.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics image processing, pp. 722-729. IEEE, 2008.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang. Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. arXiv preprint arXiv:2302. 12400, 2023.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEFF, 2012.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pp.
2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp. 5389-548444 5400. PMLR, 2019.
540..
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization vision-language models. Advances in Neural Information Processing Systems, 15:14274-1428, 2022.
K Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 111.::121 2012.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time train- ing with superviinn for generalization under distribution shifts. In International conference on machine learning, pp. 9229-9248. PMLR, 2020.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 10726, 2020.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Jinyu Yang, Jiali Duan, Son Tran, Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision-language pre-training with triple contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15671-15680, 2022.
Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Mark Josegnwa-Johnson, Yingzhen Li, and Chang D Yoo. C-tpt: Calibrated test-time prompt tuning for vision-language models via text feature dispersion. arXiv preprint 4rXiv:2403. 14119, 2024.
13

---

Published as a conference paper at ICLR 2026
Maxime Zanella and Ismail Ben Ayed. On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23783 23783323773, 2024.
Ce Zhang, Simon Stepputtis, Katia Sycara, and Yaqi Xie. Dual prototype evolving for test-time generalization of vision-language models. Advances in Neural Information Processing Systems, 37:32111-32138 2024a.
Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-free free adaption of clip for few-shot classification. In European conference on computer vision, pp. 493-510. Springer, 2022.
Ruipeng Zhang, Ziqing Fan, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. conaan-inspired -wware aware minimization under domain shifts. arXiv preprint 2xXv:24455/886/1 2024b.
sharpness-awe
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16816 16816-16825, 2022a
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-2344 2022b.
Yingtian Zou, Kenji Kawaguchi, Yingnan Liu, Jiashuo Liu, Mong- Li Lee, and Wynne Hsu. Towards robust out-of-distribution generalization bounds via sharpness. arXiv preprint arXi::403.00639, 2024.
14


Reviews:
[{"review_id": "yjDLfPCnEv", "reviewer": "Reviewer_RFcj", "summary": "The article proposes an approach for test-time adaptation of vision language models based on flat minima. Specifically, the approach (FGA) is composed of two elements: (i) sharpness-aware prompt tuning (SAPT), which adds a regularizer toward flat minima when tuning prompts in the style of CoOp (Zhou et al. 2022b), (ii) sharpness-based test-time sample selection (STSS), which augments test samples and selects the top augmented views according to the flatness of their loss landscape. Experiments on three benchmarks show the efficacy of the approach.", "questions": "1. Is SAPT an effective strategy for improving prompt-tuning with few shots?\n2. Is STSS an effective strategy for improving samples selection for TTA?\n3. How would the method compare with more recent baselines (even those merging recent TTA and prompt-tuning strategies)?\n4. How would the method (and its individual components) compare with adaptations of SAR and SoTTA?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. To my knowledge, this is the first work tackling TTA (and prompt tuning) for vision-language models from the point of view of flat minima. This perspective on the problem is sound, supported by both findings of previous works (e.g., Niu et al. 2023, Gong et al. 2023) as well as the theoretical analysis of Sec. 4, and the achieved experimental results. The paper confirms the potential of this strategy, proposing variants suitable for the increased representation capabilities of VLMs.\n\n2. The strategy for selecting samples based on their sharpness in the loss landscape (STSS), is an interesting contribution, providing an alternative way to the commonly employed entropy (e.g., Shu et al. 2023) and being the element with the highest impact on the performance (Fig. 3). These showcases that flatness priors can guide the model at multiple levels, both during training for prompt tuning and during inference for data filtering.", "weaknesses": "**1.** While FGA is an interesting approach, its two components could be techniques applied independently for prompt-tuning (SAPT) and for sample selection (STSS). Specifically, these two approaches might be treated independently from each other, as SAPT is a strategy for prompt tuning while STSS is a strategy for TTA with sample selection. Thus:\\\n&nbsp;&nbsp;&nbsp;**1.1**  SAPT could be, in principle, compared with CoOp and related variants on prompt-tuning benchmarks (e.g., base vs novel categories, as in Zhou et al. 2022a). This would show the benefit of flatness-aware prompt tuning with strong empirical evidence against prompt tuning alternatives. Similarly, as the method builds on CoOp, adding the flatness-aware regularizer of SAPT on top of other prompt-tuning approaches based on simple contrastive objectives (e.g., Zhou et al. 2022a, Khattak et al. 2023) would showcase that the advantages of flatness-aware prompt tuning are not restricted to CoOp-based tuning.\\\n&nbsp;&nbsp;&nbsp;**1.2** On the other hand, STSS could bring benefits to various approaches performing sample selection for TTA (without considering prompt tuning). In fact, selecting augmentations and aggregating their scores might be a variant for a lot of TTA approaches performing sample selection based on alternative criteria (e.g., entropy-based, as in TPT, Shu et al. 2023, DPE, Zhang et al. 2024a, and ZERO, Farina et al. 2024). This would demonstrate that flatness-aware selection can improve a wide range of methods, strengthening the case for its wider applicability.\n\n**2.** Given that the method is based on both offline prompt tuning and TTA, baselines should comprise three categories of methods: (i) TTA-only strategies, (ii) prompt-tuning only strategies, and (iii) the two merged. While this is the case, the selection of methods for the third category is limited, as, beyond PromptAlign (Abdul Samadh et al. 2024), only TPT and DiffTPT are considered as potential TTA strategies applied to prompt tuning approaches, and not in a consistent manner (e.g., Tab. 1 reports both applied to CoOp, Tab. 2 only TPT on top of CoOp and MaPLe). To fully demonstrate the strength of the approach, more competitive baselines should be built using more recent TTA approaches (e.g., C-TPT, ZERO, [b]) coupled with more recent prompt tuning strategies (e.g., [c,d]), reported in all tables. Moreover, following on the previous point, tables could report the results for SAPT and STSS alone (i.e., coupled with CoOp or alternatives), as prompt-tuning and TTA methods, respectively, ensuring a fair comparison beyond merging solutions for the two problems. \n\n**3.** Another set of strong baselines missing are those related to previous methods proposing flatness-aware optimization for online TTA, i.e.,  SAR (Niu et al. 2023) and SoTTA (Gong et al. 2023). It is worth noting that while the problem they address is different (i.e., online TTA with continuous updates), tables already include comparisons with methods employed in this setup (e.g., TDA, Karmanov et al. 2024, and DPE, Zhang et al. 2024a). Moreover, while it is true that they were not designed for VLMs, an adaptation of these approaches (as done in [a] for SAR) would suffice and strengthen the claim that FGA is the TTA strategy using flatness principles that is most suited for VLMs.\n\n**Minors:**\n\n4. Tables 1 and 2 appear far from the pages where they are described (e.g., Tab. 1 is on page 6 but discussed only on page 8). Reporting the table closer would avoid back and forth between pages when reading the manuscript.\n\n5. In the appendix, the name of the method changes to TLLA.\n\n**References:**\\\n[a] Wang, Zixin, et al. \"Is less more? exploring token condensation as training-free test-time adaptation.\" ICCV 2025.\\\n[b] Zanella, Maxime, and Ismail Ben Ayed. \"On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?\" CVPR 2024.\\\n[c] Yao, Hantao, Rui Zhang, and Changsheng Xu. \"Visual-language prompt tuning with knowledge-guided context optimization.\" CVPR 2023.\\\n[d] Roy, Shuvendu, and Ali Etemad. \"Consistency-guided Prompt Learning for Vision-Language Models.\" ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RVA12a266m", "reviewer": "Reviewer_j4NZ", "summary": "This paper proposes Flatness-Guided Adaptation (FGA) for test-time adaptation of CLIP, leveraging sharpness-aware training to align training and test loss landscapes without updating parameters at test time.It introduces SAPT and STSS to guide adaptation using flatness as a geometric clue, enhancing generalization.", "questions": "See the Weaknesses", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a FGA method that cohesively links training and test-time adaptation through the concept of loss landscape flatness, improving robustness to distribution shifts without updating model parameters during inference.\n2. FGA avoids backpropagation and prompt updates at test time, resulting in lower computational overhead while still improving performance.", "weaknesses": "1. The scope of the paper is mismatched: While the title refers to test-time adaptation (TTA) for ``vision-language models\" broadly, all experiments are limited to CLIP variants (ViT-B/16 and ResNet50). This narrow scope excludes newer or structurally different VLMs such as  LLAVA or CLIP variants such as SigLIP, SigLIP-v2, or other similar models, undermining the generality claim. \n2. The phrase ``training data on downstream task\" is vague. From the experimental section, it seems the prompt tuning is conducted on ImageNet, while evaluation is on ImageNet variants. If ImageNet is treated as the training data, this may conflict with standard TTA setups, where benchmark dataset ImageNet should be unseen during training to preserve test-time integrity.\n3. The method combines sharpness-aware prompt tuning (SAPT) and test-time selection based on sharpness. While effective, this formulation closely relates to ideas explored in prior work like DPO [1], which also applies perturbation-based optimization for TTA, albeit in 3D object detection. A discussion of its differences/similarities would strengthen the positioning of this work.\n4. The experimental comparisons are inconsistent. While Table 1 includes TDA, Table 2 omits it despite being a relevant baseline for cross-dataset generalization. \n5. Recent methods like TCA [2] report higher performance than FGA without requiring augmentations or training overhead. The absence of direct comparison with such state-of-the-art approaches weakens the empirical claims. \n\n[1] DPO: Dual-Perturbation Optimization for Test-time Adaptation in 3D Object Detection \n\n[2] Is Less More? Exploring Token Condensation as Training-free Test-time Adaptation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ro6bhc0VG2", "reviewer": "Reviewer_P1eB", "summary": "This paper addresses test-time adaptation (TTA) for vision-language models (VLMs) such as CLIP. Test-time prompt tuning (TPT), which optimizes text prompts corresponding to classes, is the mainstream approach in this field. TPT optimizes prompts for test samples using surrogate losses like entropy minimization. Since this optimization relies on backpropagation, it incurs significant overhead and substantially increases inference time. To this end, this paper proposes a novel test-time adaptation paradigm called flatness-guided adaptation (FGA). The core idea of FGA consists of two parts: adding regularization during the prompt-tuning phase to flatten the loss landscape, and selecting data augmentation samples during inference based on sharpness scores from the surrogate loss. This enables adapting the model to the test distribution without backpropagation during testing. Theoretically, the paper demonstrates that selecting similar samples via sharpness scores can bring the test distribution closer to the training distribution, potentially improving generalization performance. Experimentally, across standard benchmarks, the paper shows that FGA significantly improves performance compared to existing TPT baselines while substantially reducing inference time.", "questions": "Please respond to the concerns raised in the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **S1.** The paper proposes a novel TTA paradigm that eliminates the need for backpropagation during testing. This is innovative.\n- **S2.** The paper theoretically discusses the relationship between sharpness and generalization performance on the loss landscape, justifying the effectiveness of the proposed method. If this analysis is novel, it is expected to have a significant impact on the field.\n- **S3.** The paper evaluates the performance and inference speed improvements achieved by the proposed method through experiments using standard benchmarks. In particular, the comparison with the baseline combining CoOp and TPT is carefully designed and fair.", "weaknesses": "- **W1.** Section 4's theoretical analysis discusses generalization performance using the loss $l^\\rho$ (e.g., cross-entropy for classification), but the proposed method shown in Section 3 actually evaluates sharpness using the surrogate loss $l_\\text{SRG}$, causing a discrepancy with reality and theory. The paper should add theoretical supplements describing the effects by this gap. Furthermore, practical insights such as the correlation between sharpness computed via cross-entropy using ground-truth labels and sharpness computed via entropy would also be desirable.\n- **W2.** FGA can indeed avoid backpropagation during inference, but the additional training cost due to flatness regularization is significant. While the paper introduces a reasonable approximation calculation via Taylor expansion in Eq. (4), a quantitative discussion is needed on how much training time increases compared to existing prompt-tuning, such as CoOp.\n- **W3.** FGA relies on a training phase with teacher labels. This is a significant practical constraint. The paper primarily trains using 16-shot ImageNet, implying a requirement for 16,000 training samples. Collecting datasets of this scale is a major burden in contexts like medical image processing. In this sense, demonstrating experimental results with fewer shots would significantly strengthen the paper's claims and enhance its value for practitioners, though it already shows \"zero-shot\" performance as \"FGA wo/ SAPT\" in Table 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SkY67c2HyP", "reviewer": "Reviewer_BBAK", "summary": "Most of existing test-time adaption methods isolate the test phase adaption from the training phase, leading to sub-optimal generalization performance. To mitigate this gap, this paper leverages the alignment between the training minimum and test loss flat regions to guide the test-time adaptation process, and proposes a flatness-guided adaptation framework (FGA). The effectiveness of FGA is verified both by theoretical analyses and extensive experiments.", "questions": "* In Theorem 1, what are the definitions of $d_{F\\Delta F}$ and $\\hat{\\ell}^{\\rho}$ ? Does the loss function in the theorem i.e., $\\ell^p()$, equal to $\\ell_{STSS}$ ?\n\n\n* In (5), why are the perturbations sampled from the standard normal distribution? Is it used to approximate the direction of gradient (similar to (4))? If so, why is such an approximation strategy reasonable?\n\n* In Theorem 4, what is the relation between $\\xi$ and $\\beta$ ? In Theorem 1, if $\\delta>\\frac{2}{\\mathrm{e}^2}$, then there is a trivial upper bound on the generalization, i.e., $M$. By Definition 2, we can conclude $\\beta=0$. In this case, is $\\xi=M$ ?\n\n* FGA is independent of the methods that augment test samples. In the experiment, how does FGA augment test samples?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* This paper is well-written and easy to follow.\n\n* The proposed framework FGA is novelty and well-motivated. The experimental results also demonstrate its effectiveness.\n\n* The experimental results are sufficient.", "weaknesses": "There are some concerns on the theoretical analyses.\n\n* The generalization error bound in Theorem 1 is not tight and its effectiveness can be very restricted.\n\nIn Theorem 1, there is a term $M\\sqrt{\\frac{1}{2}\\cdot\\ln\\frac{2}{\\delta}}$ on the upper bound. It seems that this generalization error bound is efficient only for $\\delta>\\frac{2}{\\mathrm{e}^2}$. Otherwise, this upper bound is larger than $M$, a trivial result. \n\n* In Theorem 4, there is a lack of an upper bound on $\\xi$. If $\\xi$ is very large, then it is hard to distinct two test distributions by (12).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)