Preserve and Sculpt: Manifold-Aligned Fine-tuning of Vision-Language Models for Few-Shot Learning
Authors: Dexia Chen
Qianjie Zhu
Weibing Li
Yue Yu
Tong Zhang
Ruixuan Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Vision-Language Model
Few-shot Transfer
Image Classification
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: Transferring vision-language models to visual classification via Manifold-Preserving and Sculpting Tuning (MPS-Tuning).

Abstract: Pretrained vision-language models (VLMs), such as CLIP, have shown remarkable potential in few-shot image classification and led to numerous effective transfer learning strategies. These methods leverage the pretrained knowledge of VLMs to enable effective domain adaptation while mitigating overfitting through parameter-efficient tuning or instance-based consistency constraints. However, such regularizations often neglect the geometric structure of data distribution, which may lead to distortion of the overall semantic representation. To overcome this limitation, we propose a novel fine-tuning method, Manifold-Preserving and Sculpting Tuning (MPS-Tuning). Regarding the data distribution in feature space as a semantic manifold, MPS-Tuning explicitly constrains the intrinsic geometry of this manifold while further sculpting it to enhance class separability. Specifically, MPS-Tuning preserves both macroscopic and microscopic topological structures of the original manifold by aligning Gram matrices of features before and after fine-tuning. Theoretically, this constraint is shown to approximate an upper bound of the Gromov-Wasserstein distance. Furthermore, features from the image and text modalities are paired, and pairwise similarities are optimized to enhance the manifold’s class discriminability. Extensive experiments demonstrate that MPS-Tuning significantly improves model performance while effectively preserving the structure of the semantic manifold.

Published as a conference paper at ICLR 2026
PRESERVE AND SCULPT: MANIFOLD-ALIGNER FINE- TUNING OF VISION-LANGUAGE MODELS FOR FEW- SHOT LEARNING
Dexia Chen', Qianjie Zhu', Weibing Li', Yue Yu², Tong Zhang?; Ruixuan Wangl.2 Sun Yat-sen University, PPeng Cheng Laboratory, 'Guangxi University
ABSTRACT
Pretrained vision-language models (VLMs), such as CLIP, have shown remark- able potential in few-shot image classification and led to numerous effective transfer learning strategies. These methods leverage the pretrained knowledge of VLMs enable effective domain adaptation while mitigating overfitting through parameter-efficient tuning instance-based consistency constraints. However, such regularizations often neglect the geometric structure of data distribution, which may lead to distortion of the overall semantic representation. To overcome this limitation, we propose a novel fine-tuning method, Manifold- Preseoevi and Sculpting Tuning (MPS-Tuning Regarding the data distribution in feature space as a semantic manifold, MPS-Tuning explicitly constrains the intrinsic geometry enhance class separability. Specifi-
of this manifold while further sculpting cally, MPS- TuSS-Tnning preserves both macroscopic and microscopic topological struc- tures of the original manifold by aligning Gram matrices of features before and after fine-tuning. Theoretically, this constraint is shown to approximate an up- per bound of the CromoovWasserstein distance. Furthermore, features from the image and text modalities are paired, and pairwise similarities are optimized to enhance the manifold's 's class discriminability. Extensive experiments demonstrate that MPS- Tuning significantly improves model performance while effectively pre- serving the structure of the semantic manifold.
INTRODUCTION
Vision-language models (VLMs), exemplified by CLIP (Radford et al., 2021), have made significant progress by training on massive image-text pairs using contrastive learning. These models create joint embedding spaces where images and texts with similar meanings are well aligned. For instance, the visual representation of a "cat" becomes positioned near the textual representation of "feline" but far from semantically distant concepts like "truck". This intuitive structure of the embedding space directly contributes to the models' strong generalization across diverse tasks.
However, preserving this intricate semantic structure presents significant challenges during task adaptation, especially in few-shot learning scenarios. Standard fine-tuning approaches tend to semantic structure collapse where limited training samples cause catastrophic forgetting of pre- trained representations, ultimately manifesting as severe degradation in generalization performance.
To address these challenges, two main paradigms of approaches have been proposed (Fig. [T] The first paradigm encompasses parameter-efficeet fine-tuning (PEFT) methods, including prompt- based techniques such as CoOp (Zhou al. (0026) and adapter-based frameworks like CLIP- Adapter (Gao et al.] 2024a), which mitigate overfitting by constraining the number of trainable parameters. The second paradigm comprises consistency-driven approaches, such as Prompt- SRC (Khattak et al. 2023b), which enforce consistency between the features or logits of individual samples before and after fine- tine-tuning. Despite the demonstrated efficacy of these approaches, they either rely on implicit regularization of few wwwparramterr fine-tuning, which limits model flexibility, or
"Corresponding author. Email: wangruix5@ mail.sysu.edu.cmm changt02@pcl.ae.. @pl..ac.c 'The code is available htpp://gthhhh.cmm/aaeeer/////

---

Published as a conference paper at ICLR 2026
photio al (CLASST
By
XXXX XXXXXXXx (CLASS)
Ey
Ill-s.
By
(a) CLIP
(b) PEFT-based
Partal Trainable
photio of (CLAGST
Ey
"w photo of (CLASS)'
E
Origincl Fecture
Fine Taned Festure
By
(c) Point-wise Consistency
(d) Manifold Regularization (Ours)
Figure 1: Comparison of regularization paradigms for VLMs. Previous fine-tuning methods (b, c) primarily constrain the number of tunable parameters or apply point-wise consistency constraints, potentially limiting the model's learning capacity while neglecting the complete semantic manifold structure. In contrast, our method explicitly preserves the semantic manifold structure, significantly enhancing both generalization and learning capabilities.
restrict variations in individual sample representations, which neglects the preservation of pretrained model's semantic structure.
In contrast existing methods that treat image data isolated points, propose Manifold- Preserving and Sculpting Tuning (MPS-Ture Tuning), which views data distribution in the feature spaces as continuous semantic manifold, and aims enhance its discrimination for downstream tasks while maintaining the intrinsic manifold structure. To preserve the manifold structure, we constrain the Wassoovssssnrttinn (GW) distance (Mémoli 2011) between the semantic manifolds derived from the features distributions of the fine-tuned and original models during training. Since directly com- puting GW distance is NP-hard and impractical for optimization, we simplify this problem and the- oretically prove that the Lp-norm of the difference between corresponding Gram matrices provides an upper bound approximation to the GW distance of order p. Based on this theoretical insight, propose Manifold Alignment Regularization, which preserves global topological structure via batch-level Gram matrices and maintains local geometric structure through token-level Gram ma- trices. For manifold seulpting, introduce Hierarchical Manifold Sculpting with multimodal query-support matching task, where each query achieves higher similarity with same-category pairs and lower similarity with different-categon pairs. This sculpting mechanism is extended from the model's 's output features to its intermediate layer features, further enhancing the discrimination of the manifold. Through manifold alignment and sculpting, robust adaptation vision-lant laiiuul models is effectively achieved.
Our main contributions are summarized as follows:
We propose a novel few-shot fine-tuning framework called MPS-Tuning, which enhances model performance while alleviating overfitting by explicitly aligning and sculpting the manifold geometry.
We design new regularization method, called Manifold Alignment Regularization, and establish its theoretical connection to the Gromov-Wasserse distance for the first time, offering deeper insights into the preservation of manifold geometry.
We introduce an optimization strategy called Hierarchical Manifold Sculpting to actively enhance the discriminability of the manifold and further improve model performance.
We evaluated the method's performance on 11 datasets and conducted generalization eval- uation on two datasets. Experimental results demonstrate that our method significantly outperforms current state-of-the-ar approaches in few-shot image classification tasks.
RELATED WORK
2.1
VISION-LANGUAGE MODELS
In recent years, vision- languaee models (Radford et al.
2021; Sun et al.
2023
Xu et al. Pal al.
2024 2025
Gao et al. 2024b Huang et al. 2024 Zhai et al. 2023 Tschannen al.
2025
2

---

Published as a conference paper at ICLR 2026
pretrained via contrastive learning large-scale image-text pairs have demonstrated strong zero- shot generalization capabilitiees by aligning correctly matched images and texts. This enables them to be directly applied to various downstream tasks. Typically, such models consist of image encoder Fy and a text encoder Fr- For a given classification task, text prompts (e.g.. "a photo of a {class)") first constructed for each class, and set of normalized class-specific textual features {t, .1.9 tx is extracted using the text encoder ET. where K denotes the number of classes. Subsequently, for an input image X, its normalized visual feature representation is obtained through the image encoder Ev. The probability of the image belonging to each class is then computed by applying the softmax function to the cosine similarities between and each class text feature f2 i.e.,
exp( (x)
Ply = Cx
Σラn1 expl fj)/T)*
where T is a learnable temperature coefficient, and (-.-) represents the inner product between two vectors. The model's final prediction is given by the class with the highest probability, i.c.,
ÿ = arg max of ((1.....))) P(y Ck cx(x)
(2)
2.2 EFFICIENT TRANSFER LEARNING
To adapt VLMs efficiently while mitigating overfitting, two main transfer learning paradigms have been explored (Yang et al. 2024 Zhu et al. 2023 Xie et al. [202]] khattak et al. 2025; Guo et al., 2023 Zhu et al.j 2024). The first comprises PhFi-hased methods that reduce computational cost through selective parameter updates, such as prompt-based approaches (e.g.. CoOp (Zhou et al., 2022b), CoCoOp (Zhou et al., 2022al) and adapter-based designs (e.g., CLIP-Adapter (Gao et al,, 2024a), Tip-Adapter (Zhang al. 2022 ). The second paradigm emphasizes consistency regular- ization, exemplified by PromptSRC Khattak et al., 2023b), which applies feature-level L1 loss and logit-level KL divergence (Kullback & Leibler 1951) to retain pre-trained knowledge. While these methods demonstrate impressive performance on few-shot tasks, their flexibility is constrained by dependencies PEFT strategies and point-based constraints. In contrast, our manifold alignment regularization enhances adaptability while better preserving pre-trained knowledge.
2.3
GROMOV-WASSERSTEE DISTANCE
compares the intrinsic geometric struc- tures of two metric measure spaces by matching their internal pairwise distance relations rather than individual points. Consider two discrete metric spaces represented by their pairwise distance matri- Let /1 E A, and E Am be probability vectors supported on
The wassertWssen (GW) distance (Mémoli 2011
ces Dx E R and Dy E m these spaces, where An {u E R > u, = 1}, then the set of couplings between μ and v is
[p, v) { R mm T1m = p. π 1n = =/}
(3)
where THe represents the joint probability mass assigned to the pair (I,, yk). and 1m E RR",, 1, E R vectors of ones.
For 2 1, the discrete GW distance is defined as
1/p
GW,(kV)=
min xEIl(A.))
Σ Σ +J= k.1=1
(Dx)i - IDxx)
(Dy)all
(4)
The GW objective seeks an optimal coupling क that minimizes the expected p-power discrepancy between pairwise distances in the two spaces. By operating on the relational information encoded in the distance matrices, rather than the coordinates of individual points, this objective naturally becomes invariant to isometric relabelings. This invariance is precisely what enables GW distance to robustly capture and compare intrinsic geometric structures. However, finding is significant bottleneck, as it requires solving nonconvex quadratic program reducible to the ha-hard quadratic assignment problem, making it intractable for large-scale applications. To address this, we adopt fixed coupling scheme, which provides a tractable upper bound on the GW distance and transforms it into computationally feasible regularization. To the best of our knowledge, this work represents the first application of GW distance theory to VLM fine-tuning, offering a principled approach to preserving geometric knowledge in pretrained models. More details in Sec.
3

---

Published as a conference paper at ICLR 2026
Manifold Alignment Regularization
ALL tokens
Gram Matrix
Gram Matrix
tokens
Eooat
Gram
gloobal
Ev
Ev
logits
Lce
logits
[CTS] token
Text Embeddings
[CLS] taken
Pseudo Forward
Hierarchical Manifold Sculpting
Support Set
Sculpt
X
Matches
Query Set
Frozen
Partial Trainable
[CLS] token
patch token
Dot Product
Figure 2: Overview of the MPS-Tuning, which integrates Manifold Alignment Regularization and Hierarchical Manifold Sculpting. Manifold Alignment Regularization prevents knowledge degrada- tion by aligning Gram matrices across fine-tuned and original CLIPs at both global and local scales. Hicrarchical Manifold Sculpting enhances local manifold adaptability via query-suppor matching, tailoring representations to downstream tasks. Through Pseudo Forward, this sculpting process ex- tends to intermediate layers, ensuring effective manifold refinement. Ey denotes the visual encoder.
3 METHOD
To facilitate effective downstream adaptation without disrupting the inherent structure of the pre- trained representation manifold, we propose a novel approach termed Manifold-Presservin and Sculpting Tuning (MPS-Tuning). This method employs Manifold Alignment Regularization (MAR) to prevent drastic alterations in the semantic structure of feature manifold and incorporates Hierar- chical Manifold Sculpting (HMS) progressively refine local manifold structures. Specifically, MAR aligns the Gram matrices of the fine-tuned and the original models at both the batch and token levels, thereby maintaining consistency semantic geometry and mitigating overfitting risks. In parallel, HMS refines local manifold structures by performing a multimodal query-suppon match- ing task between image and text representations, optimizing similarity at both intermediate and output feature levels. This results in more compact intra-class clusters and better separated inter- class distributions. By jointly applying MAR and HMS, MPS-Tuning achieves robust and efficient adaptation to tasks, maintaining the valuable structural knowledge of the pre-tree pre-trained model and demonstrating strong performance in few-shot learning scenarios.
3.1 MANIFOLD ALIGNMENT REGULARIZATION
The feature distribution learned by pre-trained model can be regarded well-structured seman- tie manifold, whose geometric structure encodes rich prior knowledge. Preserving this geometric structure allows for the retention of more comprehensive pre-trained knowledge. To this end, pro- pose Manifold Alignment Regularization (MAR), which enforces alignment between the geometric structures of feature manifolds before and after fine-tuning, thereby enhancing model performance.
As a metric designed to quantify the similarity between different metric spaces, the GW distance serves powerful tool for evaluating changes in the structure of feature manifolds induced by model fine-tuning. Our MAR provides an efficient upper-bound approximation to the GW distance. To formally justify this approximation, we present the following theorem:
Theorem The alignment of the Gram matrices under the L, norm serves as an approximate upper bound of the p-order Gromov- Wasserstein distance.
4

---

Published as a conference paper at ICLR 2026
Attention
Parameter Sharing
P/raa
MatMMul
MatMui
Add & Norm
FFN
Outputs
Inputs
Kpray
Pseudo Outputs
Vprof
Add & Norm
FFN
Pseudo Forward
Figure 3: Pscudo-Forward projection bypass the attention allocation component in the model, and map intermediate layer features to the output feature space.
Proof Outline. We consider the feature spaces of the original CLIP model and the fine-tuned CLIP model as two metric spaces. By fixing a natural coupling (i.e., assuming a one-to-one correspon- dence between features of the same sample across the models), the NP-hard computation of the GW distance is reduced to an efficient upper-bound approximation. Specifically, the approximate upper bound of the discrete GWp distance between the two metric spaces is given by the -nrmo of the difference between their respective Gram matrices. Refer to B for more details.
Guided by this theory, alignment regularizations are introduced at two distinct levels.
Global topologiaal alignment To preserve the global manifold structure, relational constraints among samples are enforced at the batch level. Given mini-batch of N samples, we extract nor- malized [CLS] token features from the pre-trained model as (Zi..... ZNv E R R.Nxd and from the fine-tuned model as {z' ...,, zz } E Rg-xd The Gram matrices S, S' E N are computed via inner products Sij- (Zi, 2,) and 1/ (z';,z')) z')) The global alignment loss is defined as
N
N
MAR
Si 54yh1
(5)
Local geometric alignment To retain the internal geometric structure within individual samples, regularization is separately performed on the interactions between the [CLS] token and the patch tokens, well on the internal relations among the patch tokens. For the i-th sample, we Ss 21 ..... ZM E {(M+1)xd and after fine-tuning as collect features before fine-tuning as {zess
(z((1 z((0) (M E RMM+1)xd where M denotes the number of patch tokens. The intra- sample Gram matrices Simtra S E ( (M++1)x(M1I) are computed using inner products among (CL.S) and patch tokens. The local alignment loss is given by
Clocal MAR
1
Σ
1
ΣΣ
N
(M (M+
i=1
1=
The final manifold alignment regularization term is the sum of the above two terms, i.c.,
LMAR
cglohal
MAR + MAR
clocal
3.2 HIERARCHICAL MANIFOLD SCULPTING
To facilitate the acquisition of task-specific knowledge, we propose hierarchical optimization of the local feature manifold. This process is formulated as a query-support matching task, which aims encourage high similarity for positive image-text matches. Let @ {qi, q2, qND } be the set of normalized image features used as queries, and let T (t, .2..... ... tx] be the of frozen text embeddings. The support set is then defined as the union S = QUT. Positive matches are defined based on category identity. Since limited batch sizes may lead missing visual positives, data augmentation applied to generate two augmented views per image, enriching the image pool. The task is optimized via sculpting loss, which applies contrastive learning between each query and its positive matches:
image-image pairs and discourage incorrect
1
exp((g.s))/
L souppe (q. S)
log
(8)
Pa
Ex Ex'S exp((q.s/))/7'
sEPa
5

---

Published as a conference paper at ICLR 2026
Caltect222
ColoerdPets
HirCArcrat
Figure 4: Performance comparison on benchmark datasets.
where 7' is temperature factor and Pg represents the set of samples from that are positively paired with the query q. The final objective over the batch is obtained by averaging this loss across all queries:
Esculpe S) Eg o [query (q.
Furthermore, the manifold refinement is extended to intermediate transformer blocks to further sculpt the manifold. However, intermediate features z(()) are not compatible with text embeddings. To address this issue, we implement a pseudo-forward projection (Fig. 3). skipping attention mod- ules while retaining essential transformations in model, i.e..
2 (1)
(
v FFN(+11
Proj (2(10)
(10)
The overall HMS loss thus aggregates both final and intermediate layer alignment as follows,
LuMS Escuipt (Q..))
Σ TETana
Esculpe (1)) (
(11)
where é, S are the output query and support sets, Q00 (() are their counterparts at layer / after pseudo forward projection, and Llookke denotes the layer scope of HMS.
3.3 TRAINING AND INFERENCE
Leveraging the MPS-Tuning's strong knowledge retention capability, we can fine-tune partial model weights directly without causing overfitting, thereby substantially enhancing the model's learning capacity. See Sec. for details.
During training and inference, the final logits are weighted sum of fine-tuned and original outputs:
logits logits, + (1 - -a) logits
(12)
The overall training loss incorporates the cross-entropy term alongside regularization terms:
L Les Xι LMAR AzLums
(13)
where D and X2 are hyperparameters that balance the contributions of the regularization terms.

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
2022b Yu et al. 2023). we evaluated our method on 11 datasets, including ImageNet (Deng al. 2009). Caltech (Fei-Fei et al. 2007). Food101 (Bossard et al. (2014), DTD (Cimpoi et al. 2014), EuroSAT AT (Helber et al.) 2019), FGV- CAircraft (Maji et 2013), Flowers (Nilsback & Zisserman 2008), OxfordPets (Parkhi al, 2012), StanfordCars (Kaause et al., 2013), SUN397 (Xiao et al., 201O), and UCF101 (Soomro et al.,
Datasets Following previous work Zhou et al.
2012 Additionally, domain generalization capabilities were further assessed using the ImageNet- 2019) datasets.
Sketch (Wang al. 2019) and ImageNet-V2 (Recht et al.
Implementation Following previous work (Zhou et al. 2022b; Yu et al., 2023), the model was trained on K-shot settings (K = 1, 2, 4, 8, 16) and evaluated on the full test set using CLIP with ViT-B/16 (Dosovitskiy et al. 2021 y and predefined text templates. Optimization was performed using SGD with cosine learning rate decay over 50 epochs, where warm-up strategy increased the learning rate linearly from le-5 to 0.002 during the first epoch. Data augmentation strategies consistent with those used in CoOp, including random cropping and random flipping, were applied. For hyperparameter configuration, the weights for MAR (A1), HMS (A2) and logits weight (a) were 0.5, 0.1 and 0.3, respectively, with HMS applied to the last two layers. All results were averaged over three random seeds.
Baselines To demonstrate the superiority of our method, comprehensive comparisons were con- ducted against several SOTA methods, including CoOp (Zhou et al. 20226). Tip-Adapter-F (Zhang
al., 2022), PLOT++ (Chen al., (023). MaPle (Khattak et al., 2023a), PromptSRC (Khattak et al.
2023b), AMU-Tuning (Tang et al. 2024). TCP (Yao et al. 2024), DePT (Zhang et al. 2022)).
GalLop
(Lafon et al., 2024), TextRefiner (iie et al., (024), MMRL. (Guo & Gu 2025), SkipT (Wu
cet al.
2025). LDC (Li al. 2025). and TAC (Hao et al. 2025).
4.2 EFFICACY STUDY
Classification Results As shown in Fig. our method achieves superior average performance across all shot settings compared to competing methods, with the performance gap widening as the number of training samples increases. Specifically, under the 1-shot, 4-shot, and 16-shot condi- tions, our approach improves accuracy by 0.88%, I 1.27%, and 2.51 %, respectively, over the strongest baseline. On natural image datasets such as ImageNet and SUN397, where the pre-trained CLIP model has already encoded rich visual knowledge, MAR facilitates the integration of downstream task learning while preserving this prior knowledge, leading to significant performance gains. For datasets with greater cross-domain challenges, such StanfordCars, FGVCAircraft, and UCF101, the synergistic operation of HMS and MAR enables our method to effectively balance novel knowl- edge acquisition with pre-trained knowledge retention, yielding significant performance advantages.
Domain Generalization Results To validate the robustness of MPS-Tuning, models were trained on the ImageNet dataset and subsequently evaluated on both ImageNet-V2 and ImageNet-Sketch datasets. As demonstrated in Tab. [ MPS-Tuning consistently outperforms all baseline methods across the three datasets, thereby confirming its superior domain generalization capabilities.
Efficiency The training and inference FPS of MPS-Tuning were evaluated on the SUN397 dataset on single RTX 3090 GPU to assess its efficiency. As shown in Tab. 3 the method demonstrates comparable efficiency to existing approaches.
4.3 ABLATION STUDY
Impact of Model Components To establish the individual contribution of each proposed module, ablation studies were conducted on the ImageNet, StanfordCars, and SUN397 datasets under 16- shot settings. Tab. iillustrasss that both modules contribute meaningful performance gains over the standard cross-entropy loss when applied separately, while their joint application achieves additional improvements.
Impact of HMS depth The application scope of HMS was further explored. As shown in Fig. 5
applying HMS to final layer and penultimate layer yields the best performance gain,
7

---

Published as a conference paper at ICLR 2026
Table 2: Ablation study on different compo- nents, "Avgl1" is the average over all 11 bench- mark datasets.
Table 1: Generalization results on ImageNet and its variants.
Source ImageNet
Target
Compunents
Method
Avg
-kketch
Los CMAR
Cuss
ImageNet
Cars
SUN397
Avg!1
60.83
$7.90
90.00
76.30
Linear Probe CLIP
65.85
34.77
56.26
52.29
74.77
75.30
90.77
90.80
77.80
78.07
86.20
86.44
CoOp
71.92
46.71
64.18
60,94
PLOT+
72.48
47.13
65.07
61.56
MaPLe
72.56
49.20
64.10
61.95
PrompeSCC
73.17
49.10
65.70
62.66
Table 3: Efficiency comparison on SUN397.
TCP
72.40
48.17
64.83
61,80
DePT
73.35
46.43
64.63
61.47
AMU-Tuning
50.37
65.42
63.57
Method
Training FPS
Inference FPS
MMRL
TextRefiner
70.90
72.03
48.07
49.17
63.37
64.47
61.89
60.78
CLIPP
617
SkipT
72.77
49.73
65.67
62.72
CoCoOp
4.20
13.0
LDC
73.88
48.85
10
62.94
TCP
120.9
617
TAC
66.23
62.94
TextRefiner
85,30
553
MPS-Tuning (Ours)
75.60
50.10
67.53
64.41
MPS-Tuning (Ours)
95.65
535
Table 4: Ablation study on MAR. "Avgl1" is the average over benchmark datasets.
I-shot
4-shot
16-shot
Method
ImageNet
UCF101
Avg11
ImageNet
UCF101
Avgi1
ImageNet
UCF101
Avgl1
None
69.33
76.73
72.35
71.57
83.63
79.60
74.77
89.23
86.20
only Global
70.03
77.37
73.42
72.23
83.90
80.18
75.17
89.53
86.57
only Local
69.90
76.73
72.82
72.43
84.27
80.21
75.57
89.60
86.67
MAR Looal)l)Lccal)
70.37
77.40
73,55
72.57
84.30
80.47
75.60
89.87
86.85
Table 5: Ablation study of different consistency constraints. "Avgl1" is the average over all
benchmark datasets.
1-shot
4-shot
16-shot
Consistency Construint
ImageNet
UCF101
Avg11
ImageNet
UCF101
Avg11
ImageNet
UCF101
Avg11
69.73
75.37
72.77
71.83
82.63
79.36
74,73
88.20
86.07
Feature-based
&
69.43
76.73
72.41
71.63
83.63
79.66
74.83
89.23
86
2
69.33
72.35
71.63
83.60
79.65
74.77
89.17
86.20
kl
69.67
73.00
78.70
77.77
73.87
84.97
84.20
Logits based
f1
69.50
75.00
71.91
71.50
81.03
78.22
74.67
87.67
85.65
& 69.47
74.93
71.85
71.57
81.10
77.51
74.70
87.73
85.59
Manifold- based
CMAR
89.87
86.85
while extending to carlier layers results in degradation. This finding aligns with the seman- tie hierarchy neural networks (Zeiler) [014
86.9
66.86
Aacanacy (%))
96 74
Krizhevsky al.] 2012; Gandelsman 2024): deeper layers encode class-specific fea-
al.
86.77
tures that benefit from HMS, whereas intermedi- ate layers capture more generic representations. Applying HMS too early may induce premature specialization, leading to overfitting. Therefore, HMS used in the last two layers enhance high-level representation learning while main- taining generalization.
HMS
Figure 5: Ablation study on HMS depth. The x- axis N means applying HMS the last N layers.
Impact of Global and Local Alignment To assess the roles of global topological and local geomet- ric alignment, we perform ablation studies under 1-shot, 4-shot, and 16-shot settings. As shown in Tab. both alignments consistently improve performance, but exhibit different preferences across sample sizes. Global alignment is more beneficial low-shot scenarios, while local alignment be- comes more effective as sample size increases. This is because global alignment offers essential
8

---

Published as a conference paper at ICLR 2026
Table 6: Performance of distinct tuning methods equipped with vs. without MPS-Tuning. "PFT"
indicates our default partial fine-tuning, and "FFT" stands for full fine-tuning.
Orivelkkmm
S.N097
EITAMMSS Traing
06:03
10197
81:33
04:13
11.33
87:07
14:07
61.0
16:50
13:13
01.77
61.83
L.RA
$1.77
70.19
94.30
11.33
17.33
83:23
23.35
0000
19.47
12.30
23.33
25.17
08-40
18.00
16,37
16.58
20.00
13.55
PITTMPSS Traing
06:11
00.85
01:03
03
50-10
0345
33.0
61.0
61:01
06.90
05.30
1.8 AASAAA Tlrnag
11.53
06.75
03.35
13.00
09.00
87
11.3
7758
63:40
50.0
03.00
18
IP/TAMPSTTONN Toaing
12.99
16.90
13.97
11.17
56.68
$7.07
11.97
76.53
08.58
$5.90
04.30
18.47
68.33
1007
$5.30
03.55
13:47
12.30
23.87
66.00
15.10
00.0
68.00
05.35
15:4
13.00
17:17
24.43
11.00
18:43
57.87
04.30
7687
1333
03.00
1130
13.13
IPTONIS Toiing
15.48
97.37
14.77
91.33
99.37
18.35
75.47
11.38
16.37
0938
$5.35
relational priors when the model lacks enough data to capture inter-class structure, whereas local alignment enhances robustness by preventing shorteut learning from incidental factors when more data is available. Combining both leads to the best performance, confirming their complementary nature at different structural levels.
Comparison to other Consistency Constraints The effectiveness of MAR was further validated through performance comparisons with standard point-wise consistency constraints applied at fea- ture and logit levels. As shown in Tab. []] under I-shot, 4-shot, and 16-shot settings, our method outperformed the strongest baselines by 0.64%, 0.74%, and 0.77% ImageNet, and by 0.78%, 0.81 %6 and 0.61% % when averaged across all 11 datasets. These results confirm that manifold-baser consistency regularization yields superior performance improvements compared to point-alignment- based consistency constraints.
Applicability Study We evaluate the versatility of MPS-Tuning by integrating it into both full- parameter fine-tuning and LoRA (Hu et al. 2222)) As shown in Tab. 6, our method yields consistent performance improvements across these strategies, validating its broad applicability.
142
6K-
Scoee
15A
02-
suila
uchio
Imagrnal
SuNbor
uchie
Figure 6: Representation similarity analysis of Figure 7: Class separability assessment of
different consistency constraint mechanisms.
different component categories using Calinski- Harabasz score.
4.4 INTERPRETABILITY STUDY
Quantitative Analysis of Manifold Preservation To further validate MAR's manifold preservation capabilities, representational similarity analysis (Kriegeskorte et al.) 2008) applied to models before and after fine-tuning, enabling quantitative evaluation of semantic manifold structural vari- ations. Fig. 6 shows that MAR achieves the best manifold structure preservation on natural image datasets like ImageNet and SUN397, where pretrained CLIP already contains sufficient knowledge. Furthermore, we find that MAR's preservation capability spontaneously diminishes on datasets with extensive novel tesk-relevant knowledge (UCF101,, FGVCAircraft). This adaptive behavior results from MAR's relaxed Gram matrix alignment rather than rigid feature or logits constraints, enabling knowledge preservation when beneficial while allowing adaptation when necessary. Additionally, we find that cosine similarity and KL divergence impose stricter manifold structure preservation compared to other point-based consistency constraints through their holistic vector-level restric- tions, whereas L1 and L.2 losses independently constrain each channel's variation, potentially caus- ing minimal per- per-channel changes but substantial overall vector modifications. However, their exces- sive strictness can limit learning (c.g.. UCF101 and Avgll I results in Tab, (5), whereas MAR's Gram matrix consistency provides dynamic constraints that balance preservation with learning capability.

---

Published as a conference paper at ICLR 2026
Quantitative Evaluation of Class Separation To assess feature clustering behavior, apply the Calinski-Harn Harabaszz Index (Calinski et al. 1974) on test sets from ImageNet, SUN397, UCF101 and FGVCAircraft. As shown in Fig. 7 both HMS and MAR individually improve clustering perfor- mance compared to cross-entropy loss, with HMS performing particularly well on cross-domain dataset UCF101. The combination of HMS and MAR yields similar scores those of HMS alone, suggesting that our method effectively improves feature separability.
ImageNet
FGVCAircraft
SUN397
UCF101
0L04
0.00
Esoch
Epoch
Epoch
Esoch
Figure 8: Comparison of GW distance: Natural coupling upper bound vs. POT numerical solution.
4.5 TIGHTNESS ANALYSIS
We approximate the upper bound of the p-th order GW distance via a fixed natural coupling. Its tightness is validated against exact GW distances calculated using POT (Flamary et al. 2021 2024) on natural (ImageNet, SUN397) and cross- coos-domain (FGVCAircraft, UCF101) datasets, with negligi- ble divergence (Fig. ( confirming its reliability and precision.
0.0025
0.0069
S
S
Figure 9: GW distances across manifold pairs.
DISCUSSION
5.1
WASSERSSNNNN DISTANCE AND MANIFOLD ALIGNMENT
By minimizing intra-space distance discrepancies, the Gromov- Wasserstein distance aligns metric measure spaces while ensuring invariance to rigid transformations and sampling. We verify this using an S-shaped manifold AA alongside its isometric variants A', noisy perturbations Aperts and topologically distinct rings B. Results in Fig- D bustness GW metrics, highlighting their capacity to distinguish topological discrepancies.
Gromov-Wasserstén
corroborate the isometric invariance and noise
5.2 LIMITATION
While approach demonstrates robust performance gains across the majority of scenarios, it falls short of SOTA results on specific 1-shot and 2-shot datasets. This could be attributed to the Gram matrix alignment strategy in MAR which struggles capture the semantic manifold structure under extreme data scarcity. Future research could address this by incorporating unlabeled external data or employing generative models for data augmentation to enhance manifold structure preservation.
CONCLUSION
In this work, propose MPS-Tuning a manifold alignment-based fine-tuning framework that pre- serves the intrinsic structure of pre-trained models via Manifold Alignment Regularization, while enhancing the discriminability of semantic manifolds through Hierarchical Manifold Sculpting, We theoretically show that Manifold Alignment Regularization provides an approximate upper bound of the Gromov- Wasserstein distance, establishing its theoretical soundness. Extensive experiments demonstrate consistent improvements in few-shot VLM performance, positioning MPS- MUPS-Tuning as a promising paradigm for advancing fine-tuning methodologies across diverse domains.
Kromov-Wasserstenn
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported in part by the National Natural Science Foundation of China (grant No. 62571559), the Major Key Project of PCL (grant No. PCL2025AS209), the Guangdong Excel- lent Youth Team Program (grant No. 2023B (5144055))40025) the Guangdong Provincial Science and Technology Program (grant No. 2025A0505080012), and the Guangzhou Basic and Applied Basic Research Foundation (grant No. 2025A0415280).
ETHICS STATEMENT
This research adheres to the ICLR Code of Ethics. We utilize pre-trained vision-language models (e.g., CLIP) and are aware that such models may learn societal biases from their web-scale training data. Our work is a fundamental algorithmic study improving few-shot learning performance, with experiments conducted solely on public academic datasets, involving no sensitive data or hu- man subjects. We therefore consider the direct ethical risks of this research to be low. Nevertheless, we urge any researchers or developers applying this technology to real-world applications to be vigilant and proactively address potential issues of bias and faimess inherited from the foundation models.
REPRODUCIBILITY STATEMENT
To support the reproducibility of this work, we provide comprehensive theoretical proofs and ex- perimental details in the appendix. Detailed mathematical proofs for all theoretical claims in the paper can be found in Sec. [B] All key information required to reproduce our experimental results, including detailed descriptions of the datasets (Sec. E and complete hyperparameter configurations (Sec. C is also provided. We plan to publicly release our source code upon publication.
REFERENCES
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-100 101 mining discriminative com- ponents with random forests. ECCV, 2014.
Calinski, T., Harabasz, and J. A dendrite method for cluster analysis. Comm in Stats Simulation & Comp, 1974.
Guangyi Chen, Weiran Yao, Xiangehen Song, Xinyue Li, Yongming Rao, and Kun Zhang. PLOT: prompt learning with optimal transport for vision-language models. In ICLR, 2023.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Li Fei-Fei, Robert Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Comput. Vis. Image Underst., 106(1):59-70. 2007.
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanis- las Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Foumner, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy. levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica Sutherland, Romain Tavenard, Alexander Tong. and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22:1-8, 2021.
11

---

Published as a conference paper at ICLR 2026
Rémi Flamary, Cédric Vincent-Cuare Nicolas Courty, Alexandre Gramfort, Oleksii Kachaiev, Huy Quang Tran, Laurène David, Clément Bonet, Nathan Cassereau, Théo Gnassounou, Eloi Tanguy, Julie Delon, Antoine Collas, Sonia Mazelet, Lactitia Chapel, Tanguy Kerdoncuff, Xizheng Yu, Matthew Feickert, Paul Krzakala, Tianlin Liu, and Eduardo Fernandes Montesuma. Pot python optimal transport (version 0.9.5), 2024. httpsts:::/ t/tb. wtt.ttthaa..ooccccoooo//oo/a/// wttpa///yyhoooooooTooo
Yossi Gandelsman, Alexei Efros, and Jacob Steinhardt. Interpreting CLIP's image representation via text-based decomposition. In ICI.R, 2024.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better rision-language models with feature adapters. International Journal of Computer Viston, 132(2):581 1-955, 2024a.
Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu, Enwei Zhang, Ke Li, Jie Yang. Wei Liu, and Xing Sun. Softclip: Softer cross-modal alignment makes clip stronger. In AAAI, 2024b.
Yuncheng Guo and Xiaodong Gu. Mmrl: Multi-modal representation learning for vision-language models. In CVPR, 2025.
Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, and Bin Cui. CALIP: zero-shot enhancement of CLIP with parameter-free attention. In AAAI, 2023.
Fusheng Hao, Fengxiang He, Fuxiang Wu, Tichao Wang, Chengqun Song. and Jun Cheng, Task- aware clustering for prompting vision-language models. In CVPR. 2025.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs Remote. Sens., 12(7):2217-222 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Yufeng Huang, Jiji Tang, Zhuo Chen, Rongsheng Zhang, Xinfeng Zhang. Weijie Chen, Zeng Zhao, Zhou Zhao, Tangjie Lv, Zhipeng Hu, and Wen Zhang. Structure-clip Towards scene graph knowledge to enhance multi-modal structured representations. In AAAI, 2024.
Muhammad Uzair Khattak, Hanoona Abdul Rasheed, Muhammad Maaz, Salman H. Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, 2023a.
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsual Yang, and Fahad Shahbaz Khan. Self-regulatnng prompts: Foundational model adaptation without forgetting. In ICCV, 2023b.
Muhammad Uzair khattak, Muhammad Ferjad, Nascer Muzzamal, Lue Van Gool, and Federico Tombari. Learning to prompt with text only supervision for vision-language models. In AAAI, 2025.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV, 2013.
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis -connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2008.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurIPS, 2012.
Solomon Kullback and Richard A Leibler. On information and sufficiency. Annals of Mathematical Statistic, 22(1): 1951.
Mare Lafon, Elias Ramzi, Clément Rambour, Nicolas Audebert, and Nicolas Thome. Gallop: Learn- ing global and local prompts for vision-language models. In ECCV, 2024.
Shuo Li, Fang Liu, Zehua Hao, Xinyi Wang, Lingling Li, Xu Liu, Puhua Chen, and Wenping Ma. Logits deconfusion with clip for few-shot learning. In CVPR, pp. 1-41122544,, 2025.
12

---

Published as a conference paper at ICLR 2026
Subhramsu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. CoRR, abs/1306.55l 2013. URL http://arxiv.org
5151
Facundo Mémoli.
Cromov-wassersee distances and the metric approach to object matching. Foun- dations of compulational mathematics, 2011.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large num- ber of classes. In Sixth Indian Conference on Computer Vision, Graphics & Image Processing, ICVGIP 2008, Bhubaneswar, India, 16-19 December 2008, pp. 722-729, 2008
Avik Pal, Max van Spengler, Guido Maria D' D'Amely di Melendugno, Alessandro Flaborea, Fabio Galasso, and Pascal Mettes. Compositional entailment learning for hyperbolic vision-language models. In ICLR, 2025.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCFI01: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1212.0402, 2012. URL http::/arxxx:
org/aba htg//bs/// 12111111222222 0402
Quan Sun, Yuxin Fang. Ledell YU Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale. ArXiv, 2023.
Yuwei Tang, Zhenyi Lin, Qilong Wang. Pengfei Zhu, and Qinghua Hu. Amu-tuning: Effective logit bias for clip-based few-shot learning. In CVPR, 2024.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint wrXiv22502.14788 2025.
Haohan Wang, songwei Ge, Zachary C. Lipton, and Eric P. Xing. Learning robust global represen- tations by penalizing local predictive power. In NeurIPS, 2019.
Shihan Wu, Ji Zhang, Pengpeng Zeng, Lianli Gao, Jingkuan Song, and Heng Tao Shen. Skip tuning: Pre-trained vision-language models are effective and efficient adapters themselves. In CVPR, 2025.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
Jingjing Xie, Yuxin Zhang, Jun Peng, Zhaohong Huang, and Liujuan Cao. Textrefiner: Inter- nal visual feature as efficient refiner for vision-language models prompt tuning. arXiv preprint arXiv:2412.08870 2024.
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024.
Lingxiao Yang, Ru- Yuan Zhang, Yanchen Wang, and Xiaohua Xie. Mma: Multi-modal adapter for vision- Laniuage models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, 2024.
13

---

Published as a conference paper at ICLR 2026
Hantao Yao, Rui Zhang, and Changsheng Xu. Tep: Textual-base based class-aware prompt tuning for lisual-eanguage model. In CVPR, 2024.
Tao Yu, Zhihe Lu,, Xin Jin, Zhibo Chen, and Xinchao Wang. Task residual for tuning rision-language models. In CVPR, 2023.
MD Zeiler. Visualizing and understanding convolutional networks. In ECCV, 2014.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023.
Zhang, Shihan Wu, Lianli Gao, Heng Tao Shen, and Jingkuan Song. Dept: Decoupled prompt tun- ing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
Renrui Zhang, Wei Zhang, Rongyao Fang. Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training free adaption of clip for few-shot classification. In ECCV, 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for lision-language models. In CVPR, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-2344 2022b.
Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In ICCV, 2023.
Yuhan Zhu, Yuyang Ji, Zhiyu Zhao, Gangshan Wu, and Limin Wang. Awt: Transferring vision-language models via augmentation, weighting, and transportation. arXiv preprint 2AXV::2007.04603. 2024.
A STATEMENT ON THE USE OF LARGE LANGUAGE MODELS (LLMS)
In the preparation of this paper, utilized Large Language Models (LLMs), including ChatGPT, Gemini, Claude, Qwen, etc., as general-purpos assistive tools. Their primary use was for improving language expression, correcting grammatical errors, and polishing certain paragraphs enhance the overall readability of the manuscript. We confirm that all core research ideas, experimental designs, result analyses, and final conclusions were conceived and executed by the human authors. The entire content of the paper was carefully reviewed and revised by the authors, who bear full responsibility for the final submission.
B DETAILED PROOF FOR THE UPPER BOUND ON BROMOV-WASSERSTEIN DISTANCE
Theorem 2 The alignment of Gram matrices under the L T--norm serves as a tractable upper bound for the p-th order distance.
distance (GW,,, p). when the underlying metric is the cosine
Proof The objective is to formally demonstrate that the Manifold Alignment Regularization (MAR) strategy, which aligns Gram matrices, minimizes computationally tractable upper bound of the p-th order Gromov- Wasserstein (GW) distance between the feature manifolds of the original pre- pre-trained model and the fin- funettued model.
1. Definition of Metric Spaces. and (Y, dy. v).
We consider two discrete metric probability spaces, (X, dx, 11)
Space X: Represents the feature space of the original. frozen model. For a mini-batch of N samples, extract set of normalized features 7. (21, 22. 2N' where 2; E Rdd
14

---

substitute our fixed coupling that into the p-th power of Published as a conference paper at ICLR 2026
Space y: Represents the feature space of the fine-tuned model. For the same the corresponding set of normalized features is Z' 22.,, where 2 E R
Metric d: We employ the Cosine Distance as the metric. For pair of normalized vectors a and the distance is d(a, b) = -(a,b) b). The intra-space distance matrices, Dz and Dz', are thus:
(Dz)s = =d(z,, 25) (21, (Dz))is = d(z,, 21)===1))11- 11 (2/ z(y)
(14)
(15)
Probability Distributions μ,, Vz For a discrete batch of N samples, we ussume a uniform probability distribution, i.e., μ = V E where 6, is the Dirac measure.
2. Gromov- Wassesseenn Distance and the Concept of Coupling. The discrete p-th order GW distance is formulated as:
1/2
N
N (D2)s Ty
GW =
min TEll(AP))
(16)
LJ=-RN=1
Here, π E I(,,,) v) is a coupling, which is joint probability distribution over the product space Xxy. Intuitively, a coupling can be understood as a probabilistic transportation plan that describes how to map or associate the points from space X (with distribution (s) to the points in spuce y (with distribution v). The GW distance seeks the optimal coupling n" that minimizes the expected difference between pairwise distances in the two spaces. Finding this optimal plan involves solving quadratic assignment problem, which computationally intractable (NP-hard) for non-trivial cases.
3. Simplification via Fixed Coupling. To derive a computationally feasible upper bound, we forgo the optimization over all possible couplings and instead select single, fixed coupling. As outlined in the paper, we adopt a natural coupling, wha which assumes a one-to-one correspondence between the features of the same sample before and after fine-tuning. This coupling is formally defined as:
[1/N 0
fi Figk
That,ik
(17)
By defimition, the GW distance the minimum over all couplings. Therefore, using our specific That provides an upper bound on the true GW distance.
Derivation of the Upper Bound. the GW distance formula:
GW
(Dz) )Dzl) -Taat Tnot.j]
(18)
111=1k1=11
Due to the structure of Taan the term Tno ja is non-zero only when = and j = I, where it evaluates to (1/N)(1/N) = 1/N?. This simplifies the quadruple summation into a double summa- tion:
N N
GW ΣΣ
(Dz),, (Dz
(19)
5. Connection to Manifold Alignment Regularization (MAR). Let S and S' denote the Gram matrices (cosine similarity matrices). The absolute difference in distances can be expressed in terms of the Gram matrices:
(Dz) - (Dz) )syl =1(1 2,)) - (1 (2(,251)
=1S²/ Siyl
(20)
15

---

Published as a conference paper at ICLR 2026
Substituting this back into our inequality, we arrive the final upper bound:
N
N
GWA
Sig
(21)
This expression shows that the p-th power of the GW distance is upper-bounded by the scaled Lp- norm of the difference between the Gram matrices of the two feature spaces.
Conclusion. We have formally shown that minimizing the Ly-norm of the difference between Gram matrices corresponds to minimizing tractable upper bound the p-th power of the Gromov- Wasserstein distance. The Manifold Alignment Regularization (MAR) loss presented in the paper, LMAR. is specific instance of this principle using the Lı-norm (p 1). This provides strong theoretical justification for how aligning Gram matrices effectively preserves the geometric structure of the feature manifold during fine-tuning.
Table 7: Default hyperparameter settings used for training MPS-Tuning.
Hyperparameter
Value
Optimizer
SGD
Batch Size
32
Total Epochs
50
Peak Learning Rate
0.002
LR Scheduler
Cosine Decay
Logits Weight
0.3
MAR Loss Weight À
0.5
HMS Loss Weight A
0.1
HMS Depth
2
C HYPERPARAMETER SETTINGS
This section provides a detailed overview of the hyperparameter settings used in experiments for HPS--unnin. The primary hyperparameter values, which serve as the default for most datasets, are presented in Table 7 All experiments and were averaged over three different random seeds to ensure the reliability of our results.
conducted using the CLIP ViT-B/16 the base model
While most parameters were kept consistent to ensure fair evaluation, certain key hyperparam- eters were adjusted for specific datasets to optimize performance. Specifically, the weight for the Manifold Alignment Regularization loss (A₂) was increased to 2.0 for datasets with natural images, namely ImageNet, OxfordPets, Food101, and SUN397, to enforce stronger preservation of the rich pre-trained manifold. For all other datasets, the default value of 0.5 was used. Similarly, the batch size was set to 64 for the large-scas scale ImageNet dataset to ensure stable gradient estimation, while batch size of 32 was used for all other datasets.
Furthermore, we employed a dynamic temperature scheduling for the HMS loss (r') using cosine annealing strategy over the training epochs. For datasets including OxfordPets, Food101, Describ- ableTTxxtres, EuroSAT, and UCF101, the temperature was annealed from an initial value of 0.5 down to 0.07. For the remaining datasets, a more conservative schedule from 0.1 to 0.05 was ap- plied.
To mitigate the risk overfitting caused by excessive category-specifee supervision on intermediate layers, we incorporate simple layer-wise decay scheme in HMS. Concretely, the final layer is assigned a weight of 1, and the weight each preceding layer is defined recursively as half that of its subsequent layer:
Wr. 1,
W141 for = L L - 2, .,
(22)
where I. denotes the total number of layers.
16

---

Published as a conference paper at ICLR 2026
Regarding data augmentation, followed the standard protocol used in CoOp. This includes 'Ran- domResizedCrop with a scale range of (0.3, 1.0) and random horizontal flipping. No other complex augmentations were used.
D TRAINABLE MODULES
Benefiting from the powerful regularization capacity of MAR, direct fine-tuning of pre-trained mod- els becomes feasible in few-shot scenarios. Specifically, a hierarchical fine-tuning strategy is em- ployed for the visual encoder. The modules in the ViT/B-16 backbone are grouped based on se- mantic hierarchy (Gandelsman et al. 2024), with every four layers forming group. The first group remains frozen during training to retain general representation learning. In the second group, each Transformer block is paired with zero- zero-initialle initialized linear layer operating in parallel. Inputs processed by both branches, and their outputs are summed to allow lightweight adjustments in in- termediate representations. The third group is fully fine-tuned facilitate adaptation to downstream tasks.
Table 8: Summary of 11 datasets for few-shot learning and target datasets of domain general- ization. The 7 selected templates (hang et al.) (022) for ImageNet series datasets are "itap of [class].", "a bad photo of the [class].", "a origami [class].", "a photo of the large [class].", "a [class] in video game.", "art of the [class]." and "a photo of the small [class]."
Size (Thin Tex0
Desciption
Teaplate
ImogeNer
1000
1.30M/ /5000-
Recogrittin geazric New tea dan for Imogehet
ImgrrNen
1889
-7-7 10020
Peoertile of selected tereplates
ImgeNe:SSeh
Wagge or 2019
1000
-J-/ -J-/50639 50G39
Skotbbatle images ImageNet classss
Catehho0
Recogtition gesetiz objects
photo Iclass]"
2944 7336//776//3
SlanfiedCas
Krans
-1:4
196
6509 / 1635 1904
Fine gninel clasificaius of
"S photo of [clas].
X046
100
408/1 1633 /M663
Ftre-grained cusitcction of fowerrs
"a ghono of a [cltos] a type Bowse""
Foodi01
B Maqubail..ngg
101
50600/ 20200 30300
Fire staied dassifiction foods
"a plato of (clasl), a lape food."
A
VT13 1151
Pinz-grained ctectication sinenels.
"a photo of (clsol typa arrorth
1125/ 1692
Testue clasiliation
"[clas) testure"
Techere
12 211V
15500)
$400 88100
Land and unerr clacllizaioon with vatelite Action recogrition
"s centeraal sacilito phano (ctoal"
UCF161
101
7629
/395/3773
photo person doing (class).
E DATASETS
In the main text, method was assessed on the widely adopted CLIP Benchmark, in alignment
with previous work (Zhou et al. 20226] Zhang et al., [022]] Yu et al. 2023). The benchmark com-
prises 11 diverse datasets, including ImageNet (Deng al. 2009). Caltech101 (Fei-Fei al. (007),
Oxford Pets (Parkhi et al. 2012). Stanford Cars (Krause et al.] Flowers 102 (Nilsback & Zis-
serman 2008). Food101 Bossard et al. 2014), FGVCAircraft (Maji al. 2013). SUN397 (Xiao
et al.) 2010). DTD (Cimpoi et al.) 2014). EuroSAT (Helber et al. 2019). and UCFI01 (Soomro 2012). These datasets span a broad range of image classification scenarios, encompassing general object recognition, fine -raiind object recognition, scene recognition, texture recognition, and satellite imagery analysis, which allows for thorough assessment of our model's gemeralization capabilities across various domains. To ensure consistency with previous work (Zhou et al. 2022b Zhang et al. 2022; Yu al. 2023), the "BACKGROUND Google" and "Faces easy" classes were excluded from the ColtechT01 dataset. Additionally, robustness under domain shift was analyzed us- ing two ImageNet variants: ImageNet-Ve V2 (Recht et al. 2019), containing 200 overlapping classes, and MaageNet-Sketch (Wang et al. 2211)), encompassing 1,000 classes identical to ImageNet. Con- sistent with earlier works, ImageNet was used as the source dataset, while the two variants served target datasets. An overview of these datasets is presented in Tab. g
al.
F NUMERICAL. RESULTS
F.1 CLASSIHICATION RESULTS
Comparative evaluations (Tab. 9) were conducted across 11 benchmark datasets against state- of-the--rt methods, including CoOp (Zhou et al.] (0226). Tip-Adapter-F (Zhang et al. 2022). PLOT+ (Chen et al. 2023), MaPle (Khattak et al. 2023a PromptSRC (Khattak al. 2023h
17

---

Published as a conference paper at ICLR 2026
Table 9: Performance comparison on CLIP benchmark on ViT-B/16.
Meduel
Sattin
Tragehet
Canchili
DifiriNes
StadowrEan
RewesR22
Pecdi0t
FONCAROA
SUN391
EaNsAT
DCPOII
Avttage
Retentes 1 Prar
41.35
64:35
65:18
07:41
51.55
13.67
03.50
14.37
42.01
65:11
61.87
Cr0p 7300 01330 P00
56.13
72:00
20.33
$1.43
11:33
54.33
21.37
86.11
51:25
56.33
71:21
02:35
11 (1
06.00
61.44
1.
6881
38-08
84:15
13-99
66.77
5155
05:11
34:11
70:
0131
09.33
85.77
11.30
58.13
65-43
71:10
71.15
Malt
41.53
$1.05
08:13
23.7
7210
9940
1330
54.81
21.07
03
$5.23
75.13
74.00
72.33
07.00
93.53
91.43
05.11
16.57
54.5
23.57
54.00
73.31
70.33
belr
03.03
67.03
83.77
8110
10433
54.0
24.00
Teike
08.00
03.00
88.17
65.00
70.00
05.00
15.63
67.97
0183
53.01
08.0
06.56
8930
24.17
23.33
68.09
5897
23.33
58.50
76.00
85.67
75.30
1ac
99.64
53.79
01.23
08.24
15344
58.03
27.57
07.00
58.22
78.43
73.20
72.33
MPS 100
78.35
52.47
91.17
30.15
$6.30
96:15
36.97
10:40
55.47
76.
77.48
73.55
CxOy
67.07
33.07
5130
50.00
57.38
$4.40
26.30
65.33
$3.00
65.13
73.43
20.55
20.9
15.91
Taakko
70.30
24.23
93.1
72.0
53.00
13.07
53.45
33.57
70133
15.25
73.33
PreeraS0
09.37
54.33
12.00
73400
95.13
58.35
31.20
92.57
70.33
76.50
25.35
200s
63.01
83.31
85.1
11.15
43.0
54.4
73:55
3485
bePf Copreaeentn
65.00
91.35
93.00
11.35
58.90
85.00
13:43
10.55
99.55
08.51
17.00
75.55
22.2
$3.0
75.00
91.1
63.42
36.24
31.17
61.00
71.1
15.15
70.10
100m
3030
5485
01.57
72.00
91.31
68.33
14.25
11.55
61317
K2.87
7530
75.00
LBC
09.6
04.32
22.17
5535
18.21
96.07
28.94
75.05
24.46
30.35
92.03
74.0
91.00
55.30
36.37
13.97
01.55
21.37
24.00
TUB Natoniam
65
67:48
81:65
33.67
14.3
65:11
Ti
10.00
65.01
63.01
36.5
6061
86:63
16-15
10.77
76
36.1
76
23.13
01.5
35.23
6243
33.1
15.0
91.55
75%4
Malta 1tii--aar
30.31
23
23.23
7L.9
10447
58.63
12.55
12.73
01.17
76.11
71.03
75.13
AMEUTeerk
4435
71.00
56.31
03.51
02.00
71.52
56.76
05.03
85:11
65.50
14.05
36.37
10.34
99.33
73.33
75.0
76.63
76.13
7672
ball
71.07
03.22
79.0
16.55
90.03
4265
33.42
00.35
76.25
91.21
76.32
M00IL 127
31.40
59.00
02.57
56.17
94.00
55.13
13.05
$2.03
74.55
Sky
40
53.33
73.60
06.27
96.00
18000
13.00
65.00
85.81
03.53
TAE
73.73
55.00
53.67
79.63
04.30
55.85
41.95
14.70
65.97
88.33
02.03
76.02
72.57
96.30
$1.07
11.17
WUB
87.37
41.97
74.55
64.30
06.90
$4.08
91.77
FUP Rafinied 1
06.33
$3.35
69.55
65.55
67:44
81.55
33.69
67.50
11.5
40:31
05:11
61:3
Code
3083
34.27
04-97
02.63
30000
75.00
71.01
13.05
75.0
71.31
33:11
2110
$1.25
15.44
90.33
41.42
31.05
65.40
08.11
$2.00
79.55
Malta
71.63
$2.00
54.5
14.27
50.41
36.95
34.00
17.99
73.3
1.500
$0.97
16.27
59.00
13.09
JCP Youd
2000
71.00
$3.03
63.00
65.35
16.31
19.33
15.17
08.00
77.57
81.11
75.90
32.6
05.10
63.33
04.22
07381
43.45
15.53
$4.00
31.46
TouRe
30.11
92.00
09.35
$407
55.0
31.00
12.17
55.55
25.35
72.05
Sky
22.40
$5.00
33.40
83.35
(6.00
56.33
46.90
34.77
69.77
96.57
05.60
90.55
Loc mos
59.80
70.00
04
37:44
01.03
22.33
96.33
MPS MPS-wrnngmm milr
7480
96.33
43.77
$7.30
$7.63
$5.99
76.70
33.50
91.33
$7.83
K3.55
$1.35
15.55
6165
15.67
41.77
45:01
Dedp PaCT 114
71.37
13.27
01.37
$3.00
$4.35
43.40
34.07
01.7
96.33
82.21
70.3
PLOTee
12.00
03.50
$4.5
97.56
82.33
46.34
75.00
$2.00
Taikk
73:0
$9.57
92.9
56.0
15.90
14.00
90.5
73.11
56.00
53.7
8333
07.00
87.50
50.33
23
72.55
02.30
56
$2.91
75.00
1608
45.12
12.88
$4.66
72.40
87.23
baff
3335
05.27
04.13
1510
17.83
97.30
47.00
17.30
2410
91.43
55.53
12.00
TouRe Miten
30.00
55.27
33.13
72.00
00.30
87.33
34.37
34.3
$1.55
66.3
79.3
75.5
Maill
7340
5133
96-81
83.03
Lac
7188
05.0
6338
$1.11
V7.88
82.31
47.04
36.11
77.0
12.03
$4.65
12.0
MP5
75.0
01.37
$4.77
77.30
AMU-Tuning (Tang al.
2024), TCP Yao al. 2024), DePT (Zhang al. 2024). GalLop (Lafon
et al.
2024), TextRetiner (Xie et al. (024). ) MMRL (Guo & Gul 2025), SkipT (Wu et al. 2025). Li et al. 2025), and TAC (Hao et al. 2025). Our approach achieved the highest average perfor- mance under all few-shot settings (1, 2, 4, 8, and 16 shots), with its advantage becoming increasingly
LDC
pronounced as more samples were introduced, underscoring its robust learning capability.
F.2 ABLATION STUDY
Additional ablation results for the 1-, 2-, 4-, 8-, and 16-shot settings are provided in Tab. 10 and the overall trends are consistent with the main paper. A notable exception is the shot case, where HMS alone does not improve performance. This likely stems from the extremely limited positive sample pool, containing only the sample itself, its augmentations, and the associated textual feature, which encourages overfitting and weakens generalization. With more shots, the increased sample diversity allows HMS operate effectively and enhance performance.
18

---

Published as a conference paper at ICLR 2026
Table 10: More ablation results under 1, 2. 4, and 16-shot settings. The best result in each setting is highlighted in bold.
Setting
ImageNet
Cuzech10)
OxtordPei
SuatordCuss
Plowers
Food101
SUN397
DTD
EuroSAT
Average
94.00
90.15
86.30
25.17
09:40
57.53
74.47
CERMAR
70.30
94.07
86.93
70.20
75.73
(R-HMS
69.35
94.07
00.27
06.91
55.
54.97
27.35
09.63
56.53
74.23
76.7%
72:55
MPS-Tuning
70.37
94.47
91.17
70.17
56.50
86.10
20.97
70.40
$6.47
76.10
77.40
73,55
CE CE-MAR TF-HMMS
0M63
05.27
91. 30
72.53
91.67
85.
90.00
71.97
63.99
81.00
S0.20
s-shot
71.17
94.97
92.27
75,30
91.43
91.00
$5.40
34.17
72.07
61.17
61.63
80.97
82.03
50.43
76.49
MPSS-Tuning
91.57
33.77
72.37
63:47
31.63
CE-HIM
4-ohot
95.13
74.25
87.90
MPS-TTuxing
72.57
96.80
93.67
81.17
98.00
87.17
41.97
74.53
08.50
88.50
$4.30
$0.47
CR
71.90
96.75
50
85.17
86.20
$1.63
25.17
22.
90.13
56.15
82.9
CE+HIM
CR-MAR
8-sbot
73.17
73.67
96.30
96.50
95.87
03:47
85.35 86.20
$0.00
97.70
97.47
87.43
85.87
55.20
$4.80
75.93
76.30
22.53
22.87
90.83
90.03
$6.33
56.40
35.07
83.18
Tuning
99.03
76.70
73.30
91.13
75.30
73.07
CU-HIM
74.77
97.35
MPS--Tuning
75.00
94.77
99.37
77.20
94,37
Table 11: Sensitivity Study on MAR weight A1
Ag
ImageNet
Cahtac101
OkfoodPets
StanfordCars
Flawers022
Food101
PGVCAArerat
SUN899
DTD
EuroAAT
UCF101
Average
93.67
90.70
86.27
72.87
76.53
94.20
89.17
86.20
74.97
87.13
78.07
94.40
0.2
74.97
97.43
94.37
90.90
99.33
87.20
68.83
78.07
77.10
94.43
89.
86. 58
0.5
75.3
97.43
94.53
91.13
99.37
87,70
69.03
78.37
77.20
94.37
89.87
86. 74
1
75.43
97.43
91.70
90.97
88.00
68.83
78.43
77.30
93.37
89.63
86.76
75.60
88.00
86
5
75.47
96.93
94.37
90.03
98.83
88.03
66.50
78.13
76.33
91.30
88.20
85.83
10
75.13
96.73
91.13
88.30
98.53
88.00
64.13
71.90
76.07
91.27
87.47
85.27
Table 12: Sensitivity Study HMS weight X2
ImageeNer
Caltech101
OxfordPess
StanfordCars
Plowest022
Food101
PGVCAlreratt
SUNS97
DTD
EuroSAT
UCFF10
Average
0.01
75.37
97.13
94.37
91.03
99.27
88.03
68.93
78.10
77.20
93.75
89.17
$6.58
0.1
75.60
97.37
94.77
91.13
99.37
88.13
69.03
78.47
77.20
94.37
89.87
86.85
0.2
75.63
97.47
94.87
88.07
78.50
94,33
89:77
86 60
86.55
0.5
75.50
95.27
91.57
90.67
99.10
87.07
67.63
77.77
76.80
91.27
89.33
86.18
0.8
74.90
93.70
94.13
88.73
98.80
86.57
63.37
66.33
75.77
93.63
88.3
84-02
74.90
96.37
93.93
88.17
98.27
86.33
67.87
75.83
94.17
88.00
83.
73.82
Table 13: Sensitivity Study on logits weight ot
ImageNet
Caltech101
OxfordPets
StanfordCars
Flowers102
Food101
PGVCAirenft
SUNS39
DTD
EaroAAT
UCFL0I
Average
0.1
74.07
96.97
94.43
88.30
95.5
87.90
58.5
75,60
71.47
91.70
87,60
83.81
97.27
0.3
75.60
97.37
94,77
91.13
99.37
85.13
69.03
78.47
77.20
94.37
89.87
$6.85
0.4
75.67
91.40
94.63
91.13
99.37
88.07
68.87
78.57
77.30
94,20
89.90
86.83
0.5
75.57
97.37
94.57
91.13
09.
87.87
68.97
78.67
77.17
94.17
89.80
86.78
75.50
07.33
94.30
90.97
09. 27
68 87.87
94.00
89.57
866
75.33
99.40
89.27
86.45
0.8
75.00
97.47
92.97
89,80
98.99
86.33
67.90
76.27
76.40
93.20
88.93
85.93
0.9
34.57
97.20
91.37
89.33
99.00
81.93
67.73
77.83
76.63
93.00
88.57
85.47
1.0
74.00
97.07
89.00
88.33
98. 80
83.03
67.10
77.20
75.87
93.37
87.87
$4.09
G SENSITIVITY STUDY
We conducted sensitivity analyses for the weights of MAR (A1), HMS (A2), and the logits from the
fine-tuning branch (a), with the corresponding results shown in Tables Tab. [1 Tab. 12 and Tab. [3] respectively. Regarding Xı, we found that setting it to 0.5 or I yields the best performance when fixed value is applied across all datasets. To further improve performance, we divided the datasets into groups and set X to 0.5 and 2 for each group (see Hyperparameter Settings). respectively.
19

---

Published as a conference paper at ICLR 2026
190h weaet purgera
0.8
02
0.25-
00
SUND97
0.00
(a) Desine-Based quantification of feature shift. A lower score indicates stricter constraint on the output features.
(b) Hellinger-beeee based metric for quantifying logits shift. A lower score indicales stricter constraint the predicted outcomes.
04-
Mear Comearion
80
10.2-
cape Mese Comteroo
-06
ucf
VOVDArentt
(c) Spearman-based metric quantifying logits shift. A higher score indicates stricter constraint on the predicted outcomes.
(d) Impact MAR and HMS on feature discrimi- nation using Silhouette coefficient. higher score indicates better clustering performance.
Figure 10: Comparison different metrics and constraints: (a) Cosine, (b) Hellinger, (c) Spearman, and (d) Silhouette.
As for Ag. fixed it 0.1 across all datasets, and similarly, a was set 0.3 for all datasets. Notably, the model's performance exhibited minimal variation when these hyperparameters were adjusted around their default values, demonstrating the robustness our method to hyperparameter choices.
Table 14: Training cost under each method's default settings on SUN397.
Time(min)
Params(M)
Mem(GB)
CoOp
170.4
0.008
7.7
Tip-Adapter
9.4
3.3
2.9
MaPLe
13.6
1.2
7.2
TCP
21.8
0.3
9.0
CoCoOp
252.4
0.04
7.1
MPS-Tuning (Ours)
54.23
31
9.3
H TRAINING COST
Due to the fixed text encoder design, our method achieves high training efficiency with GPU memory usage and time costs comparable to previous methods (Tab. 14
20

---

Published as a conference paper at ICLR 2026
1 MORE INTERPRETATION RESULTS
1.1 IMPACT OF CONSISTENCY CONSTRAINTS ON MODEL ADAPTATION
To substantiate the potential limitations of different consistency constraints discussed in the inter- pretability section the main text, we conducted quantitative experiments analyzing changes in feature representations and logits distributions before and after fine-tuning.
Feature-Level Analysis We evaluated the cosine distance between model output features before and after fine-tuning across multiple datasets. In CLIP's normalized feature space used for similar- ity computation, the difference between two vectors measured by their angular separation. Since angular alignment is equivalent to vector alignment in this context, larger cosine distance between pre- and post-fine-tuning vectors indicates greater overall divergence. As shown in Fig. T0a cosine similarity constraints result in minimal angular changes between pre- and post-fine-tuning mod- els, even cross-domain scenarios (e.g., FGVCAircraft, where models typically require substantial adjustments for effective downstream task adaptation). This demonstrates that cosine similarity im- poses more stringent restrictions on feature variations compared to other feature-basee constraints.
2. Logits-Level Analysis We assessed the variation in prediction probabilities using Hellinger Distance and consistency of prediction rankings via Spearman's Rank Correlation, comparing models before and after fine-tuning. The former quantifies differences between two probability distributions, while the latter assesses the extent to which fine-tuned models maintain prediction ranking consistency with original models. The experimental results documented in Fig. 10b and Fig. [0c| show that models constrained by feature-level cosine similarity and logits-level KL di- vergence demonstrate markedly superior alignment with original model predictions compared to their LI and L.2 constrained counterparts. This superiority manifests through consistently reduced Hellinger Distance values and elevated Spearman's Rank Correlation coefficients, with the most pro- nounced differentiation occurring on the cross-domain FGVCAircraft dataset. These results confirm that cosine similarity and KL divergence heavily constrain model predictions.
These findings indicate that cosine similarity and KL divergence may impose overly rigid con- straints, potentially hampering the model's learning capacity. contrast, our proposed Mani- fold Alignment Regularization (MAR) offers dynamic constraint mechanism that adapts across datasets, enabling models engage in further learning when knowledge acquisition is necessary while effectively preserving pre-trained knowledge when high consistency exists between pre- trained and downstream task-required knowledge, thereby significantly enhancing model learning capability.
1.2 IMPACT OF MAR AND HMS ON FEATURE DISCRIMINATION
We further employed a common clustering metric, the Silhouette Coefficient, to evaluate the class separability of features in the representation space across different model components. As shown in Fig. Tod] using MAR alone improves class separability in certain scenarios, while HMS alone sig- nificantly enhances the distinction between categories. The combined use of MAR and HMS yields results comparable to using HMS alone, indicating the effectiveness of our approach in facilitating discriminative feature learning.
1.3 COMPARISON OF MANIFOLD ALIGNMENT PERFORMANCE
To rigorously verify manifold-leve alignment, we employed Topological Data Analysis (TDA) via Persistent Homology to quantify the topological consistency between the original CLIP manifold and those generated by different fine-tuning methods. Specifically, we calculated the Wasserstein distance on persistence diagrams for Ho (connected components, reflecting macro-separability)) and H, (loops, reflecting fine-grained geometry) to measure structural deviation. Empirical re- sults (Fig. II] across multiple datasets demonstrate that MPS-Tuning achieves consistently lower Wasserstein distances compared to baselines, indicating superior capability in preserving topologi- cal structures. It is important to note that while we do not claim strict homeomorphism or homology guarantees, which are intractable in deep representation learning, the TDA evidence confirms that
21

---

Published as a conference paper at ICLR 2026
regularization effectively preserves key homological features of the original manifold and pre- vents structural distortion beyond simple semantic approximation.
OC
20.0
NiS Tusiig Noons
Fertoredeee
120
Logtt Los
17.5
Festure bared LoW Loght cland Lose
WYVonseene Datunce
HD Wessorrstain Distanca
100
15.0
12.5
80
60
40
5.0
20
ImageNet
SUNb87
ucf101
FGVCNrcraft
0.0
ImageNet
SUN397
uCT1on
FGVCAirersf
(a) Hu
(b) H1
Figure 11: A comparison of manifold alignment performance is conducted using TDA analysis, where lower numerical values indicate better manifold alignment.
J VISUALIZATION
We visualize the features using t-SNE on the ImageNet, StanfordCars, and FGVCAircraft datasets under the 16-shot setting. As illustrated in Fig. 12 MPS-Tuning yields superior intra-class compact- ness and inter-class separability compared to both the non-fine-tuned model and Cross-Entropy loss (CEloss) fine-tuning. Notably. in scenarios where the original CLIP model performs well (e.g., Im- ageNet and StanfordCars). and inter-class relationships than CE loss tuning. In contrast, for datasets where CLIP underperforms (e.g., FGVCAircraft) the global semantic structure is largely retained, while local adjustments fa- cilitate improved classification.
more effectively preserves the original semantic structure
22

---

Published as a conference paper at ICLR 2026
CLIP
CE Loss
MPS-Tuning (Ours)
(a) ImageNet
CLIP
CE Loss
MPS-Tuning (Ours)
(b) StanfordCars
CLIP
CE Loss
MPS-Tuning (Ours)
FGVCAireraft
Figure 12: The t-SNE visualization, with each color denoting a distinct class.
23


Reviews:
[{"review_id": "4MRaPUUcKp", "reviewer": "Reviewer_GcHe", "summary": "This paper proposes MPS-Tuning from the perspective of manifold geometric structure preservation for robust few-shot tuning of VLM models. In details, the new method introduces two regularization strategies: Manifold Alignment Regularization (MAR) for preserving the visual manifold geometry and Hierarchical Manifold Sculpting (HMS) for enhancing the discriminability of the multi-level visual representations. With these regularizations, MPS-Tuning enables a direct fine-tuning on the VLM, which improves the data scalability on downstream tasks. Extensive experiments and ablation studies are presented to demonstrate the effectiveness of the method and the validity of each component.", "questions": "- In HMS, why using a pseudo-forward calculation that skips the attention operation to train the intermediate representations can lead to performance improvement? The intermediate representation could be very different to the original ones when the attention scores are not incorporated. Why does the regularization loss still meaningful under the circumstances?\n- I'm just wondering if the partial fine-tuning strategy described in section D could also be beneficial for other VLM tuning methods when the number of shots are large.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The idea of preserving knowledge via manifold structure regularization is intuitive and well-motivated.\n- The proposed method achieves strong performance compared to the state-of-the-art VLM few-shot learning methods, especially when the number of training samples are large (e.g., 16 shots).\n- The paper is solid with extensive experiment results supporting the major claims regarding the effectiveness of MAR and HMS.", "weaknesses": "- The method has its intrinsic limitation when the number of training samples are small: the Gram matrix is small and insufficient to capture the geometry. As a consequence, the 1-shot and 2-shot accuracies of MPS-Tuning are lower than SOTA methods on several datasets. The author is encouraged to discuss such limitation and potential solution when analyzing the results in figure 4.\n- In Table 2, the ablation study is only conducted under the 16-shot setting, which is insufficient to show the effectiveness of each component. Results of 1, 2, 4, 8-shot learning should be added.\n- (Minor) In table 1 column “-Sketch”, AMU-Tuning achieves the best result and should be marked bold.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FEWIk2gGLK", "reviewer": "Reviewer_xocW", "summary": "This paper proposes Manifold-Preserving and Sculpting Tuning (MPS-Tuning), a novel fine-tuning framework for adapting VLMs like CLIP to few-shot classification tasks. The authors' core argument is that existing methods, which rely on parameter-efficient tuning (PEFT) or point-wise consistency constraints (e.g., $L_2$ or KL loss between features), neglect the geometric structure of the pretrained feature manifold, leading to semantic distortion and overfitting. The combination of the proposed MAR (regularizer) and HMS (objective) allows for fine-tuning a substantial portion of the model while avoiding overfitting. The authors demonstrate state-of-the-art results on 11 few-shot benchmarks, with particularly strong gains in 8-shot and 16-shot settings.", "questions": "None", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Novel and Well-Motivated Conceptual Framework: The paper's primary strength is its conceptual shift from point-wise consistency to manifold-level consistency. The idea that preserving relationships between samples (in the Gram matrix) is more important than preserving the exact feature vector of each sample is intuitive and powerful.\n\nStrong Theoretical Grounding: The connection of the MAR loss to the Gromov-Wasserstein (GW) distance (Theorem 1, Appendix B) provides a solid theoretical foundation for the method. While it relies on a simplified upper bound (using a fixed coupling), this is a standard and necessary step to make the concept computationally tractable and provides a principled justification for using Gram matrix alignment.\n\nExcellent Empirical Results: The method achieves new state-of-the-art performance across a comprehensive suite of 11 few-shot datasets (Table 8). It outperforms a very strong and recent set of baselines (e.g., GalLop, TAC, MMRL). The fact that the performance gap widens as the shot count increases (e.g., the 16-shot average is 86.85%, a significant +2.5-3% jump over top competitors) suggests this is a very robust learning framework, not just a 1-shot trick. The SOTA domain generalization results (Table 1) further strengthen this.\n\nThorough and Convincing Ablation Studies: The authors have done an excellent job of validating their design choices.", "weaknesses": "The paper is very strong, and my points are primarily requests for clarification rather than major criticisms.\n\n1. Justification of \"Pseudo Forward\" Mechanism: A key component of the \"Hierarchical\" sculpting is the pseudo-forward projection (Fig. 3, Eq. 10), which projects intermediate features to the output space by skipping the Attention modules but keeping the $V_{Proj}$ and $FFN$ layers. This is a very specific and unusual design.\n\nQ1: Could the authors provide more intuition or justification for this? Why is this the correct projection? For instance, why skip Attention but keep $V_{Proj}$? Is this based on an analysis of information flow, or was it an empirical design choice that worked well?\n\n2. Clarity on Fine-Tuning Strategy (Appendix D): The paper's method is not a standard PEFT (like LoRA or Adapter) but rather a regularizer for partial fine-tuning. Appendix D details a \"hierarchical fine-tuning strategy\" where the first 4 layers are frozen, the next 4 have a parallel adapter-like module, and the last 4 are fully fine-tuned. This is a non-trivial number of trainable parameters.\n\nQ2: How critical is this specific tuning strategy to the method's success? The ablations in Table 2 all seem to use this strategy as a base. How would MPS-Tuning perform if applied to a standard full fine-tuning (FFT) baseline, or a standard PEFT method like LoRA? It is difficult to disentangle the gains from the novel regularizer (MAR+HMS) versus the gains from this specific partial-tuning strategy.\n\n3. Complexity of HMS Loss: The Hierarchical Manifold Sculpting (HMS) loss uses a combined support set $\\mathcal{S} = \\mathcal{Q} \\cup \\mathcal{T}$, meaning it performs image-image and image-text contrastive learning simultaneously.\n\nQ3: Was this combined support set found to be superior to a simpler approach, such as two separate losses (one for I-I alignment and one for I-T alignment)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Qfkd5OfHI", "reviewer": "Reviewer_TNxY", "summary": "This paper introduces MPS-Tuning (Manifold-Preserving and Sculpting Tuning), a fine-tuning framework for few-shot adaptation of pretrained vision-language models such as CLIP. The method treats the feature distribution as a semantic manifold and employs Manifold Alignment Regularization (MAR) to preserve its intrinsic geometry via Gram matrix alignment. Hierarchical Manifold Sculpting (HMS) refines local structures by enhancing intra-class compactness and inter-class separability across multiple layers.\nExtensive experiments demonstrate that MPS-Tuning outperforms state-of-the-art methods, with its effectiveness further validated by generalization studies.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper attempts to align the semantic spaces of images and texts from a manifold perspective, and designs two complementary modules — Manifold Alignment Regularization (MAR) and Hierarchical Manifold Sculpting (HMS) — to jointly balance knowledge preservation and adaptation to new domains. It also provides a theoretical foundation for semantic alignment under the Gromov–Wasserstein (GW) constraint.\n\n2. The paper conducts extensive experiments across multiple datasets, along with comprehensive ablation studies, to thoroughly validate the effectiveness of the proposed method.", "weaknesses": "1. Although the paper claims to constrain manifold alignment through the Gromov–Wasserstein (GW) distance, the metrics and formulations it employs do not appear to have an explicit mathematical correspondence to manifold geometry. Instead, the approach seems more accurately described as aligning semantic feature distributions rather than true manifolds. The use of the manifold concept thus appears somewhat overstretched.\n\n2. Since the proposed GM alignment serves as an upper-bound approximation to the GW distance, it raises questions about potential loss of alignment fidelity—specifically, whether this surrogate can genuinely achieve manifold-level correspondence, such as homeomorphism or homology preservation, or if it merely approximates semantic similarity in feature space.\n\n3. The proposed method is relatively complex, consisting of several loss and sub-loss functions, which leads to considerable computational overhead. Moreover, the paper does not provide strong empirical or theoretical evidence demonstrating the actual effectiveness of the Gromov–Wasserstein (GW) formulation in achieving meaningful manifold alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fmx8NJJpJ9", "reviewer": "Reviewer_uDYd", "summary": "The paper introduces MPS-Tuning, a novel fine-tuning strategy for vision-language models (VLMs) like CLIP in few-shot settings. It consists of two key components:\n1. Manifold Alignment Regularization (MAR): aligns Gram matrices between pretrained and fine-tuned representations to preserve both global and local manifold geometry, providing a tractable upper bound of the Gromov–Wasserstein distance.\n2. Hierarchical Manifold Sculpting (HMS): enhances intra-class compactness and inter-class separation via multimodal query-support matching, extended to intermediate layers using a “pseudo-forward” projection.\nExtensive experiments on 11 datasets show consistent improvements over state-of-the-art baselines, demonstrating better structure preservation and generalization.", "questions": "1. The ablation studies isolate MAR and HMS but do not show results for MPS-Tuning without α-fusion or with only global alignment. Would these results change the conclusion about the necessity of each component?\n2. How is the proposed “manifold preservation” paradigm fundamentally different from existing Gram-based alignment regularizations?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Clear motivation and theoretical foundation: connects manifold preservation with Gromov–Wasserstein distance, offering a principled justification.\n2. Gram-based alignment is simple yet theoretically grounded; HMS integrates seamlessly into VLM pipelines.\n3. Strong and consistent gains on 11 benchmarks and two domain generalization datasets.", "weaknesses": "1. The paper positions itself as introducing a “manifold-preserving” paradigm, but the actual mechanism Gram matrix regularization is already well explored in representation learning and knowledge distillation. The conceptual framing feels more like re-packaging than a genuine shift in understanding.\n1. The GW upper-bound proof relies on a fixed one-to-one coupling; lacks tightness or empirical verification against true GW distance.\n2. Computational cost not deeply analyzed: local Gram alignment (O(N·M²)) could be expensive; missing runtime/memory breakdown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)