Homology Consistency Constrained Efficient Tuning for Vision-Language Models
Authors: Huatian Zhang
Lei Zhang
Yongdong Zhang
Zhendong Mao
Venue: NeurIPS 2024 poster (2024)
Keywords: Efficient Transfer Learning
Vision-Language Models
Persistent Homology
Topological Data Analysis
Primary area: machine_vision

Abstract: Efficient transfer learning has shown remarkable performance in tuning large-scale vision-language models (VLMs) toward downstream tasks with limited data resources. The key challenge of efficient transfer lies in adjusting image-text alignment to be task-specific while preserving pre-trained general knowledge. However, existing methods adjust image-text alignment merely on a set of observed samples, e.g., data set and external knowledge base, which cannot guarantee to keep the correspondence of general concepts between image and text latent manifolds without being disrupted and thereby a weak generalization of the adjusted alignment. In this work, we propose a Homology Consistency (HC) constraint for efficient transfer on VLMs, which explicitly constrains the correspondence of image and text latent manifolds through structural equivalence based on persistent homology in downstream tuning. Specifically, we build simplicial complex on the top of data to mimic the topology of latent manifolds, then track the persistence of the homology classes of topological features across multiple scales, and guide the directions of persistence tracks in image and text manifolds to coincide each other, with a deviating perturbation additionally. For practical application, we tailor the implementation of our proposed HC constraint for two main paradigms of adapter tuning. Extensive experiments on few-shot learning over 11 datasets and domain generalization demonstrate the effectiveness and robustness of our method.

Homology Consistency Constrained Efficient Tuning for Vision-Language Models
Huatian Zhang, Lei Zhang, Yongdong Zhang, Zhendong Mao" University of Science and Technology of China
hutiinzhang@maill. ustc. edu. en, (leizh23, zhyd73, (dmmao)@ustte edu. cn
Abstract
Efficient transfer learning has shown remarkable performance in tuning large- scale Aision-language models (VLMs) toward downstream tasks with limited data resources. The key challenge of efficient transfer lies in adjusting image- text alignment to be task-specific while preserving pre-trained general knowledge. However, existing methods adjust image-text alignment merely set of observed samples, e.g. data set and external knowledge base, which cannot guarantee keep the correspondence of general concepts between image and text latent manifolds without being disrupted and thereby weak generalization of the adjusted alignment. In this work, we propose Homology Consistency (HC) constraint for efficient transfer on VLMs, which explicitly constrains the correspondence of image and text latent manifolds through structural equivalence based on persistent homology in downstream tuning. Specifically, we build simplicial complex on the top of data mimic the topology of latent manifolds, then track the persistence the homology classes of topological features across multiple scales, and guide the directions of persistence tracks image and text manifolds to coincide each other, with deviating perturbation additionally. For practical application, tailor the implementation of our proposed HC constraint two main paradigms adapter tuning. Extensive experiments on few-shot learning over datasets and domain generalization demonstrate the effectiveness and robustness of our method.
Introduction
Large-scale vision-language models (VLMs) such as CLIP [1] and ALIGN [2] trained on web-scale data have learned broad visual concepts and demonstrated promising generalization capability on wide range of downstream tasks, such as classification [3. 4], detection [5, 6] and segmentation [7, 8). In transferring to downstream tasks with limited data resources, the conventional full fine-tuning on VL.Ms often forgets the general knowledge learned pre-training and falls into overfitting. To mitigate this, how to efficiently transfer the knowledge from pre-trained VL.Ms to downstream tasks low-data regime has been intensively studied.
To possess both task-specific knowledge exploration and general knowledge preservation, efficient transfer learning proposes adapt VLMs fit downstream tasks by tuning a few parameters, which mainly exists in two paradigms: prompt tuning and adapter tuning. Prompt tuning methods adapt VLMs toward downstream tasks by introducing learnable prompts on the input side. Topics this branch include the configuration of learnable prompts 19-13] and injecting semantic priors, such as external knowledge [14, 15], category distribution [16] or visual diversity [17, 18], in tuning, As promising alternative, adapter tuning inserts a learnable lightweight adapter into the frozen pre-trained VLMs on the output side and the insertion manner allows modifying flexibly. By residual blending, CLIP-A M.PP-Adapter [19] tuned the image or text embeddings by appending a learnable bottleneck layer
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

to the frozen encoder. TaskRes [20] added learnable parameters as prior-independent residuals to text embeddings. Another line is based on key-value cache model. Tip-Adapter [21] proposed to construct an adapter with training images as keys and one-hot label encodings values for image recognition. Some work further enhanced the cache model by discriminative prior refinement [22] or knowledge augmentation [23]. For efficient transfer, existing methods mainly focus on how to configure tunable parameters or the leverage of external prior knowledge in tuning.
The essence of task-specific tuning on VLMs is to adjust the semantic alignment of image and text latent manifolds to fit downstream tasks. In this view, efficient transfer learning aims to establish new semantic ali alignments while keeping the correspondence of pre- pre-trained general concepts between image and text latent manifolds from being corrupted. This correspondence stems from the equivalence of semantics between the two manifolds. However, existing methods adjust image-text alignment toward downstream tasks on a set of observed samples from the latent manifolds. The discrete samples are incapable of adequately capturing the underlying structure of manifolds. The lack of perspective on latent manifolds in alignment adjusting risks devolving the desired manifold equivalence into localized closeness the observed data, particularly in a low-data regime, which causes the unguaranteed generalization of adjusted alignment beyond the data samples.
To this end, we propose to explicitly constrain the equivalence of image and text latent manifolds in transferring VLMs toward downstream tasks. We study the structure of the image and text latent manifolds from the lens of topological data analysis [24-26]. Topology encodes the connectivity of a space to describe its underlying structure. and the preservation of topology between spaces is fundamental for their structural equívalence. Topological data analysis employs homology groups to quantify the topological features of manifold structure, such as connected components, loops, voids, and eigher-dimensional holes, as homology classes and tracks the survival of the topological features across multiple scales via persistent homology [26] to capture their size and position, which summaries the global shape of manifold. These topology insights provide an avenue to achieve structural equivalence of image and text latent manifolds.
In this work, we propose Homology Consistency (HC) constraint for efficient transfer on VLMs, which constrains the structural equivalence of image and text latent manifolds based on persistent homology in downstream tuning. Given image-text data samples. we construct simplicial complex on the top of data to mimic the topological structure of latent manifolds, and induce a nested sequence of subcomplexes called filtration. Through the filtration, we capture the persistence of homology classes from their appearance to non-existence and guide the homology persistences of image and text manifolds to be consistent. Specifically, we locate the births and deaths of homology classes in latent manifolds and track these homology persistences, then guide the directions of persistence tracks in image and text manifolds to coincide each other so as to achieve homology-lew structural equivalence. Additionally, we apply a deviating perturbation to persistence-related text samples to encourage their respective semantically related images to be distributed uniformly relative to them in embedding, in order to enhance the generalization of the track coincidence in adjusting image-text alignment. Further, we tailor the implementation of the proposed HC constraint for the main residual blending and key-value cache based paradigms of adapter tuning. Extensive experiments on few-shot learning 11 benchmarks and domain generalization demonstrate the effectiveness and robustness of HC constraint in efficient transfer learning on VLMs.
Our main contributions are summarized as follows:
We propose explicitly constrain the structural equivalence of image and text latent manifolds in efficient transfer on VLMS, to improve the generalization of downstream image-text alignment adjusting beyond data samples in low-data regime.
We propose a theoretically well-founded homology consistency (HC) constraint based on persistent homology for efficient transfer on VLMs. We coincide the persistences of homology classes of topological features between image and text manifolds, and apply a deviating perturbation to generalize the persistence coincidence to unseen data.
We tailor the implementation of the proposed HC constraint for the two main paradigms of adapter tuning respectively, showing the transferability of our method.
We evaluate the proposed HC constraint on few-shot classification over 11 popular benchmarks. The extensive experiments demonstrate that HC constraint can boost the performance of baselines significantly and achieve state-of-the-art.
2

---

2 Related Work
Efficient Transfer Learning. To better transfer VLMs to downstream tasks especially with limited target domain data, a lot of research on efficient transfer learning has been done, which mainly exists in two types, prompt tuning and adapter tuning. Advanced than manual prompt that demands domain expertise to develop suitable format, prompt tuning methods design learnable prompts to adapt VLMs on downstream data. As a pioneer work, CoOp [9] for the first time composed prompts by concatenating text category embedding and learnable context vectors. The learnable prompts can be configured in text input [9, 10), image input [11] or jointly both [12, 13]. A line of work focuses injecting semantic priors, such as category-related external knowledge [14, 15], category embedding distribution [16] and the diversity of visual concepts [17. 18], in prompt tuning. In another branch, adapter tuning methods insert a learnable lightweight adapter module into the frozen pre-trained VLMs and show excellent performance. The adapter architecture and insertion manner allow for flexible modifications. There are two main paradigms: residual blending and key-value cache based. By residual blending, CLIP-Adapte [19] appended learnable bottleneck layer to frozen encoder tune embeddings. TaskRes [20] added set of prior- independent parameters to frozen text category embeddings to obtain an image classifier. GraphAdapter [27] proposed to learn downstream knowledge with inter-class relationship of image and text. samples. Based on the key-value cache model, Tip-Adapter [21] constructed an adapter with training images and label encodings as key- value to recognize query images. APE [22] refined the cache model by visual discrimination priors. CaFo [23] cascaded diverse external knowledge from DINO [28], DALL-E [29]. and GPT-3 [30] to assist recognition. However, although existing methods have attained remarkable achievements in VLMs transfer, they adjust the semantic alignment [3 [31-33] of image and text latent manifolds toward downstream tasks merely on observed samples, e.g., data sets and external knowledge bases, and lack insight into underlying manifold structure, which may cause unguaranteed generalization beyond the data samples. In this work, we propose to explicitly constrain the structural equivalence of image and text latent manifolds in downstream tuning to facilitate the transfer of VLMs.
Topological Data Analysis in Machine Learning. The area of topological data analysis [26] infers the topological structure of data spaces using algebraic tools such as persistent homology, and has been applied in many fields of machine learning, e.g., image segmentation [34-37], graph machine learning [38-40], molecular representation [41, 42], point cloud analysis [43, 44], etc. For instance, in image segmentation, [35] proposed to drive the segmentations to contain the specified topological features without requiring ground-truth labels. [36] used discrete Morse theory and persistent homology to learn the structural representation of images for fine-scale structure segmentation. In graph machine learning, [39] integrated vertex- and edge-level topological features into pressage-passing graph neural networks to boost their expressive power. In point cloud analysis, [44] developed a learnable filtration on point clouds obtain adaptive topological features for given tasks. Besides, [45] preserved the topological structures of input space into latent space of autoencoders by aligning topologically relevant distances. [46] applied representation topology divergence [47] in dimensionality reduction to force closeness on topological structures. How to analyse the structure of data spaces for semantic alignment in vision-language tasks through topology remains under-explored in the literature. In this work, we characterize and align the structure of the image and text latent manifolds by means of persistent homology in efficient transfer learning on VLMs.
3 Methodology
3.1
Preliminaries
3.1.1
Contrastive Language-Image Pre-training (CLIP)
As a representative VLM, CLIP [1] is trained on massive image-text pairs and shows promising zero-shot performance on downstream tasks. CLIP adopts two separate encoders to embed images and texts into latent manifolds, and aligns the bi-modal embeddings by contrastive learning that forces paired image and text closer and unpaired ones away. For the transfer to image classification with N classes, CLIP obtains the class embedding w; by feeding the prompt templates, e.g., "A photo of [CLASS]", filled with class name o; into text encoder, and the probability that an image belongs to category & can be formulated as p(u- c4 x) = exp (uTw/r)) Σ exp (x (wTww, /T). where the embeddings are normmaall and denotes temperature hyper- parameer.
wy
3

---

3.1.2 Persistent Homology
Discrete data points contain only observations from the latent space in which they reside and do not have interesting topology. To peek at the topological structure of latent space, we mimic its connectivity by constructing simplicial complex with the data points element, simplex a with dimension p is the convex hull set of p + 1 affinely independent points (Zo....,,Ip) Tp). Given finite data point set X in metric space (M,d) and a threshold > 0, the commonly used Vietoris-Ripe (Rips in short) complex is defined as:
vertices. As basic
Ka(X) = for C X d (T) < a, Vai, EoJ,
(I)
which is fully determined by the pairwise distances of X. The dimension of K, is the maximum dimension of any simplex within it. The formal sums of p-simplices added with Zg-additions form chain group Cp(Ka) Ka) (Cp for brevity). Then, define a boundary operator Öp on p-simplex as map that sends to the (p 1 --chaii consisting of d's (p- 1) faccs referred as a 's boundary. Applying
Og to the chain groups can obtain a sequence of homomorphisms: C₁. She CR-1 M Co. All p-chains whose boundaries are empty form a cycle group Zpp which the kernel Öp. The image of boundary operator 0p+1 on Cp+1 forms boundary group Bp. Further, taking the quotient of the Zp with By. the p-th homology group Н Zp/B, classifies the p-cycles Zp by collecting those cycles that differ by boundary into the same homology class. In a topological view, the rank of homology group Hp captures the number of p-dimensional holes in space.
Persistent homology offers a way to compute the quantified summary of topological structures of the latent space from sampled data. For the data point set X in space (M, d), we define a function f M + R. f ((zo..... :xp))
maxi,je (O.....) ff ((z:.15)
on simplices. Then, given a sequence of thresholds a1 < 02 I ..... < ag the growing sublevel sets (-oo, a] at these values give rise to nested se- quence of subcomplexes, Ka, C Kaz C .... C K called a filtration F, as shown in Fig. 1. The inclu-
K,
K
K
Figure 1: The sublevel set filtration on a nested family of Rips complexes.
sions in F induce:
0 0= Hp (Ko) +... Hp (Ki) + - (K,) ..... - Hpp(K)
(Kn))
(2)
where the images of the homomorphisms hid persistent homology groups H non-trivial homology class E Hp (Ke) is born at K2, i E Ha but e H.-1.a. Likewise. the homology class e dies entering Kj. i E Ho but & Hg.. The persistence homology class is the lifespan from its birth death. See Appendix A for more details.
3.2 Homology Consistency
Given a set of pre-trained image embed- dings X and text embeddings T, con- struct a Rips complex Ka (X) with the maximum pairwise distance aM of X and further derive a sublevel set filtration, F(K), the nested family of subcom- plexes Ka, S Ka, S ... C K at the increasing scale sequence pairwise distances {a: J.M, where ag = 0. Then we arrive at p-th persistent homology groups that capture the survival of the homology classes of p-dimensional topo- logical features (e.g., O-dimension: con- nected components, I-dimension: loops, 2-dimension: voids, etc.) and pair the birth and death times of p-th homology
unseen data
Lac
in-sample alignment driven by main tuning ubjective
data samples
(a) Persistence Track Coincidence
(b) Deviating Perturbation
Figure 2: Schematic illustration of proposed HC con-
straint. (a) TC guides the directions of the persistence tracks to coincide each other to establish the alignment of underlying structures beyond the observed samples. (b) DP encourages samples to be uniformly distributed.
classes, following [45]. Since the edge skeleton of Rips complex fully determines all of its simplices,

---

under the assumption that pairwise distances of X are unique, every birth-death time pair can be mapped back the simplices that respectively created and destroyed the unique corresponding homology class. The schematic illustration of HC constraint is shown in Fig. 2.
Persistence Track Coincidence. Let f be a p-th homology class that is born at as and dies at da- then the birth simplex 3 forms j. in Ka, and the death simplex ó causes to disappear entering Kaa- For example, if j were be O-dimensional, it emerges at ag. and is the edge that joins it with some other point; if je were to be I-dimensional, then 3 is the edge that forms the loop corresponding to in Kay, and 8 the triangle that incurs the loop to be contractible in Kaz- Since the Rips complex is pairwise distance based, on every p-simplex we have:
(((oo....,ppp))) (.........))
max f ((1;;
For the birth simplex & and death simplex 8, this means they will not be established until their longest edge (Ia(s); (z(0) largmax
f(0) mag.aa and (Zi(6)) Iz(0)) (ard)))rrmaa f(o) appear at time GN and ags which we refer to as birth edge and death edge, respectively. It can be said that the simplex 3 is completed by the birth edge and 8 by the death edge. Such edges mark the creation and destruction of the homology class j in the image latent manifold. Defining the direction of the edges as from to j such that i < in the given vertex sequence of simplices, we track the persistence direction of p-th homology class μ from its birth edge to death edge as:
trap(u, X) = (Z)(6)) T5(5)) arggmxx (()) - (T)()))(5((0)) aarg max [(33)-
Then we obtain the persistence track of μu in text latent manifold, trap(u, T), through determining its birth and death edges directly by taking the corresponding texts of the end- end-points of p's birth and death edges in image manifold accordingly. Further, for aligning the structure of image and text latent manifolds, we guide the track coincidence (TC) of p-th homology classes between the two by:
(.: =
1
Σ
potrra (1,, X), trap(u, T)),
ppl
μ
where r is the set of p-th homology classes in latent manifolds and & is cosine similarity.
Deviating Perturbation. In a low-data regime where samples are not sufficient to fully characterize the topology of the latent manifold of interset, the sight of persistence track coinciding is confined on the given limited samples, which hinders the generalization of structural equivalence guided by track coincidence. For all end-point images of birth and death edges of p-th homology classes in r', and their corresponding texts, we drive every text to deviate from its semantically related images in embedding (what we consider here that multiple images are related to the same category text in classification) without breaking the track coincidence, so as to encourage the text-related images beyond samples in latent manifold to be uniformly distributed around the text.
Specifically, since the persistence tracks of the 0-th homology classes are exactly the death edges (birth edges are 0), their end-point samples are available from the given dataset. We quantify the degree of deviation as the similarity between the orientations of the text's embedding relative to the embeddings of its related images. Then, for 0-th homology classes in T', we enlarge the deviation between end-point texts and their respective semantically related images by reducing the variation in their relative orientations, by applying a deviating perturbation (DP) as:
LDP (oP()T.TTTTT-
[1
E xTE
Σ
- p(x. (2.) (x.)r)).
(6)
Xro Xre
X
where Xro denotes the set of all end-point images of 0-th homology classes in To, (x;)r denotes a;'s 's corresponding text, X E Xr. denotes a set of images of the same category that are semanti- cally related to x; and oo is cosine similarity. The deviating perturbation can benefit the structural cquivalence to get rid of a biased reconstruction of latent topology caused by insufficient sampling.
Homology Consistency. To constrain the structural equivalence of image and text latent manifolds, we coincide persistence tracks of homology classes along with the deviating perturbation by:
(T X,T)
Σ (r, + (To, X,7), p=0
where À is the hyper-parameter controlling the perturbation strength. We mainly focus on the 0-th homology classes (p = 0), since higher-dimensional classes significantly increase computational cost but bring almost no additional performance benefits in VLMs tuning in practice.
5

---

3.3 Implementation
To examine the structural equivalence constrained by our proposed homology consistency in the efficient transfer of VLMs and show its transferability, we tailor the implementation of homology consistency constraint to the two main paradigms of adapter tuning, residual blending based and key-value cache model based, respectively.
Residual blending based tuning methods construct an adapter to produce learnable residuals and blend them with the pre-trained features. As a representative, TaskRes [20] adds prior-independent parameters
residual the pre-trained text embeddings to form learnable image classifier t' t'=t 8 + Oz, where a is a scaling factor, and updates the classifier by cross-entropy loss LCE.
In this paradigm, given frozen pre-trained image embeddings X and tunable text embeddings T. our method can naturally construct a filtration on X to capture the image persistence tracks and further obtain their text tunable counterparts, and then employ the persistence track coincidence with deviating perturbation through Licc (T, X, T)) in downstream tuning together with LCE.
Key-value cache based tuning methods build adapters via key-value cache model with pre- traiiannd embeddings of all training images as keys and one-hot encodings of corresponding labels as values. The image keys can be unfreezed as learnable parameters. As representative, to recognize an image J, Tip-Adapter-F [21] first measures its affinity weights with the cached keys Fiain by A = exp (-3 (1 x F' )). then aggregates the cached values Luain with weights A as a prediction ALuam and further combines Abuain with the similarity between image æ and pre-trained text category embeddings We as final classification logit a.A + xw /TCLIP in Leg.
In this paradigm, the cache-based adapter represents the categories as one-hot label encodings Lusin- that is, the Tumin is the only textual representation of categories in this adapter. To be comparable to label encodings, we regard the affinity weights of an image as sparse visual representation of this image. For implementing homology consistency, we first capture the birth and death edges of image homology classes based on pre-trained image embeddings, then replace end-point images of the edges with their corresponding affinity weights A or label encoding Lunin and follow Eq. 4 to construct tra,(u, A) and (1, Liain) analogically. We implement HC as LHc A, Lirai)) by taking (ra,(1,, A) and trap(u, Luain) as proxies for original image and text tracks.
Optimization. Efficient transfer learning commonly adopts the cross-entropy loss LCE between labels and the predicted class probability to tune learnable parameters, e.g., 6, on downstream classification tasks. We have two ways to integrate the gradient from our proposed HC constraint into parameter tuning. (1) We alter the cross-entropy gradient VOLCE toward the direction of HC by adding the gradient of HC constraint to VOLCE with a constant factor 1) by: VoL = (2) For Tip-Adapter-F, due to its slightly complex typer-paramett configuration, fixing a constant factor is difficult to control the contribution of each gradient. We adaptively keep the gradients at the same order of magnitude through scaling the gradient of homology consistency constraint based on gradient norm by V&L= VeLCE + w VaLuella VoLHC in optimization.
VVlceg
4 Experiments
4.1 Experimental Settings
Datasets. Following previous efficient transfer learning works, we conduct the few-shot learning eval- uation on benchmark datasets including Caltech101 [48], DTD [49], EuroSAT [50], FGVCAircraft [51], Flowers 102 [52], Food101 [53], ImageNet [54], OxfordPets [55]. StanfordCars [56), SUN397 [57] and UCF101 [58]. These datasets cover a wide range of visual recognition on generic objects, fine-grained categories, scenes, actions, etc. We sample 1, 2, 4. 8 and 16 shots per class, respectively, for model training and evaluate on full test sets. In addition, we evaluate the domain generalization performance of our method with the ImageNet [54] source and its variants ImageNetV2 [59], ImageNet-Sketch [60], ImageNet-A [61] and ImageNet-R [62] as targets.
Implementation Details. In addition to HC, following TaskRes, we augment the original pre-trained text features into ones tuned on downstream tasks in HC* In implementing HC/ / HC* on TaskRes (HC-TR/ / HC)- we set the values of 7 and X on different datasets according to the procedure of factor determination in ablation studies. We set the scaling factor a to for all datasets. The training batch size is 256. We employ the Adam optimizer with an initial learning rate of -e-1

---

Imageket
Caltech101
DTD
Scom (1
Zoon
CUP Acepfer
CLPnAdapter
CLI Adlgoo
HCT
50
15
Number ofmberr traitigg oxampits per dess
Nambero of labiied traning seamples per class
Number oflabeled trairing exump ca per closs
EuroSAT
FOVCAircrott
Flowers102
75-
Scoee 2511
CuP-Aceppar
CLthädaptar
HE:E
HCm
HC cm
HCT -m
HCC 77
Tis-Acopto-
60-
Hz 132
HC 100
75
16
26
15
soamples per
training
Foo0101
OxferdPets
StanfordCars
37
Taiifia
CUP-Acapter
"bactaa HCm
CLPh-dapter
CLIP CLIP-ddgpe
Taaidica
HCTR
FICm
60-
Tip-Acipton 7
HE-TA
HC: TAr
Bumber lasebed trining evarpies per class
Munberof sbeited traking essmxiles class
nurbeer oflateled trariigg noarpres class
SUN3397
UCF 101
Average over 11 datssets
12
80.0
77.5
75.0
70-
66-
72.5
CUP-Accpter
20.0
CLP Accpter
CLi--idaporr
8
HICTR
HCT HICT-TR
HCTTB IICTTT
IIC TT
00.0
HC -7A
16
16
16
ovartpens per
troiking ecaerplos,
mairiig momppees
Figure 3: The performance comparison of baselines and our proposed HC and HC* on few-shot learning, including 1-/2-44-1--/16-she 11 benchmark datasets and the average accuracy. The full numerical results are provided in the Appendix B.
on ImageNet and 1e on others, and the learning rates decay with cosine learning rate schedule following TaskRes. In implementing HC / HC* Tip-Adapter-F (HC-TAF -TAF, and À are also set according to the factor determination procedure in ablation studies. The À here is significantly larger than the above because the magnitude difference between the VoLre and LLLDP here is relatively large. The training batch size is 256. Following Tip-Adapter-F, we employ the AdamW optimizer with cosine annealing scheduler. We set initial learning rate as 1 All experiments conducted on a single NVIDIA A40 GPU. Note that the experiments on baseline and with the HC / HC* are conducted in the same setting for fair comparison. The code is publicly available 2.
4.2 Performance Analysis
Few-shot Learning. We validate the effectiveness of our proposed HC constraint on 11 benchmark datasets using representative adapter tuning methods, the residual blending based TaskRes and key-value cache based Tip-Adapter-F, as baselines. The experimental results are shown in Fig. 3, from which it can be observed that our method brings performance improvements for to 16 shots
7

---

consistently. Taking the 16 shots average accuracy as example, HC-TR exceeds TaskRes by 0.40% and HC-TAF exceeds Tip-Adapter-F by 1.04%. At 16-shot, HC-TAF outperforms Tip-Adapter- by 0.61% % on ImageNet. For the challenging fine-grained classification dataset FGVCAircraft, the proposed HC constraint gains 1 1.38% and 1.92% on TaskRes and Tip-Adapter-F; respectively. The further improvements of - and -C*-TAF yielded by pre-tuned text classifier suggest that the HC-equipped tuning can still significantly benefit from representation enhancement.
The performance comparison of our HC constraint with the state-of state-of-the-art adapter tuning methods on ImageNet is shown in Tab. 1. APE-T performs markedly best at and 2 shots, whereas the models con- strained by homology consistency, e.g., HC*-TAF, exceed it and other state-of-the- arts on the 8-/16-shu setting more sam- ple cases. This arises from the fact that the efficacy of homology consistency vision- language aligning depends primarily the modeling capability of the simplicial com-- plex builded training data towards the topological structure of latent space. The denser the data samples, the more sufficiently they summarize the structure, and the more effective the homology consistency constraint is. is worth noting that HC boosts baselines to achieve state- of-the-art without introducing any additional training parameters. We respectively take TaskRes and Tip-Adapter-P as representative methods of residual blending based (e.g., CLIP-Adapter, TaskRes, GraphAdapter) and key-value cache based (e.g., Tip-Adapter-E, APE-T) adapter tuning for applying HC constraint. The HC constraint is verified to be effective on these two baselines by extensive experiments, and can theoretically be extended to other VLMs efficient transfer learning methods.
Table
The performance comparison of our methods with the state-of-the-an methods on ImageNet.
Method
1-shot 2-shot 4-shot 8-shot 16-shot
Zero-shot CLIP [1]
58.18 58.18
58.18
58.18
58.18
CLIP-Adapter [19]
61.20 61.52
61.84
62.68
63.59
TaskRes [20]
61.44
62.28
62.78
64.30
65.41
Tip-Adapter-F [21]
61.16 61.74
62.65
64.07
65.43
GraphAdapter [27]
61.50 62.32
63.12
64.23
65.70
APE-T [22]
62.51 63.25 63.66
64.80
66.07
HC-TR (Ours)
61.46 62.34
62.86
64.47
65.90
HC-TAF (Ours)
61.17 61.79 62.73
64.37
66.04
HC TR (Ours)
62.15
62.59 63.60
64.81
66.25
HC -AAF (Ours)
62.04 62.61 63.28
64.86
66.40
Table 2: The performance comparison on domain generalization over four CLIP visual backbones. All methods are trained the ImageNet in 16-shot setting and evaluated on the comain-shifted datasets, ImageNet-V2, -Sketch, -A, and -R.
Source
Target -A
Method
Backbone
ImageNet
.V2
-kketch
-R
Average
Zero-shot CLIP [1]
58.18
31.34
33.32
21.63
56.00
40.58
Linear Probe CLIP [1]
55.87
45.97
19.07
12.74
28.16
28.16
TaskRes [20]
65.41
56.84
35.54
21.68
59.96
43.51
ResNet-50
Tip-Adapter-F [21]
65.43
57.20
35.99
23.52
60.45
44.29
HC-TR (Ours)
65.90
56.97
35.36
21.20
59.57
43.28
HC-TAF (Ours)
66.04
57.44
36.17
23.49
60.52
44.41
Zero-shot CLIP
61.62
54.81
38.71
28.05
64.38
46.49
Linear Probe CLIP [1]
59.75
50.05
26.80
19.44
47.19
35.87
TaskRes [20]
68.26
59.94
41.30
28.91
67.36
49.38
ResNet-101
Tip-Adapter-F [21]
68.47
59.69
41.63
30.05
68.04
49.85
HC-TR (Ours)
68.62
59.66
41.12
29.07
66.97
49.21
HC-TAF (Ours)
68.80
60.30
41.76
30.08
68.15
50.07
Zero-shot CI.IP
62.05
54.79
40.82
29.57
65.99
47.79
Linear Probe CL.IP [1]
59.58
49.73
28.06
19.67
47.20
36.17
TaskRes [20]
68.45
59.54
42.09
30.60
68.80
50.18
VIT-B/32
Tip-Adapter-F [21]
68.55
59.10
42.62
32.08
69.53
50.83
HC-TR (Ours)
68.71
59.57
42.09
30.59
68.86
50.28
HC-TAF (Ours)
69.04
59.75
42.74
32.16
69.60
51.06
Zero-shot CLIP
66.73
60.83
46.15
47.77
73.96
57.18
Linear Probe CLIP [I]
65.85
56.26
34.77
35.68
58.43
46.29
TaskRes [20]
73.55
65.81
48.86
49.85
77.35
60.47
VIT-B/16
Tip-Adapter- [21]
73.77
65.90
49.13
50.81
77.96
60.95
HC-TR (Ours)
73.85
65.98
48.88
50.08
77.51
60.61
HC-TAF (Ours)
74.08
66.18
49.30
50.75
78.01
61.06
Domain Generalization. We investigate the generalization ability of the models constrained by HC
under domain shift. We train the models on ImageNet with shots and test the trained models on ImageNet variant datasets ImageNet-V2. ImageNet-Sketch. ImageNet-A A and ImageNet-R. As

---

shown in Tab. 2, HC-TR and HC-TAF outperform their respective baselines on the source dataset ImageNet across four different visual backbones ResNet-50, ResNet-100, ViT-B/32 and ViT-B/16. The performance of HC-TR is slightly inferior to that of TaskRes on ImageNet-Sketch, -A and -R with backbones ResNet-50, ResNet-101 and ViT-B/32, which concurs with TaskRes's s reported source-overitting pitfall. Nonetheless, HC-TR outperforms TaskRes with the more representative ViT-B/16. HC-TAF shows better generalization than Tip- Sip-Adppter-F in almost all target domains with various backbones. Experiments demonstrate that the performance improvement is not reliant on the shortcut to overfit the seen data domain.
4.3 Ablation Studies
track coincidence and deviating perturbation. The homology consistency constraint consists of track co- incidence and deviating perturbation, and the effect of homology consistency con- straint in efficient transfer comes from their collaboration. Here, we investigate the in- dividual roles of track coincidence and de- viating perturbation in tuning. The ablation on ImageNet dataset is shown in Tab. 3,
Constraint terms:
Table 3: The ablation studies for the constraint terms, track coincidence (TC) and deviating perturbation (DP).
Baseline
TC DP 1-shot 2-shot 4-shot 8-shot 16-shot
61.44
62.28
62.78
64.30
65.41
61.46
62.38 62.95
64.40
65.74
TaskRes
62.14
62.67
64.23
65.33
62.34
62.86
64.47
65.90
61.16
61.74
62,65
64.07
65.43
61.17
61.76
62.70
64.22
65.99
Tip-Adapter-F
from which we can observe that (1) With-
61.76
62.63
63.95
65.17
out the generalization enhancement in la- tent spaces brought by DP, the performance of TC alone decreases relative to full homology consistency constraint. (2) Conversely, if we do not guide the coincidence of persistence tracks and only apply DP to track end-point samples, the original performance of baselines will be damaged. This is because DP plays role of regularization term for TC. Without coinciding persistence tracks by TC, not only the direct constraint on the structural equivalence of latent manifolds is lost, but also the only DP will cause the track end-point samples to randomly deviate from the hotelo-modal training samples, which interferes with the downstream tuning and thus performance drop.
61.79 62.73 64.37
66.04
Scaling factors of HC gradients in tuning. The hyper-parameters y and w scale the weight of homology consistency constraint in tuning and A controls the strength of deviating perturbation within the constraint (Sec. 3.3). We adopt the constant scaling (with 17, A) for TaskRes and adaptive scaling (with w, X) for Tip-Adapter-F. In general, performance first increases and then decreases as r) (or w) and X increase, and the optimal performance is achieved by specific combination of their values. Taking 16-shot ImageNet as an example, shown in Tab. 4, the optimal vi and À are 15 (i.e., 0.15 and 2.5 on HC-TR, the optimal w and À are 0.4 and 100 on HC-TAF. In implementation, we first determine the values of scaling factors at 16-shot, and then migrate them to HC* HC*-TAF and other few-shot settings. The setting of factors on other datasets follows similar procedure.
Table 4: The ablation studies for the scaling factors of gradients in homology consistency constraint.
") (A = 2.5)
5
10
15
20
25
HC-TR
65.67
65,83
65.90
65.79
65.72
À (n = 15)
1.5
2.0
2.5
3.0
3.5
HC-TR
65.83
65.88
65.90
65.87
65.82
w (A 100)
0.1
0.2
0.3
0.4
0.5
HC-TAF
65.59
65.82
65.96
66.04
65.99
A (w 0.4)
50
100
150
200
250
HC-TAF
66.03
66.04
66.00
66.00
65.98
5 Conclusions, Limitations and Future Work
In this work, we study the generalizability of image-text alignment adjusting in the efficient transfer of VLMs under a low-data regime. We propose to explicitly constrain the structural equivalence of image and text latent manifolds in downstream tuning and design a theoretically well-founded homology consistency constraint based on persistent homology for VLMs transfer. Our method constraint coincides the persistences of homology classes of topological features between image and text manifolds and applies deviating perturbation to generalize the persistence coincidence to unseen data. Extensive experiments demonstrate the effectiveness and robustness of our method.

---

Limitations and Future Work. In structural equivalence constraint, we do not explore the effects of higher-dimensional homology classes in depth. Besides, following previous work on efficient transfer learning for VLMSs, we only apply the proposed homology consistency constraint on series of few-shot recognition tasks. As next step, we will further extend the application scenarios of our method to other VLMSs downstream tasks.
Acknowledgments and Disclosure of Funding
This research is supported by Artificial Intelligence-Natin National Science and Technology Major Project 2023ZID1221 (0TTD0112200) 21000, National Science Fund for Excellent Young Scholars No. 62222212 and National Natural Science Foundation of China No. 62336001.
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[2] Chao Jia, Yinfei Yang. Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and rision-language representation learning with noisy text supervision. International conference on machine learning, pages 4904-4916 PMLR, 2021.
[3] Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6995-7004, 2021.
[4] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8552-8562, 2022.
[5] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region- based language-iman image pretraining. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16793-16803, 2022.
[6] Hengcan Shi, Munawar Hayat, Yicheng Wu, and Jianfei Cai. Proposalclip: Unsupervised open- category object proposal generation via exploiting clip cues. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9611 9620, 2022.
[7] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang. Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guides dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 59082-1809) 2022.
[8] Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1 11686-11 1695, 2022.
[9] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9))3337-2344, 2022.
[10] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for languaee models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[11] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727. Springer, 2022.
10

---

[12] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19113-19122, 2023.
[13] Sheng Shen, Shijia Yang, Tianjun Zhang, Bohan Zhai, Joseph E. Gonzalez, Kurt Keutzer, and Trevor Darrell. Multitask vision-language prompt tuning. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 5656-5667, 2024.
[14] Hantao Yao, Rui Zhang. and Changsheng Xu. Visual-language prompt tuning with knowledge- guided context optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6757-6761 2023.
[15] Baoshuo Kan, Teng Wang. Wenpeng Lu, Xiantong Zhen, Weili Guan, and Feng Zheng. Knowledge-aware prompt tuning for generalizable vision-language models, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15670-15680, 2023.
[16] Eulrang Cho, Jooyeon Kim, and Hyunwoo J Kim. Distribution-aware prompt tuning for vision- language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22004-22013, 2023.
[17] Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pages 5206-5215, 2022.
[18] Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Plot: Prompt learning with optimal transport for vision- langugee models. In International Conference on Learning Representations, 2023.
[19] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better rision-language models with feature adapters. International Journal of Computer Vision, pages 1-15, 2023.
[20] Tao Yu, Zhihe Lu, Xin Jin, Zhibo Chen, and Xinchao Wang. Task residual for tuning vision- language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10899-10909, 2023.
[21] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training- free adaption of clip for few-shot classification. In European Conference on Computer Vision, pages 493-510. Springer, 2022.
[22] Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, and Peng Gao. Not all features matter: Enhancing few-shot clip with adaptive prior refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2605-2615, 2023.
[23] Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Yu Qiao, Peng Gao, and Hongsheng Li. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15211-1522. 2023.
[24] Allen Hatcher. Algebraic Topology. Cambridge University Press, 2002.
[25] James R Munkres. Elements of algebraic topology. CRC press, 2018.
[26] Tamal Krishna Dey and Yusu Wang. Computational topology for data analysis. Cambridge University Press, 2022.
[27] Xin Li, Dongze Lian, Zhihe Lu, Jiawang Bai, Zhibo Chen, and Xinchao Wang. Graphadapter: Tuning lasion-language models with dual knowledge graph. Advances in Neural Information Processing Systems, 36, 2023.
[28] Mathilde Caron, Hugo Touvron, Ishan Mísra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650-96600 2021.
11

---

[29] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ila Sutskever. Zero- shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831, 2021.
[30] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:1877-1901, 2020.
[31] Zheren Fu, Zhendong Mao, Yan Song, and Yongdong Zhang. Learning semantic relationship among instances for image-text matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15159 15159-15168, 2023.
[32] Huatian Zhang, Lei Zhang, Kun Zhang, and Zhendong Mao. Identification of necessary semantic undertakers in the causal view for image-text text matching. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7105-71 14, 2024.
[33] Zheren Fu, Lei Zhang, Hou Xia, and Zhendong Mao. Linguistic-awart aware patch slimming frame- work for fine-graid uine-grineddee cross-modal alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26307-26316, 2024.
[34] Xiaoling Hu, Fuxin Li, Dimitris Samaras, and Chao Chen. segmentation. Advances in neural information processing systems, 32, 2019.
deep image
[35] James R Clough, Nicholas Byme. Ilkky Oksuz, Veronika A Zimmer, Julia A Schnabel, and Andrew P King. topological loss function for deep-learning based image segmentation using persistent homology. IEEE transactions pattern analysis and machine intelligence, 44(12):8766-877 2020.
[36] Xiaoling Hu, Dimitris Samaras, and Chao Chen. Learning probabilistic topological represen- tations using discrete morse theory. In The Eleventh International Conference Learning Representations, 2022.
[37] Nico Stucki, Johannes C Paetzold, Suprosanna Shit, Bjoern Menze, and UIrich Bauer. Topologi- cally faithful image segmentation via induced matching of persistence barcodes. In International Conference on Machine Learning, pages 32698-32727. PMLR, 2023.
[38] Zuoyu Yan, Tengfei Ma, Liangcai Gao, Zhi Tang, and Chao Chen. Link prediction with persistent homology: An interactive view. In International conference on machine learning, pages 11659- 169. PMLR, 2021.
[39] Johanna Immonen, Amauri Souza, and Vikas Garg. Going beyond persistent homology using persistent homology. Advances in Neural Information Processing Systems, 36, 2023.
[40] Joshua Southern, Jeremy Wayland, Michael Bronstein, and Bastian Rieck. Curvature filtrations for graph generative model evaluation. Advances in Neural Information Processing Systems, 36, 2023.
[41] Jacob Townsend, Cassie Putman Micucci, John H Hymel, Vasileios Maroulas, and Konstanti- nos D Vogiatzis. Representation of molecular structures with persistent homology for machine learning applications in chemistry. Nature communications, 11(1):3230, 2020.
[42] Yuankai Luo, Lei Shi, and Veronika Thost. Improving self-supervised molecular representation learning using persistent homology. Advances in Neural Information Processing Systems, 36, 2023.
[43] Vincent Peter Grande and Michael T Schaub. Topological point cloud clustering, In International Conference on Machine Learning, pages 11683 PMLR, 2023.
[44] Naoki Nishikawa, Yuichi Ike, and Kenji Yamanishi. Adaptive topological feature via persistent homology: Filtration learning for point clouds. Advances in Neural Information Processing Systems, 36, 2023.
[45] Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In International conference on machine learning, pages 7045-7054. PMLR, 2020.
12

---

[46] Ilya Trofimov, Daniil Cherniavskii, Eduard Tulchinskii, Nikita Balabin, Evgeny Burnaev, and Serguei Barannikov. Learning International Conference on Learning Representations, 2022.
data representations. In The Eleventh
[47] Serguei Barannikov, Ily Trofimov, Nikita Balabin, and Evgeny Burnaev. Representation topology divergence; A method for comparing neural network representations. In International Conference on Machine Learning, pages 1607-1626. PMLR, 2022.
[48] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178-178. IEEE, 2004.
[49] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 3606-3613 2014.
[50] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
[51] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- 2013.
grained visual classification of aircraft. arXiv preprini
[52] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 729. IEEE, 2008
[53] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 1ond-i00- discriminative components with random forests. In Computer Vision-ECCV 2014: 13th Earopean Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461. Springer, 2014.
[54] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255, Iece, 2009.
[55] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEEE, 2012.
[56] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[57] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 14485-3492 IEEE, 2010.
[58] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
[59] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400. PMLR, 2019.
[60] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global repre- sentations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
[61] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15262-15271, 2021.
[62] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: critical analysis of ditto---istribution generalization. In Proceedings of the IEEF/CVF international conference computer vision, pages $340-8349 2021.
13

---

A Simplicial Complex, Homology Class, Persistent Homology and Homotopy Equivalent Assumption
A.1 Simplicial Complex
simplicial complex consists of a set of simplices, such as points, line segments, triangles, and p-dimensional analogues. A simplex with dimension is the convex hull of set of + affinely independent points (fo...., Tp). For O I p' S p. a p'' p'-fa of a is a p' -ismmllex that is the convex
hull of non-empty subset. A simplicial complex K consists of a set of finite simplices that satisfy: K contains every face of each simplex in K and for any two simplices o, T E K, their intersection O1 T is either empty of any simplex in K. Given a finite data point set X in metric space (M, d) and a real a >> 0, the Vietoris-Rips complex K (X) is the set of simplices such that d(xx, x²) I for every pair of vertices of a, and the Čech complex KCech()) is the set of simplices such that the closed balls centering its vertices have a non-empty interaction.
face of both a and т. The dimension of K is the maximum dimension
A.2 Homology Class
In complex K, let the number of p-simplices be mp, a p-chain is formal sum of p-simplices added with some coefficients, i.e., c Σin 0, In particular, the p-chains with Zzdadiitions form a chain group Cp( K)) where the identity is the chain O = pp 00 and since c+ = 0, the inverse of a chain is itself. When K is clear from the context, Cp (K) can also be notated Cy- Define boundary operator 0p on p-simplex a as a map that sends a to the (p - 1)-chain consisting of a's (p 1)-faces referred 's boundary, i.e., 0, =
... Li ..., Fpp) where è; indicates that æ; is omitted. The boundary of vertex is empty. The Öp induces a homomorphism Op C, - Cp-1 that produces a (p- - 1)-chain when extended to a p-chain e tthrough 0pe = >mn Og (0po:). Applying Op to the chain 2. Co 2b C-1 0. where 0 denotes a trivial group, is obtained. A p-chain c is a p-cycle if has empty boundary. All p-cycles together form cycle group Zp under the Zz-addition operation inherited from Cp. Zp is the subgroup of Cp that is mapped to the of Cp-1 by 0p. i.e., Zp is the kernel of 0,.. The image of the boundary operator 0p is a subgroup of (p 1)-chains, called boundary group B = Op (Cp). For any p-simplex a, every (p- facee is involved in exactly two (p - 1)-faces in 0,0. thus 0p-1Bp-1 0 for p 0 and Bp-1 S Zp-1. The homology group Hp is defined to classify the cycles in Zp by collecting those cycles that differ by a boundary into the same class. This is achieved by Hp =7p/ /pp taking the quotient of the 7pp with boundary group Bp. By definition, the elements of Hp are cosets of Bp in Zep. Le.,, {c + Bp E 7-p) For cycle c, c e Bp in H, forms its homology class [c). Two cycles and e' in the same class [c] [c'] are homologous. In fact, [c] [c'] holds if and only e c' + Bp.
groups, sequence of homomorphism, 0 0= CC++1 a C* a. Ch-1.... C₁
A.3 Persistent Homology
For data point set X in metric space (M, d), define a function ʃ M + RR, ((Fo...., ma (p....p) ff ((1,,
xp))
on simplices. Then, given a sequence of thresholds ai S a2 S .... S an. the growing sublevel sets f (-o0, a] at these values give rise to a nested sequence of subcomplexes, Ka, S Ka, .... C Ka,, called filtration F. The inclusions in filtration FF induce a sequence
- Hp (K,) - Hp (K,) - Hp (K,) = (K).. called a homology module. The images of the homomorphisms h are persistent homology groups HiS for the module, H = im h²S for i S n. The non trivial elements of persistent homology groups H consist of homology classes that survive from K; K,. i.e., the homology classes that do not quotient out by boundaries, which implies that H = 7, (K,) /B, (K₂) Z. (K,)). A non-trivial homology class E H, (Ka) is born at K2, if e EE H5s" but & HE-1.e, Likewise, the dies entering Kj. if é E Hę but e & H When a class dies, it may be merged with several classes and raises new birth. The persistence of homology class that is created at K; and destoryed at K, defined as a; - aj. The homology classes that never die set to remain alive till an+1 Oco 00. As visual representation, the persistence diagram
of homomorphisms h4 HpF 0- Hp (Kn) Hp (K1)
14

---

D (F) of filtration F draws the paired birth at a; and death at oj that bound the survival interval
of one or more homology classes as a point (aa, az) on the extended plane R := (R U (+oo))?
A.4 Assumption on Homotopy Equivalent of Image and Text Latent Manifolds
Topology specifies how points are connected and key geometric signature of metric spaces. Topological equivalence can be formalized by continuous functions that map points from one space the other while preserving the connectivity. A straightforward equivalence is that the points as well as their neighborhoods in two spaces are in one-to-on one correspondence, called being homeomorphic [24, 26). which requires a bijecitve map with continuous inverse. Two homeomorphic spaces share the exact same topological properties, i.e., from the topological point of view, the two are completely consistent. Another less rigid relation is homotopy equivalent [24, 26]. Formally, given spaces X and W, two maps fo, fi X + Y are homotopic if they can be joined by continuous function F X [0, 1] + such that F F(x, 0) = fo(x) and F(z, 1) = fi (α) for every x E X, denoted as fo = fi. The spaces X and Y are homotopy equivalent if there exist two maps f X Y and g:Y + such that f g = idy and g o f idy, which suggests an intuitive fact that space X is homotopy equivalent to Y if and only if both X and Y are deformation retracts of a common space Z.
Vision and language are key information modalities for human cognition and have intrinsic corre- spondence on semantics. Observe that similar image scenes tend to be described by synonymic texts, that is, the connectivity of image data samples is preserved after mapping them to the text latent manifold, and vice versa. This preservation of connectivity suggests that the two manifolds have similar topological structures. Besides, image and text data are both relevant and complementary concrete expressions of the real world. The latent manifolds in which they reside can be seen as deformation retracts of the general world knowledge space. Thus, it is rational to consider that the image latent manifold and text latent manifold are homotogy equivalent, As topological invariant of homotogy equivalent spaces, the homology a viable tool for quantifying and aligning the structure of image and text latent manifolds.
B Full numerical results on few-shot learning
15

---

Table 5: Full numerical results of performance comparison on few-shot learning.
PGVCAireaa
StanfondCars
ImageNer
Caltech
EurSAAT
Flower1122
Food101
OxfordPets
SUN397
UCF101
DTD
Method
Setting
Zero-Shot CLIP
58.18
86.29
37.56
17.28
66.14
77.31
85.77
55.61
58.52
61.46
ALIp-Adapter
61.20
88.60
45.80
61.40
73.49
76.82
85.99
55.13
61.30
62.20
TaskRes
61.44
89.49
49.65
63.52
21.36
79.09
75.03
84.60
59.78
61.87
64.26
Tip-Adupter-F
61.16
89.21
49.17
59.10
20.46
79.29
77.52
86.43
59.69
61.95
65.05
I-shot
HC-TR
61.46
89.78
49.88
63.43
21.39
79.37
75.25
84,66
59.68
62.13
64.16
HC*-TR
62.15
89.61
49.70
64.02
21.48
80.02
75.47
84.57
59.84
62.92
64.79
HC-TAF
61.17
89.57
61.70
20.67
80.23
77.59
86.64
59.61
62.03
65.69
HC*
62.04
89.33
50.77
60.65
21.39
80.39
77.89
86.84
59.56
63.75
66.51
Zero-Shot CLIP
58.18
86.29
42.32
37.56
17.28
66. 14
77.31
85.77
55.61
58.52
61.46
CLIP-Adapter
61.52
89.37
51.48
63.90
20.10
81.61
77.22
86.73
58.74
63.29
67.12
TaskRes
62.28
89.86
54.43
65.99
23.67
83.09
75.88
85.64
62.70
64.89
67.41
Tip-Adapter-F
61.74
89,86
66.19
85.30
77.73
87.05
61.52
64.25
68.28
2-shot
HC-TR
62.34
89,90
54.79
66.91
24.63
83.11
76.03
85.66
63.06
65.04
67.57
HC*-TR
62.59
90.14
55.67
66.22
25,20
85.71
75.81
85.83
64.48
65.41
69.18
HC-TAF
61.79
90,26
54.61
67.53
24.21
85.99
77.78
87.22
63.11
64.42
68.31
HC*-TAF
62.61
90.63
56.09
67.33
24.69
87.49
77.65
88.
64.48
65.77
69.94
Zero-Shot CLIP
58.18
86.29
42.32
37.56
17.28
14
77.31
85.77
55.61
58.52
61.46
CLIP-Adapter
61.84
56.86
73.38
22.59
87.17
77.92
87.46
62.45
65.96
69.05
TaskRes
62.78
74.42
26.49
89.97
76.45
87.03
66.60
67.22
69.36
Tip-Adapter-F
62.65
90.95
58.22
73.46
25.98
89.69
78.12
87.49
65.73
67.13
71.05
4-shol
HC-TR
62.86
90.83
59.40
75.02
26.76
90.86
76.49
87.10
66.98
67.34
69.92
HC*-TR
63.60
91.60
59.34
75.14
27.69
92.13
76.86
87.22
68.54
67.36
71.64
HC-TAF
62.73
78.54
28.65
87.71
67.07
67.06
72.03
ITCF
63.28
91.76
$9.99
78.74
28.80
91.80
78.53
88.17
68.29
67.
73.17
Zero-Shot CLIP
58.18
86.29
42.32
37.56
17.28
66.14
77.31
85.77
55.61
58.52
61.46
CLIP-Adapter
62.68
91.40
61.00
77.93
26.25
91.72
78.04
87.65
67.89
67.50
73.30
TaskRes
64.30
92.09
63.48
78.07
32.25
94.05
77.82
87.35
70.70
69.29
74.46
Tip-Adapter-F
8-shol
64.07
92.25
63.12
78.04
32.16
91.76
78.52
87.79
70.12
69.43
74.97
HC-TR
64.47
78.35
32.34
87.79
71.38
74.81
HC*-TR
64.81
92.62
64.30
78.96
33.57
94.88
77.99
88.12
73.03
69.39
75.84
HC-TAF
64.37
92.45
63.83
81.35
34.62
93.02
78.92
88.61
71.98
69.74
76.02
-C*-TAF
64,86
93.06
65.13
81.83
35.52
94.36
78. 86
89.13
73.39
69.88
77.24
Zero-Shot CLIP
58.18
86,29
42.32
37.56
17.28
66.14
77.31
85.77
55.61
58.52
61.46
CLIP-Adapter
63.59
92.49
65.96
84.43
32.10
93.9
78.25
87.84
74.01
69.55
76.76
TaskRes
65.41
83.25
35.82
78.33
88.53
77.07
71.41
78.43
Tip-Adapter-F
65.43
93.06
66.55
84.98
36.42
94.64
79.4
89.67
75.80
71.52
78.09
16-shot
IIC-TR
65.90
93.59
67.61
83.19
37.20
96.06
78.57
88.93
77.61
71.64
78.54
HC TT
66.25
93.75
67.85
83.78
38.25
95.98
78.49
88.96
77.75
71.34
78.20
HC-TAF
66.04
93.35
68.32
86.65
38.34
95.37
79.54
90.49
77.55
72.01
79.22
HC -TFF
66.40
93.83
85.73
95.82
79.61
90.24
77.64
71.62
79.54
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly express our motivation and insight in the Abstract and Introduction. The contributions are summarized in the Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We clearly express our limitations in the Sec. 5, Conclusions, Limitations and Future Work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [Yes]
Justification: We discuss the homotopy equivalent assumption underpinning our method in Appendix A.4 and explain the rationale for such an assumption.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: Our paper provides a detailed account of implementation details and hyper- parameter settings of our method for reproducing.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, other means that appropriate the research performed.
also be provided via detailed instructions
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either be a way to access this model for reproducing the results or way reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path reproducing verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will make all code avaliable when paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nipel cc/
sosiesooiiissiiooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
cttpss.ccppublcc/guuee Ciaeeee/e/ooSSSSeiiSeeeee ssitimmmisiiioiiiiiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide implementation details Sec. 4.1, Experimental Settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We fix the random seeds in experiments for fair comparison.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
19

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It OK to report Isiigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of eπrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specify the type of compute workers in implementation details in Sec. 4.1.
Guidelines:
The answer NA means that paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eurrrp.ppeeee//ee wc/ppblic/.thhcccuteeeeeee
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We believe that our study will not pose any negative societal due to its theoretical nature.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI.).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our paper does not contain data or models that are at high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite the datasets used in our paper and give brief introduction.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package
should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's creators.
21

---

13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code//modll as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper has nothing to do with crowdsourcing and human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
22


Reviews:
[{"review_id": "xSnMaWnDtt", "reviewer": "Reviewer_yCSE", "summary": "This paper proposes Homology Consistency (HC) constraint for transfer learning on VLMs, and it explicitly constrains the correspondence of image and text latent manifolds by structural equivalence based on persistent homology in downstream tuning.", "questions": "see weaknesses.", "limitations": "The limitations are not discussed sufficiently (see weaknesses).", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method is well-founded and clearly explains the proposed homology consistency (HC) constraint.\n\n2. Extensive experiments are performed on 11 benchmark datasets.", "weaknesses": "1. The paper lacks discussions on the computational cost of the proposed techniques.\n\n2. The proposed method for constraining the structural equivalence of image and text latent manifolds seems generalizable to other learning tasks for vision-language models. However, the proposed method is only evaluated for few-shot learning of vision language models.\n\n3. Although the model outperforms other methods in most cases, the improvements are relatively marginal.\n\n4. The paper only applies the method to a limited number of adapter models (TaskRes and Tip-Adapter-F).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable concerns, and we would like to clarify as follows.\n\n**W1**: Since our proposed HC constraint incurs no additional cost in inference and the cost increase in offline training is marginal (less then 1.0%), we do not discuss computational cost in our paper. Following your suggestion, we take the transfer of VLMs on 16-shot ImageNet as an example to analyze the training cost of our HC. The training time of baseline Tip-Adapter-F is 15 s/epoch, and that of HC+Tip-Adapter-F constrained by our HC is 21 s/epoch. The peak total GPU memory usage of baseline Tip-Adapter-F is 16707MB, and 16751MB for HC+Tip-Adapter-F, only a 0.26% increase. We will add a discussion on computational cost in revision.\n\n**W2**: The efficient transfer learning on VLMs is to tune the large-scale VLMs (e.g., CLIP) toward downstream tasks under low-data regime. It aims to achieve considerable improvements in tuning VLMs on target tasks with limited data resources.\n\nFew-shot learning is a *widely-acknowledged evaluation standard for the efficient transfer learning*. Specifically, based on limited samples per class, few-shot learning on VLMs is to tune the pre-trained semantic alignment to handle the correspondence between images and category texts in target domain. Therefore, the performance on few-shot learning can effectively reflect the ability of efficient transfer.\n\nIn our paper, we conduct experiments on few-shot learning over 11 datasets (covering a wide range of visual recognition on generic objects, fine-grained categories, scenes, actions, etc.), domain generalization over 4 visual backbones from ImageNet source to 4 target domains ImageNet-V2/-Sketch/-A/-R, and a series of necessary ablation studies, to demonstrate the effectiveness and robustness of our method.\n\n**W3**: The performance improvements brought by our method are robust and reliable. For instance, we fix the random seed in training on 16-shot ImageNet from 1 to 4, and the performance of the baselines and constrained by our method are shown below.\n\n|      Method      | random seed 1 | random seed 2 | random seed 3 | random seed 4 |\n|:----------------:|:-------------:|:-------------:|:-------------:|:-------------:|\n|      TaskRes     |     65.41%    |     65.47%    |     65.49%    |     65.47%    |\n|    HC*+TaskRes    |     66.25%    |     66.13%    |     66.19%    |     66.24%    |\n|   Tip-Adapter-F  |     65.43%    |     65.05%    |     65.15%    |     64.99%    |\n| HC*+Tip-Adapter-F |     66.40%    |     66.27%    |     66.14%    |     66.05%    |\n\nAs shown in table, the average performance gain of HC*+Tip-Adapter-F over baseline Tip-Adapter-F is 1.06%, and the standard deviation is 0.098%. The average gain of HC*+TaskRes over TaskRes is 0.74%, and the standard deviation is only 0.069%.\n\nIn fact, our performance improvements are very competitive in the field of efficient transfer learning of VLMs. On the same evaluation on 16-shot ImageNet, GraphAdapter [1] (NeurIPS 2023) improves by 0.95% over the SOTA it compares to (refer to Tab. 1 in GraphAdapter [1]), APE-T  [2] (ICCV 2023) outperforms its baseline Tip-Adapter-F by 0.56% (refer to the numerical results APE-T [2] released), which is significantly lower than our average gain of 1.06% on the same baseline.\n\n[1] Li X, Lian D, Lu Z, et al. Graphadapter: Tuning vision-language models with dual knowledge graph[J]. Advances in Neural Information Processing Systems, 2023, 36.\n\n[2] Zhu X, Zhang R, He B, et al. Not all features matter: Enhancing few-shot clip with adaptive prior refinement[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 2605-2615.\n\n**W4**: We would like to clarify why we take TaskRes and Tip-Adapter-F as the baselines for applying our proposed HC constraint. TaskRes and Tip-Adapter-F belong to the two mainstream paradigms of adapter tuning (residual blending based and key-value cache based), respectively. These two paradigms cover almost all existing adapter tuning methods of efficient transfer on VLMs. In our paper (Tab. 1), we compare our performance with a range of SOTA adapter methods, e.g., residual blending based CLIP-Adapter, TaskRes, GraphAdapter, and key-value cache based Tip-Adapter-F, APE-T, etc, and outperform them on the most evaluation.\n\nThe models TaskRes and Tip-Adapter-F are representative in their respective paradigms. With the common residual blending idea in the same paradigm, the residual setting of TaskRes is relatively simple yet effective. For the key-value cache based paradigm, Tip-Adapter-F is the pioneering work that established the cache framework and subsequent methods such as APE-T are also improved from it.\n\nFor the efficient transfer learning on VLMs, our HC constraint proposes to preserve the vision-language alignment of pre-trained general concepts by constrain the topological equivalence between image and text latent manifolds in downstream tuning, to enhance the generalization of tuned alignment. This perspective has not been explored by existing VLMs transfer methods, and is verified to be effective in extensive experiments on two representative baselines. We believe that our HC constraint can be extended to other efficient tuning methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1pqTG3AkzE", "reviewer": "Reviewer_Ls33", "summary": "The paper introduces a Homology Consistency (HC) constraint for efficient transfer learning on vision-language models (VLMs), ensuring task-specific image-text alignment while preserving general knowledge by using structural equivalence based on persistent homology. This approach mimics the topology of latent manifolds and tracks the persistence of topological features.", "questions": "1. Can you visualise the topology of the data before and after adaptation with the HC constraint? It will be very interesting to see how actually the HC constraint can preserve the topology of the manifold during transfer learning.", "limitations": "My concern is the performance improvement is marginal and limited to more shots setting (16-shot).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well motivated, and the motivation of using homology consistency is interesting. \n2. This paper has a good theoretic support.", "weaknesses": "1. The performance of the proposed method is worse than the baseline method in low-shot (1-shot and 2-shot) tasks. \n2. The improvement in Table 2 is marginal. Is the comparison fair with the same random seed? How many runs did you conduct? Could the authors also report the standard deviation of the score? \n3. Moreover, is 16-shot common in this benchmark? 16 shot seems a lot in few-shot learning. \n4. Can you also elaborate more why with only DP, the performance drops in Table 3? \n5. In addition, could you elaborate more why choosing 0-th homology classes? What are the potential effects of using other homology classes?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments.\n\n**W1**: Our main results, the performance comparisons between baselines and our proposed HC/HC* on 1-/2-/4-/8-/16-shot settings over 11 benchmark datasets are shown in Fig. 3, and the corresponding detailed numerical results are in Appendix C. It can be found that HC/HC*+TaskRes and HC/HC*+Tip-Adapter-F outperform their baselines on 1-shot and 2-shot settings in most cases. We wonder if we misinterpreted your question due to some misunderstanding. We sincerely look forward to your reply.\n\n**W2**: Thank you for your concern. Experiments on baseline and with HC / HC* constraint are conducted in the same setting (as stated in Sec. 4.1 Experimental settings). For all experiments reported in paper, we fixed the random seed as 1. Our performance comparison is fair.\n\nHere, to verify the robustness of our improvement, we take 16-shot ImageNet as an example, and set the random seed from 1 to 4. The test accuracies of HC*+Tip-Adapter-F, HC*+TaskRes and their baselines are listed below.\n\n|      Method      | random seed 1 | random seed 2 | random seed 3 | random seed 4 |\n|:----------------:|:-------------:|:-------------:|:-------------:|:-------------:|\n|      TaskRes     |     65.41%    |     65.47%    |     65.49%    |     65.47%    |\n|    HC*+TaskRes    |     66.25%    |     66.13%    |     66.19%    |     66.24%    |\n|   Tip-Adapter-F  |     65.43%    |     65.05%    |     65.15%    |     64.99%    |\n| HC*+Tip-Adapter-F |     66.40%    |     66.27%    |     66.14%    |     66.05%    |\n\nAs shown in table, the average gain of HC*+Tip-Adapter-F compared to Tip-Adapter-F is 1.06% and the standard deviation is 0.098%. The average gain of HC*+TaskRes compared to TaskRes is 0.74% and the standard deviation is only 0.069%.\n\nIn fact, our performance improvements are very competitive in the field of efficient transfer learning of VLMs. On the same evaluation on 16-shot ImageNet, GraphAdapter [1] (NeurIPS 2023) improves by 0.95% over the SOTA it compares to (refer to Tab. 1 in GraphAdapter [1]), APE-T  [2] (ICCV 2023) outperforms its baseline Tip-Adapter-F by 0.56% (refer to the numerical results APE-T [2] released), which is significantly lower than our average gain of 1.06% on the same baseline.\n\n[1] Graphadapter: Tuning vision-language models with dual knowledge graph.\n\n[2] Not all features matter: Enhancing few-shot clip with adaptive prior refinement.\n\n**W3**: Following the commonly-used evaluation standard in previous work, we perform experiments on few-shot learning in 1-/2-/4-/8-/16-shot settings over 11 datasets (Fig. 3 in paper), and domain generalization on 16-shot ImageNet across 4 visual backbones (Tab. 2 in paper). Meanwhile, in transferring pre-trained VLMs, compared to the web-scale volume of pre-training data, the tuning data of 16 samples per class is significantly less, so it is reasonable to evaluate efficient transfer learning on VLMs in 16-shot setting.\n\n**W4**: Our proposed homology consistency (HC) constraint consists of track coincidence (TC) term and deviating perturbation (DP) term. TC term is responsible for guiding the persistence tracks in image and text manifolds to coincide each other to achieve a homology-level structural equivalence, and DP term drives end-point samples of tracks to deviate from their semantically related hetero-modal seen samples toward unseen ones in embedding (to be distributed uniformly), to enhance the generalization of TC in constraining structural equivalence of latent manifolds. \n\nIn other words, TC is leading role in achieving homology consistency, and DP can be regarded as its regularization term. Without coinciding persistence tracks by TC, not only the direct constraint on the structural equivalence of latent manifolds is lost, but also the only DP will cause the track end-point samples to randomly deviate from the hetero-modal training samples, which interferes with the downstream tuning and thus performance drop.\n\n**W5**: We mainly consider two aspects: computational cost and efficacy. *First*, in practice, the training time for constraining both 0-th and 1-st persistence tracks to coincide is about 20 times that for only the 0-th track coincidence. *Secondly*, we find that in tuning VLMs, adding 1-st homology consistency in addition to 0-th homology consistency brings almost no additional gain. We conjecture it is because the 1-st homology classes (i.e., loops) are too intricate for constraining the structural equivalence of image and text latent manifolds, and the existing VLMs' ability to represent data samples is insufficient to support the alignment of such precise structures.\n\n**Q1**: Following your suggestion, we visualize the persistence intervals of homology classes from their birth to death time through persistence diagram (PD), to show the change in the topology of data between the tuning with and without our HC constraint. Persistence diagram draws the paired birth at $b$ and death at $d$ that bound the survival interval of homology classes as a point $(b, d)$, which is a widely-used visual representation of persistent homology in topological data analysis.\n\nSpecifically, we visualize the persistence of homology classes of Rips complexes built on the top of test images, text embeddings tuned by TaskRes without HC constraint and text embeddings tuned by HC+TaskRes after applying HC, respectively, on 4 datasets including 2 generic object datasets, ImageNet and Caltech101, and 2 fine-grained object datasets, FGVCAircraft and StanfordCars.\n\nAs shown in Fig. 1 in the PDF we submitted, where the coordinates of the red dots represent the paired birth time and death time of the corresponding 0-th homology classes (Note that 0-th homology classes all emerge at 0). It can be found that, compared with the text manifold tuned without HC, the 0-th homology persistence of text manifold tuned by HC+TaskRes is evidently more akin to the image latent manifold.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ao6bvzlaa8", "reviewer": "Reviewer_KyVk", "summary": "The paper identifies a key issue with existing methods for tuning pre-trained vision-language models to downstream tasks with limited data: they adjust the alignment between image and text based solely on observed samples, which may not generalize well beyond the training data. To address this issue, the paper proposes a novel constraint from the perspective of topological data analysis.\n\nThis constraint employs persistent homology to ensure the structural equivalence of image and text latent manifolds during tuning.", "questions": "1.  Could you please provide insights into the fundamental differences and advantages of topological data analysis in your method over other alignment methods?\n\n2.  Where do you foresee potential challenges in extending your proposed method to tasks outside the few-shot learning domain, particularly in scenarios such as zero-shot learning, or applications involving detection and segmentation?\n\n3.  Could you elaborate on how incorporating higher-dimensional homology classes into the tuning process might impact the model's performance and behavior, beyond the computational cost?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.  The paper offers a new way of looking at model tuning through the lens of topological analysis, with a focus on understanding the structure of data spaces for better semantic alignment in vision-language tasks. I appreciate this perspective on the issue.\n\n2.  The proposed method exhibits a thoughtful theoretical underpinning, using persistent homology to enhance the generalizability of image-text alignment adjusting.&#x20;\n\n3.  The paper is well-written and the reason for leveraging topological data analysis to enhance semantic alignment during the tuning process is reasonable and easy to follow up.", "weaknesses": "The paper does not adequately discuss how it relates to existing image and text alignment techniques, including those based on distance metrics, mutual information, adversarial training, and attention mechanisms. This lack of comparative analysis creates a gap in fully appreciating the distinctive contributions and potential advantages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and insightful questions.\n\n**W1&Q1**: Our main differences and advantages over existing image-text alignment techniques are that we explicitly constrain the structural equivalence between image and text latent manifolds, and achieve a topological consistency of the latent manifolds beyond the localized closeness of discrete samples, so as to enhance the generalization of the learned image-text alignment.\n\nIn the widely-used alignment techniques, contrastive learning (e.g., InfoNCE loss, triplet loss) forces semantically related image-text pairs closer and pull unrelated ones away, to enhance the discrimination in the common embedding space. Mutual information maximization is to increase the correspondence probability between semantically related samples to overlap image and text latent manifolds for aligning. Cross-modal attention mechanism learns to focus on similar content in images and texts, thereby capturing the semantic associations across modalities accurately.\n\nDespite various approaches, the focus of existing alignment techniques tends to be limited to the localized closeness on the observed data samples, due to the lack of perspective on the underlying structure of image and text latent manifolds in training. The learned alignment cannot guarantee its generalization beyond training samples, especially in low-data regime.\n\nOur theoretically well-founded HC constructs the discrete analogues of image and text latent manifolds based on data samples, explicitly constrain the structural equivalence between image and text latent manifolds, and achieve a topological consistency on the underlying manifold structures beyond the localized closeness of discrete samples. As a constraint term in the efficient transfer learning on VLMs with limited data resources in our paper, the effectiveness and robustness of HC are demonstrated by extensive experiments.\n\n**Q2**: Thank you for your constructive comments.\n\nFor zero-shot learning: In capturing the structural association between a given test sample of unseen category and the training samples in their latent manifold for zero-shot inference, The embedding of test samples of unseen categories may introduce noise into the construction of simplicial complex and topological feature extraction. In addition, if the volume of pre-training set is too large, it is costly to obtain topological structure around the given test sample in manifold.\n\nFor image understanding tasks such as detection and segmentation: Unlike manifolds in semantic space considered by our HC, in the pixel space of interest for object detection and segmentation, the topological information of objects, such as shape, connectivity, and boundaries, is more intricate, thus requiring more sensitive topological structure representing methods. Meanwhile, topology information data in these fields is scarce, and annotation requires specialized knowledge.\n\n**Q3**: Beyond the computational cost, we find that adding 1-st homology consistency in addition to 0-th homology consistency brings almost no additional performance improvement in tuning VLMs. We conjecture it is because the 1-st homology classes (i.e., loops) are too intricate for constraining the structural equivalence of image and text latent manifolds, and the existing VLMs' ability to represent data samples is insufficient to support the alignment of such precise structures. In addition, the limited data samples may not be sufficient for capturing the persistence of higher-dimensional homology features in latent manifolds.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4cOdEh2D12", "reviewer": "Reviewer_zhsp", "summary": "A Homology Consistency (HC) constraint for efficient transfer on VLMs is proposed in this paper, which explicitly constrains the correspondence of image and text latent manifolds through structural equivalence based on persistent homology in downstream tuning.\nThe proposed method tracks the persistence of the homology classes of topological features across multiple scales and guide the directions of persistence tracks in image and text manifolds to coincide each other. Additionally, a deviating perturbation is applied to generalize the persistence coincidence to unseen data. Experiments on recognition and generalization tasks show the superior performance.", "questions": "The hyper-parameters η, λ, ω should be determined at 16 shots and then migrated to other few-shot settings. If the number of samples is less than 16, how should the aforementioned hyper-parameters be set, and will there be a significant difference in performance?", "limitations": "The experiments only conducted on recognition tasks.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-written with a straightforward motivation.\n2. A theoretically well-founded homology consistency (HC) constraint based on persistent homology is proposed for efficient transfer on VLMs.\n3. Experiments on recognition tasks show the superior performance.", "weaknesses": "The hyper-parameters η, λ, ω should be determined at 16 shots and then migrated to other few-shot settings. If the number of samples is less than 16, how should the aforementioned hyper-parameters be set, and will there be a significant difference in performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable concerns.\n\n**W1&Q1**: Referring to the widely-acknowledged evaluation standard in the same field, we conduct experiments on few-shot learning in the 1-/2-/4-/8-/16-shot setting. We find that perturbations near the optimal scaling hyper-parameters $\\eta$, $\\lambda$, $\\omega$ have little effect on model performance, as shown in the ablation study of hyper-parameters $\\eta$, $\\lambda$, $\\omega$ in our paper. That is, our performance is robust to these hyper-parameters. Therefore, we determine the values of these hyper-parameters in the setting with the largest number of shots (i.e., 16) and transfer them to other shot settings on the same dataset.\n\n**L1**: We would like to clarify why we conducted on  few-shot recognition tasks.\n\nThe efficient transfer learning on VLMs that we focus on is to tune the large-scale VLMs toward downstream tasks under low-data regime. It aims to achieve considerable improvements in tuning VLMs on target tasks with limited data resources.\n\nFew-shot recognition is a *widely-acknowledged evaluation standard* for the efficient transfer learning. Specifically, based on limited samples per class, few-shot  recognition on VLMs is to tune the pre-trained semantic alignment to handle the correspondence between images and category texts in target domain. Therefore, the performance on few-shot recognition can effectively reflect the ability of efficient transfer.\n\nIn our paper, we conduct experiments on few-shot recognition over 11 datasets (covering a wide range of visual recognition on generic objects, fine-grained categories, scenes, actions, etc.), domain generalization over 4 visual backbones from ImageNet source to 4 target domains ImageNet-V2/-Sketch/-A/-R, and a series of necessary ablation studies, to demonstrate the effectiveness and robustness of our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for reviewing this paper, and have provided detailed responses to all the concerns raised by the reviewers.

Meta-review: This paper suggests a novel homology consistency constraint to align language and vision representations for VLM transfer learning. Adding these constraints to two baseline transfer learning methods (TaskRes and Tip-Aligner-F) yields consistent improvements.  The reviewers agreed this approach has potential, but voiced some concerns regarding the experiments, mainly that they are only for few-shot classification, and that in some scenarios the results are not state of the art. Nonetheless, I feel this paper should be accepted: (a) because of the original idea of using persistence homology for VLMs and (b) because the topological regularization does consistently improve upon the backbone methods
Decision: Accept (poster)