Homology Consistency Constrained Efficient Tuning for Vision-Language Models

Authors: huatian  zhang
lei  zhang
yongdong  zhang
zhendong  mao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 10:01:40

Abstract: Efficient transfer learning has shown remarkable performance in tuning large-scale vision-language models (VLMs) toward downstream tasks with limited data resources. The key challenge of efficient transfer lies in adjusting image-text alignment to be task-specific while preserving pre-trained general knowledge. However, existing methods adjust image-text alignment merely on a set of observed samples, e.g., data set and external knowledge base, which cannot guarantee to keep the correspondence of general concepts between image and text latent manifolds without being disrupted and thereby a weak generalization of the adjusted alignment. In this work, we propose a Homology Consistency (HC) constraint for efficient transfer on VLMs, which explicitly constrains the correspondence of image and text latent manifolds through structural equivalence based on persistent homology in downstream tuning. Specifically, we build simplicial complex on the top of data to mimic the topology of latent manifolds, then track the persistence of the homology classes of topological features across multiple scales, and guide the directions of persistence tracks in image and text manifolds to coincide each other, with a deviating perturbation additionally. For practical application, we tailor the implementation of our proposed HC constraint for two main paradigms of adapter tuning. Extensive experiments on few-shot learning over 11 datasets and domain generalization demonstrate the effectiveness and robustness of our method.

Main paper: 
 introduction: 
 Large-scale vision-language models (VLMs) such as CLIP [1] and ALIGN [2] trained on web-scale data have learned broad visual concepts and demonstrated promising generalization capability on a wide range of downstream tasks, such as classification [3, 4] , detection [5, 6] and segmentation [7, 8] . In transferring to downstream tasks with limited data resources, the conventional full fine-tuning on VLMs often forgets the general knowledge learned in pre-training and falls into overfitting. To mitigate this, how to efficiently transfer the knowledge from pre-trained VLMs to downstream tasks in a low-data regime has been intensively studied.To possess both task-specific knowledge exploration and general knowledge preservation, efficient transfer learning proposes to adapt VLMs to fit downstream tasks by tuning a few parameters, which mainly exists in two paradigms: prompt tuning and adapter tuning. Prompt tuning methods adapt VLMs toward downstream tasks by introducing learnable prompts on the input side. Topics in this branch include the configuration of learnable prompts [9] [10] [11] [12] [13] and injecting semantic priors, such as external knowledge [14, 15] , category distribution [16] or visual diversity [17, 18] , in tuning. As a promising alternative, adapter tuning inserts a learnable lightweight adapter into the frozen pre-trained VLMs on the output side and the insertion manner allows modifying flexibly. By residual blending, CLIP-Adapter [19] tuned the image or text embeddings by appending a learnable bottleneck layer to the frozen encoder. TaskRes [20] added learnable parameters as prior-independent residuals to text embeddings. Another line is based on key-value cache model. Tip-Adapter [21] proposed to construct an adapter with training images as keys and one-hot label encodings as values for image recognition. Some work further enhanced the cache model by discriminative prior refinement [22] or knowledge augmentation [23] . For efficient transfer, existing methods mainly focus on how to configure tunable parameters or the leverage of external prior knowledge in tuning.The essence of task-specific tuning on VLMs is to adjust the semantic alignment of image and text latent manifolds to fit downstream tasks. In this view, efficient transfer learning aims to establish new semantic alignments while keeping the correspondence of pre-trained general concepts between image and text latent manifolds from being corrupted. This correspondence stems from the equivalence of semantics between the two manifolds. However, existing methods adjust image-text alignment toward downstream tasks on a set of observed samples from the latent manifolds. The discrete samples are incapable of adequately capturing the underlying structure of manifolds. The lack of perspective on latent manifolds in alignment adjusting risks devolving the desired manifold equivalence into localized closeness on the observed data, particularly in a low-data regime, which causes the unguaranteed generalization of adjusted alignment beyond the data samples.To this end, we propose to explicitly constrain the equivalence of image and text latent manifolds in transferring VLMs toward downstream tasks. We study the structure of the image and text latent manifolds from the lens of topological data analysis [24] [25] [26] . Topology encodes the connectivity of a space to describe its underlying structure, and the preservation of topology between spaces is fundamental for their structural equivalence. Topological data analysis employs homology groups to quantify the topological features of manifold structure, such as connected components, loops, voids, and higher-dimensional holes, as homology classes and tracks the survival of the topological features across multiple scales via persistent homology [26] to capture their size and position, which summaries the global shape of manifold. These topology insights provide an avenue to achieve a structural equivalence of image and text latent manifolds.In this work, we propose a Homology Consistency (HC) constraint for efficient transfer on VLMs, which constrains the structural equivalence of image and text latent manifolds based on persistent homology in downstream tuning. Given image-text data samples, we construct simplicial complex on the top of data to mimic the topological structure of latent manifolds, and induce a nested sequence of subcomplexes called filtration. Through the filtration, we capture the persistence of homology classes from their appearance to non-existence and guide the homology persistences of image and text manifolds to be consistent. Specifically, we locate the births and deaths of homology classes in latent manifolds and track these homology persistences, then guide the directions of persistence tracks in image and text manifolds to coincide each other so as to achieve a homology-level structural equivalence. Additionally, we apply a deviating perturbation to persistence-related text samples to encourage their respective semantically related images to be distributed uniformly relative to them in embedding, in order to enhance the generalization of the track coincidence in adjusting image-text alignment. Further, we tailor the implementation of the proposed HC constraint for the main residual blending and key-value cache based paradigms of adapter tuning. Extensive experiments on few-shot learning over 11 benchmarks and domain generalization demonstrate the effectiveness and robustness of HC constraint in efficient transfer learning on VLMs.Our main contributions are summarized as follows:• We propose to explicitly constrain the structural equivalence of image and text latent manifolds in efficient transfer on VLMs, to improve the generalization of downstream image-text alignment adjusting beyond data samples in a low-data regime.• We propose a theoretically well-founded homology consistency (HC) constraint based on persistent homology for efficient transfer on VLMs. We coincide the persistences of homology classes of topological features between image and text manifolds, and apply a deviating perturbation to generalize the persistence coincidence to unseen data.• We tailor the implementation of the proposed HC constraint for the two main paradigms of adapter tuning respectively, showing the transferability of our method.• We evaluate the proposed HC constraint on few-shot classification over 11 popular benchmarks.The extensive experiments demonstrate that HC constraint can boost the performance of baselines significantly and achieve state-of-the-art. 
related work: 
 Efficient Transfer Learning. To better transfer VLMs to downstream tasks especially with limited target domain data, a lot of research on efficient transfer learning has been done, which mainly exists in two types, prompt tuning and adapter tuning. Advanced than manual prompt that demands domain expertise to develop suitable format, prompt tuning methods design learnable prompts to adapt VLMs on downstream data. As a pioneer work, CoOp [9] for the first time composed prompts by concatenating text category embedding and learnable context vectors. The learnable prompts can be configured in text input [9, 10] , image input [11] or jointly both [12, 13] . A line of work focuses on injecting semantic priors, such as category-related external knowledge [14, 15] , category embedding distribution [16] and the diversity of visual concepts [17, 18] , in prompt tuning. In another branch, adapter tuning methods insert a learnable lightweight adapter module into the frozen pre-trained VLMs and show excellent performance. The adapter architecture and insertion manner allow for flexible modifications. There are two main paradigms: residual blending and key-value cache based. By residual blending, CLIP-Adapter [19] appended learnable bottleneck layer to frozen encoder to tune embeddings. TaskRes [20] added a set of prior-independent parameters to frozen text category embeddings to obtain an image classifier. GraphAdapter [27] proposed to learn downstream knowledge with inter-class relationship of image and text samples. Based on the key-value cache model, Tip-Adapter [21] constructed an adapter with training images and label encodings as keyvalue to recognize query images. APE [22] refined the cache model by visual discrimination priors. CaFo [23] cascaded diverse external knowledge from DINO [28] , DALL-E [29] , and GPT-3 [30] to assist recognition. However, although existing methods have attained remarkable achievements in VLMs transfer, they adjust the semantic alignment [31] [32] [33] of image and text latent manifolds toward downstream tasks merely on observed samples, e.g., data sets and external knowledge bases, and lack insight into underlying manifold structure, which may cause unguaranteed generalization beyond the data samples. In this work, we propose to explicitly constrain the structural equivalence of image and text latent manifolds in downstream tuning to facilitate the transfer of VLMs.Topological Data Analysis in Machine Learning. The area of topological data analysis [26] infers the topological structure of data spaces using algebraic tools such as persistent homology, and has been applied in many fields of machine learning, e.g., image segmentation [34] [35] [36] [37] , graph machine learning [38] [39] [40] , molecular representation [41, 42] , point cloud analysis [43, 44] , etc. For instance, in image segmentation, [35] proposed to drive the segmentations to contain the specified topological features without requiring ground-truth labels. [36] used discrete Morse theory and persistent homology to learn the structural representation of images for fine-scale structure segmentation. In graph machine learning, [39] integrated vertex-and edge-level topological features into message-passing graph neural networks to boost their expressive power. In point cloud analysis, [44] developed a learnable filtration on point clouds to obtain adaptive topological features for given tasks. Besides, [45] preserved the topological structures of input space into latent space of autoencoders by aligning topologically relevant distances. [46] applied representation topology divergence [47] in dimensionality reduction to force closeness on topological structures. How to analyse the structure of data spaces for semantic alignment in vision-language tasks through topology remains under-explored in the literature. In this work, we characterize and align the structure of the image and text latent manifolds by means of persistent homology in efficient transfer learning on VLMs. As a representative VLM, CLIP [1] is trained on massive image-text pairs and shows promising zero-shot performance on downstream tasks. CLIP adopts two separate encoders to embed images and texts into latent manifolds, and aligns the bi-modal embeddings by contrastive learning that forces paired image and text closer and unpaired ones away. For the transfer to image classification with N classes, CLIP obtains the class embedding w i by feeding the prompt templates, e.g., "A photo of a [CLASS]", filled with class name c i into text encoder, and the probability that an image x belongs to category c i can be formulated as p(y = c i |x) = exp x ⊤ w i /τ / N j=1 exp x ⊤ w j /τ , where the embeddings are l 2 -normalized and τ denotes a temperature hyper-parameter. 
persistent homology: 
 Discrete data points contain only observations from the latent space in which they reside and do not have interesting topology. To peek at the topological structure of latent space, we mimic its connectivity by constructing a simplicial complex with the data points as vertices. As a basic element, simplex σ with dimension p is the convex hull of a set of p + 1 affinely independent points (x 0 , . . . , x p ). Given a finite data point set X in metric space (M, d) and a threshold a > 0, the commonly used Vietoris-Rips (Rips in short) complex is defined as:EQUATIONwhich is fully determined by the pairwise distances of X. The dimension of K a is the maximum dimension of any simplex within it. The formal sums of p-simplices added with Z 2 -additions form a chain group C p (K a ) (C p for brevity). Then, define a boundary operator ∂ p on p-simplex σ as a map that sends σ to the (p -1)-chain consisting of σ's (p -1)-faces referred as σ 's boundary. Applying ∂ p to the chain groups can obtain a sequence of homomorphisms: Persistent homology offers a way to compute the quantified summary of topological structures of the latent space from sampled data. For the data point set X in space (M, d), we define a function f : M → R, f ((x 0 , . . . , x p )) = max i,j∈{0,...,p} f ((x i , x j )) on simplices. Then, given a sequence of thresholds a 1 ≤ a 2 ≤ . . . , ≤ a n , the growing sublevel sets f -1 (-∞, a] at these values give rise to a nested sequence of subcomplexes, K a1 ⊆ K a2 ⊆ • • • ⊆ K an , called a filtration F, as shown in Fig. 1 . The inclusions in F induce:C k ∂ k -→ C k-1 • • • C 1 ∂1 -→ C 0 . All p-EQUATIONwhere the images of the homomorphisms h i,j p are persistent homology groups H i,j p . A non-trivial homology class ϵ ∈ H p (K a ) is born atK i , if ϵ ∈ H i,a p but ϵ / ∈ H i-1,a p. Likewise, the homology class ϵ dies entering K j , if ϵ ∈ H a,j-1 p but ϵ / ∈ H a,j p . The persistence of a homology class is the lifespan from its birth to death. See Appendix A for more details. Given a set of pre-trained image embeddings X and text embeddings T , we construct a Rips complex K a M (X) with the maximum pairwise distance a M of X and further derive a sublevel set filtration, F(K), as the nested family of subcomplexes K a0 ⊆ K a1 ⊆ • • • ⊆ K a M at the increasing scale sequence of pairwise distances {a i } M i=0 where a 0 = 0. Then we arrive at p-th persistent homology groups that capture the survival of the homology classes of p-dimensional topological features (e.g., 0-dimension: connected components, 1-dimension: loops, 2-dimension: voids, etc.) and pair the birth and death times of p-th homology classes, following [45] . Since the edge skeleton of Rips complex fully determines all of its simplices, under the assumption that pairwise distances of X are unique, every birth-death time pair can be mapped back to the simplices that respectively created and destroyed the unique corresponding homology class. The schematic illustration of HC constraint is shown in Fig. 2 .Persistence Track Coincidence. Let µ be a p-th homology class that is born at a b and dies at a d , then the birth simplex β forms µ in K a b and the death simplex δ causes µ to disappear entering K a d . For example, if µ were to be 0-dimensional, it emerges at a 0 , and δ is the edge that joins it with some other point; if µ were to be 1-dimensional, then β is the edge that forms the loop corresponding to µ in K a b , and δ is the triangle that incurs the loop to be contractible in K a d . Since the Rips complex is pairwise distance based, on every p-simplex we have:EQUATIONFor the birth simplex β and death simplex δ, this means they will not be established until their longest edge (x i(β) , x j(β) ) arg max f (β) and (x i(δ) , x j(δ) ) arg max f (δ) appear at time a b and a d , which we refer to as birth edge and death edge, respectively. It can be said that the simplex β is completed by the birth edge and δ by the death edge. Such edges mark the creation and destruction of the homology class µ in the image latent manifold. Defining the direction of the edges as from i to j such that i < j in the given vertex sequence of simplices, we track the persistence direction of p-th homology class µ from its birth edge to death edge as:EQUATIONThen we obtain the persistence track of µ in text latent manifold, tra p (µ, T ), through determining its birth and death edges directly by taking the corresponding texts of the end-points of µ's birth and death edges in image manifold accordingly. Further, for aligning the structure of image and text latent manifolds, we guide the track coincidence (TC) of p-th homology classes between the two by:EQUATIONwhere Γ p is the set of p-th homology classes in latent manifolds and φ is cosine similarity.Deviating Perturbation. In a low-data regime where samples are not sufficient to fully characterize the topology of the latent manifold of interset, the sight of persistence track coinciding is confined on the given limited samples, which hinders the generalization of structural equivalence guided by track coincidence. For all end-point images of birth and death edges of p-th homology classes in Γ p and their corresponding texts, we drive every text to deviate from its semantically related images in embedding (what we consider here is that multiple images are related to the same category text in classification) without breaking the track coincidence, so as to encourage the text-related images beyond samples in latent manifold to be uniformly distributed around the text.Specifically, since the persistence tracks of the 0-th homology classes are exactly the death edges (birth edges are 0), their end-point samples are available from the given dataset. We quantify the degree of deviation as the similarity between the orientations of the text's embedding relative to the embeddings of its related images. Then, for 0-th homology classes in Γ 0 , we enlarge the deviation between end-point texts and their respective semantically related images by reducing the variation in their relative orientations, by applying a deviating perturbation (DP) as:EQUATIONwhere X Γ0 denotes the set of all end-point images of 0-th homology classes in Γ 0 , (x i ) T denotes x i 's corresponding text, X ′ i ∈ X Γ0 denotes a set of images of the same category that are semantically related to x i and φ is cosine similarity. The deviating perturbation can benefit the structural equivalence to get rid of a biased reconstruction of latent topology caused by insufficient sampling.Homology Consistency. To constrain the structural equivalence of image and text latent manifolds, we coincide persistence tracks of homology classes along with the deviating perturbation by:EQUATION)where λ is the hyper-parameter controlling the perturbation strength. We mainly focus on the 0-th homology classes (p = 0), since higher-dimensional classes significantly increase computational cost but bring almost no additional performance benefits in VLMs tuning in practice.To examine the structural equivalence constrained by our proposed homology consistency in the efficient transfer of VLMs and show its transferability, we tailor the implementation of homology consistency constraint to the two main paradigms of adapter tuning, residual blending based and key-value cache model based, respectively.Residual blending based tuning methods construct an adapter to produce learnable residuals and blend them with the pre-trained features. As a representative, TaskRes [20] adds prior-independent parameters x as a residual to the pre-trained text embeddings t to form a learnable image classifier t ′ = t + αx, where α is a scaling factor, and updates the classifier by cross-entropy loss L CE .In this paradigm, given frozen pre-trained image embeddings X and tunable text embeddings T , our method can naturally construct a filtration on X to capture the image persistence tracks and further obtain their text tunable counterparts, and then employ the persistence track coincidence with deviating perturbation through L HC (Γ, X, T ) in downstream tuning together with L CE .Key-value cache based tuning methods build adapters via a key-value cache model with pre-trained embeddings of all training images as keys and one-hot encodings of corresponding labels as values.The image keys can be unfreezed as learnable parameters. As a representative, to recognize an image x, Tip-Adapter-F [21] first measures its affinity weights with the cached keys F train byA = exp -β 1 -xF T train, then aggregates the cached values L train with weights A as a prediction AL train , and further combines AL train with the similarity between image x and pre-trained text category embeddings W c as final classification logit αAL train + xW T c /τ CLIP in L CE . In this paradigm, the cache-based adapter represents the categories as one-hot label encodings L train , that is, the L train is the only textual representation of categories in this adapter. To be comparable to label encodings, we regard the affinity weights A of an image as a sparse visual representation of this image. For implementing homology consistency, we first capture the birth and death edges of image homology classes based on pre-trained image embeddings, then replace end-point images of the edges with their corresponding affinity weights A or label encoding L train and follow Eq. 4 to construct tra p (µ, A) and tra p (µ, L train ) analogically. We implement HC as L HC (Γ, A, L train ) by taking tra p (µ, A) and tra p (µ, L train ) as proxies for original image and text tracks.Optimization. Efficient transfer learning commonly adopts the cross-entropy loss L CE between labels and the predicted class probability to tune learnable parameters, e.g., θ, on downstream classification tasks. We have two ways to integrate the gradient from our proposed HC constraint into parameter tuning. (1) We alter the cross-entropy gradient ∇ θ L CE toward the direction of HC by adding the gradient of HC constraint to ∇ θ L CE with a constant factor η by: ∇ θ L = ∇ θ L CE +η∇ θ L HC . (2) For Tip-Adapter-F, due to its slightly complex hyper-parameter configuration, fixing a constant factor is difficult to control the contribution of each gradient. We adaptively keep the gradients at the same order of magnitude through scaling the gradient of homology consistency constraint based on gradient norm by∇ θ L = ∇ θ L CE + ω ∥∇ θ L CE ∥2 ∥∇ θ L HC ∥2 ∇ θ L HC in optimization.4 Experiments 
experimental settings: 
 Datasets. Following previous efficient transfer learning works, we conduct the few-shot learning evaluation on 11 benchmark datasets including Caltech101 [48] , DTD [49] , EuroSAT [50] , FGVCAircraft [51] , Flowers102 [52] , Food101 [53] , ImageNet [54] , OxfordPets [55] , StanfordCars [56] , SUN397 [57] and UCF101 [58] . These datasets cover a wide range of visual recognition on generic objects, fine-grained categories, scenes, actions, etc. We sample 1, 2, 4, 8 and 16 shots per class, respectively, for model training and evaluate on full test sets. In addition, we evaluate the domain generalization performance of our method with the ImageNet [54] as source and its variants ImageNetV2 [59] , ImageNet-Sketch [60] , ImageNet-A [61] and ImageNet-R [62] as targets.Implementation Details. In addition to HC, following TaskRes, we augment the original pre-trained text features into ones tuned on downstream tasks in HC * . In implementing HC / HC * on TaskRes (HC-TR / HC * -TR), we set the values of η and λ on different datasets according to the procedure of factor determination in ablation studies. We set the scaling factor α to 1 for all datasets. The training batch size is 256. We employ the Adam optimizer with an initial learning rate of 1e -4 Few-shot Learning. We validate the effectiveness of our proposed HC constraint on 11 benchmark datasets using representative adapter tuning methods, the residual blending based TaskRes and key-value cache based Tip-Adapter-F, as baselines. The experimental results are shown in Fig. 3 , from which it can be observed that our method brings performance improvements for 1 to 16 shots consistently. Taking the 16 shots average accuracy as an example, HC-TR exceeds TaskRes by 0.40% and HC-TAF exceeds Tip-Adapter-F by 1.04%. At 16-shot, HC-TAF outperforms Tip-Adapter-F by 0.61% on ImageNet. For the challenging fine-grained classification dataset FGVCAircraft, the proposed HC constraint gains 1.38% and 1.92% on TaskRes and Tip-Adapter-F, respectively. The further improvements of HC * -TR and HC * -TAF yielded by pre-tuned text classifier suggest that the HC-equipped tuning can still significantly benefit from representation enhancement. The performance comparison of our HC constraint with the state-of-the-art adapter tuning methods on ImageNet is shown in Tab. 1. APE-T performs markedly best at 1 and 2 shots, whereas the models constrained by homology consistency, e.g., HC * -TAF, exceed it and other state-of-thearts on the 8-/16-shot setting in more sample cases. This arises from the fact that the efficacy of homology consistency in visionlanguage aligning depends primarily on the modeling capability of the simplicial complex builded on training data towards the topological structure of latent space. The denser the data samples, the more sufficiently they summarize the structure, and the more effective the homology consistency constraint is. It is worth noting that HC boosts baselines to achieve stateof-the-art without introducing any additional training parameters. We respectively take TaskRes and Tip-Adapter-F as representative methods of residual blending based (e.g., CLIP-Adapter, TaskRes, GraphAdapter) and key-value cache based (e.g., Tip-Adapter-F, APE-T) adapter tuning for applying HC constraint. The HC constraint is verified to be effective on these two baselines by extensive experiments, and can theoretically be extended to other VLMs efficient transfer learning methods. 2) Conversely, if we do not guide the coincidence of persistence tracks and only apply DP to track end-point samples, the original performance of baselines will be damaged. This is because DP plays a role of regularization term for TC. Without coinciding persistence tracks by TC, not only the direct constraint on the structural equivalence of latent manifolds is lost, but also the only DP will cause the track end-point samples to randomly deviate from the hetero-modal training samples, which interferes with the downstream tuning and thus performance drop. Scaling factors of HC gradients in tuning. The hyper-parameters η and ω scale the weight of homology consistency constraint in tuning and λ controls the strength of deviating perturbation within the constraint (Sec. 3.3). We adopt the constant scaling (with η, λ) for TaskRes and adaptive scaling (with ω, λ) for Tip-Adapter-F. In general, performance first increases and then decreases as η (or ω) and λ increase, and the optimal performance is achieved by a specific combination of their values. Taking 16-shot ImageNet as an example, as shown in Tab. 4, the optimal η and λ are 15 (i.e., 0.15/τ CLIP ) and 2.5 on HC-TR, the optimal ω and λ are 0.4 and 100 on HC-TAF. In implementation, we first determine the values of scaling factors at 16-shot, and then migrate them to HC * -TR / HC * -TAF and other few-shot settings. The setting of hyper-parameter factors on other datasets follows a similar procedure.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Our paper provides a detailed account of implementation details and hyperparameter settings of our method for reproducing. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusions, limitations and future work: 
 In this work, we study the generalizability of image-text alignment adjusting in the efficient transfer of VLMs under a low-data regime. We propose to explicitly constrain the structural equivalence of image and text latent manifolds in downstream tuning and design a theoretically well-founded homology consistency constraint based on persistent homology for VLMs transfer. Our method constraint coincides the persistences of homology classes of topological features between image and text manifolds and applies a deviating perturbation to generalize the persistence coincidence to unseen data. Extensive experiments demonstrate the effectiveness and robustness of our method.Limitations and Future Work. In structural equivalence constraint, we do not explore the effects of higher-dimensional homology classes in depth. Besides, following previous work on efficient transfer learning for VLMs, we only apply the proposed homology consistency constraint on a series of few-shot recognition tasks. As a next step, we will further extend the application scenarios of our method to other VLMs downstream tasks.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We will make all code avaliable when paper is accepted. Guidelines: • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: We believe that our study will not pose any negative societal due to its theoretical nature. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our paper does not contain data or models that are at high risk of misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite the datasets used in our paper and give a brief introduction. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
 
 Appendix: A.1 Simplicial Complex A simplicial complex consists of a set of simplices, such as points, line segments, triangles, and p-dimensional analogues. A simplex σ with dimension p is the convex hull of a set of p + 1 affinely independent points (x 0 , . . . , x p ). For 0 ≤ p ′ ≤ p, a p ′ -face of σ is a p ′ -simplex that is the convex hull of a non-empty subset. A simplicial complex K consists of a set of finite simplices that satisfy: K contains every face of each simplex in K and for any two simplices σ, τ ∈ K, their intersection σ ∩ τ is either empty or a face of both σ and τ . The dimension of K is the maximum dimension of any simplex in K. Given a finite data point set X in metric space (M, d) and a real a > 0, the Vietoris-Rips complex K Rips a (X) is the set of simplices σ such that d(x, x ′ ) ≤ a for every pair of vertices of σ, and the Čech complex K Cech a (X) is the set of simplices σ such that the closed balls centering its vertices have a non-empty interaction.In complex K, let the number of p-simplices be m p , a p-chain c is a formal sum of p-simplices added with some coefficients, i.e., c = mp i=1 α i σ i . In particular, the p-chains with Z 2 -additions form a chain group C p (K) where the identity is the chain 0 = mp i=1 0σ i , and since c+c = 0, the inverse of a chain c is itself. When K is clear from the context, C p (K) can also be notated as C p . Define a boundary operator ∂ p on p-simplex σ as a map that sends σ to the (p -1)-chain consisting of σ's (p -1)-faces referred as σ 's boundary, i.e., ∂ p σ = p i=0 (x 0 , . . . , xi , . . . , x p ) where xi indicates that x i is omitted. The boundary of a vertex is empty. The ∂ p induces a homomorphism ∂ p : C p → C p-1 that produces a (p -1)-chain when extended to a p-chain c through ∂ p c = mp i=1 α i (∂ p σ i ). Applying ∂ p to the chain groups, a sequence of homomorphism, 0 = C k+1 ∂ k+1 ---→ C k ∂ k -→ C k-1 • • • C 1 ∂1 -→ C 0 ∂0 -→ C -1 = 0,For data point set X in metric space (M, d), define a function f : M → R, f ((x 0 , . . . , x p )) = max i,j∈{0,...,p} f ((x i , x j )) on simplices. Then, given a sequence of thresholds a 1 ≤ a 2 ≤ . . . , ≤ a n , the growing sublevel sets f -1 (-∞, a] at these values give rise to a nested sequence of subcomplexes,K a1 ⊆ K a2 ⊆ • • • ⊆ K an , called a filtration F. The inclusions in filtration F induce a sequence of homomorphisms h i,j p , H p F : 0 = H p (K 0 ) → H p (K 1 ) → • • • → H p (K i ) → h i,j p • • •→ H p (K j ) → • • • → H p (K n ) = H p (K), called a homology module. The images of the homomorphisms h i,j p are persistent homology groups H i,j p for the module, H i,j p = im h i,j p , for 0 ≤ i ≤ j ≤ n. The non trivial elements of persistent homology groups H i,j p consist of homology classes that survive from K i to K j , i.e., the homology classes that do not quotient out by boundaries, which implies thatH i,j p = Z p (K i ) / (B p (K j ) ∩ Z p (K i )). A non-trivial homology class ϵ ∈ H p (K a ) is born at K i , if ϵ ∈ H i,a p but ϵ / ∈ H i-1,a p. Likewise, the ϵ dies entering K j , if ϵ ∈ H a,j-1 p but ϵ / ∈ H a,j p . When a class dies, it may be merged with several classes and raises a new birth. The persistence of homology class that is created at K i and destoryed at K j is defined as a i -a j . The homology classes that never die are set to remain alive till a n+1 = a ∞ = ∞. As a visual representation, the persistence diagram Dgm p (F) of filtration F draws the paired birth at a i and death at a j that bound the survival interval of one or more homology classes as a point (a i , a j ) on the extended plane R 2 := (R ∪ {±∞}) 2 .Topology specifies how points are connected and is a key geometric signature of metric spaces. Topological equivalence can be formalized by continuous functions that map points from one space to the other while preserving the connectivity. A straightforward equivalence is that the points as well as their neighborhoods in two spaces are in one-to-one correspondence, called being homeomorphic [24, 26] , which requires a bijecitve map with continuous inverse. Two homeomorphic spaces share the exact same topological properties, i.e., from the topological point of view, the two are completely consistent. Another less rigid relation is homotopy equivalent [24, 26] . Formally, given spaces X and Y, two maps f 0 , f 1 : X → Y are homotopic if they can be joined by a continuous functionF : X × [0, 1] → Y such that F (x, 0) = f 0 (x) and F (x, 1) = f 1 (x) for every x ∈ X, denoted as f 0 ≃ f 1 .The spaces X and Y are homotopy equivalent if there exist two maps f : X → Y and g : Y → X such that f • g ≃ id Y and g • f ≃ id X , which suggests an intuitive fact that space X is homotopy equivalent to Y if and only if both X and Y are deformation retracts of a common space Z.Vision and language are key information modalities for human cognition and have intrinsic correspondence on semantics. Observe that similar image scenes tend to be described by synonymic texts, that is, the connectivity of image data samples is preserved after mapping them to the text latent manifold, and vice versa. This preservation of connectivity suggests that the two manifolds have similar topological structures. Besides, image and text data are both relevant and complementary concrete expressions of the real world. The latent manifolds in which they reside can be seen as deformation retracts of the general world knowledge space. Thus, it is rational to consider that the image latent manifold and text latent manifold are homotogy equivalent. As a topological invariant of homotogy equivalent spaces, the homology is a viable tool for quantifying and aligning the structure of image and text latent manifolds. Answer: [Yes] Justification: We discuss the homotopy equivalent assumption underpinning our method in Appendix A.4 and explain the rationale for such an assumption. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.https://github.com/htzhang-code/HC 


Decision: This paper suggests a novel homology consistency constraint to align language and vision representations for VLM transfer learning. Adding these constraints to two baseline transfer learning methods (TaskRes and Tip-Aligner-F) yields consistent improvements.  The reviewers agreed this approach has potential, but voiced some concerns regarding the experiments, mainly that they are only for few-shot classification, and that in some scenarios the results are not state of the art. Nonetheless, I feel this paper should be accepted: (a) because of the original idea of using persistence homology for VLMs and (b) because the topological regularization does consistently improve upon the backbone methods

Reviews: review: paper_summary: paper_summary: This paper proposes Homology Consistency (HC) constraint for transfer learning on VLMs, and it explicitly constrains the correspondence of image and text latent manifolds by structural equivalence based on persistent homology in downstream tuning.
strength_weakness: strength_weakness: 1. The paper lacks discussions on the computational cost of the proposed techniques.

2. The proposed method for constraining the structural equivalence of image and text latent manifolds seems generalizable to other learning tasks for vision-language models. However, the proposed method is only evaluated for few-shot learning of vision language models.

3. Although the model outperforms other methods in most cases, the improvements are relatively marginal.

4. The paper only applies the method to a limited number of adapter models (TaskRes and Tip-Adapter-F).
questions: questions: see weaknesses.
limitations: limitations: The limitations are not discussed sufficiently (see weaknesses).
paper_summary: paper_summary: The paper introduces a Homology Consistency (HC) constraint for efficient transfer learning on vision-language models (VLMs), ensuring task-specific image-text alignment while preserving general knowledge by using structural equivalence based on persistent homology. This approach mimics the topology of latent manifolds and tracks the persistence of topological features.
strength_weakness: strength_weakness: 1. The performance of the proposed method is worse than the baseline method in low-shot (1-shot and 2-shot) tasks. 
2. The improvement in Table 2 is marginal. Is the comparison fair with the same random seed? How many runs did you conduct? Could the authors also report the standard deviation of the score? 
3. Moreover, is 16-shot common in this benchmark? 16 shot seems a lot in few-shot learning. 
4. Can you also elaborate more why with only DP, the performance drops in Table 3? 
5. In addition, could you elaborate more why choosing 0-th homology classes? What are the potential effects of using other homology classes?
questions: questions: 1. Can you visualise the topology of the data before and after adaptation with the HC constraint? It will be very interesting to see how actually the HC constraint can preserve the topology of the manifold during transfer learning.
limitations: limitations: My concern is the performance improvement is marginal and limited to more shots setting (16-shot).
paper_summary: paper_summary: The paper identifies a key issue with existing methods for tuning pre-trained vision-language models to downstream tasks with limited data: they adjust the alignment between image and text based solely on observed samples, which may not generalize well beyond the training data. To address this issue, the paper proposes a novel constraint from the perspective of topological data analysis.

This constraint employs persistent homology to ensure the structural equivalence of image and text latent manifolds during tuning.
strength_weakness: strength_weakness: The paper does not adequately discuss how it relates to existing image and text alignment techniques, including those based on distance metrics, mutual information, adversarial training, and attention mechanisms. This lack of comparative analysis creates a gap in fully appreciating the distinctive contributions and potential advantages.
questions: questions: 1.  Could you please provide insights into the fundamental differences and advantages of topological data analysis in your method over other alignment methods?

2.  Where do you foresee potential challenges in extending your proposed method to tasks outside the few-shot learning domain, particularly in scenarios such as zero-shot learning, or applications involving detection and segmentation?

3.  Could you elaborate on how incorporating higher-dimensional homology classes into the tuning process might impact the model's performance and behavior, beyond the computational cost?
limitations: limitations: Yes
paper_summary: paper_summary: A Homology Consistency (HC) constraint for efficient transfer on VLMs is proposed in this paper, which explicitly constrains the correspondence of image and text latent manifolds through structural equivalence based on persistent homology in downstream tuning.
The proposed method tracks the persistence of the homology classes of topological features across multiple scales and guide the directions of persistence tracks in image and text manifolds to coincide each other. Additionally, a deviating perturbation is applied to generalize the persistence coincidence to unseen data. Experiments on recognition and generalization tasks show the superior performance.
strength_weakness: strength_weakness: The hyper-parameters η, λ, ω should be determined at 16 shots and then migrated to other few-shot settings. If the number of samples is less than 16, how should the aforementioned hyper-parameters be set, and will there be a significant difference in performance?
questions: questions: The hyper-parameters η, λ, ω should be determined at 16 shots and then migrated to other few-shot settings. If the number of samples is less than 16, how should the aforementioned hyper-parameters be set, and will there be a significant difference in performance?
limitations: limitations: The experiments only conducted on recognition tasks.

Comments: title: Please address rebuttal
comment: Dear reviewers:

Thank you for the comments on our paper. We are grateful for your recognition of our motivation and technical contributions, and have provided detailed responses to all concerns in the rebuttal. We would appreciate it if you kindly let us know of any other concerns you may have and if we can be of any further assistance in clarifying them.  If our responses address your concerns, we hope that you can kindly consider updating the score.

Thank you once again for your contribution to the development of our paper.
Dear reviewers:

Thank you for the comments on our paper. We are grateful for your recognition of our motivation and technical contributions. Specifically, the reviewers encouraged that our method is theoretically well-founded (Reviewers yCSE, KyVk and zhsp), well-motivated (Reviewers Ls33) and has interesting perspective (Reviewer KyVk), our paper has a good theoretic support (Reviewer Ls33), is well-written (Reviewers KyVk and zhsp) and easy to follow-up (Reviewer KyVk).

We have provided detailed responses to all concerns in the rebuttal. We find that the more common concerns lie in the evaluation approach (Reviewers yCSE and zhsp) and performance improvement (Reviewers yCSE and Ls33). In our rebuttal, we clarify in detail that our evaluation scheme is the widely-acknowledged standard in the field of VLMs efficient transfer learning and commonly-used in previous work, and our performance improvement is relatively very competitive in the same field. 

We appreciate all valuable comments. Please let us know if you have additional questions so that we can address them during the discussion period. If our responses address your concerns, we hope that you can kindly consider updating the score.

Thank you all again.
Dear reviewers,

Thank you for your reviews. The authors have tried to address your concerns in the rebuttal. If you have not already done so, please carefully read their rebuttal, and let them know what you think, and whether there is any more clarifications you require. Note that author-reviewer discussion ends on August 13th.
Thanks!
the AC