Title: Homology Consistency Constrained Efficient Tuning for Vision-Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Efficient Transfer Learning

Vision-Language Models

Persistent Homology

Topological Data Analysis
Abstract: Efficient transfer learning has shown remarkable performance in tuning large-scale vision-language models (VLMs) toward downstream tasks with limited data resources. The key challenge of efficient transfer lies in adjusting image-text alignment to be task-specific while preserving pre-trained general knowledge. However, existing methods adjust image-text alignment merely on a set of observed samples, e.g., data set and external knowledge base, which cannot guarantee to keep the correspondence of general concepts between image and text latent manifolds without being disrupted and thereby a weak generalization of the adjusted alignment. In this work, we propose a Homology Consistency (HC) constraint for efficient transfer on VLMs, which explicitly constrains the correspondence of image and text latent manifolds through structural equivalence based on persistent homology in downstream tuning. Specifically, we build simplicial complex on the top of data to mimic the topology of latent manifolds, then track the persistence of the homology classes of topological features across multiple scales, and guide the directions of persistence tracks in image and text manifolds to coincide each other, with a deviating perturbation additionally. For practical application, we tailor the implementation of our proposed HC constraint for two main paradigms of adapter tuning. Extensive experiments on few-shot learning over 11 datasets and domain generalization demonstrate the effectiveness and robustness of our method.

Homology Consistency Constrained Efficient Tuning for Vision-Language Models

Abstract

1 Introduction

Large-scale vision-language models (VLMs) such as CLIP [1] and ALIGN [2] trained on web-scale data have learned broad visual concepts and demonstrated promising generalization capability on a wide range of downstream tasks, such as classification [3, 4], detection [5, 6] and segmentation [7, 8]. In transferring to downstream tasks with limited data resources, the conventional full fine-tuning on VLMs often forgets the general knowledge learned in pre-training and falls into overfitting. To mitigate this, how to efficiently transfer the knowledge from pre-trained VLMs to downstream tasks in a low-data regime has been intensively studied.

To possess both task-specific knowledge exploration and general knowledge preservation, efficient transfer learning proposes to adapt VLMs to fit downstream tasks by tuning a few parameters, which mainly exists in two paradigms: prompt tuning and adapter tuning. Prompt tuning methods adapt VLMs toward downstream tasks by introducing learnable prompts on the input side. Topics in this branch include the configuration of learnable prompts [9-13] and injecting semantic priors, such as external knowledge [14, 15], category distribution [16] or visual diversity [17, 18], in tuning. As a promising alternative, adapter tuning inserts a learnable lightweight adapter into the frozen pre-trained VLMs on the output side and the insertion manner allows modifying flexibly. By residual blending, CLIP-Adapter [19] tuned the image or text embeddings by appending a learnable bottleneck layer

to the frozen encoder. TaskRes [20] added learnable parameters as prior-independent residuals to text embeddings. Another line is based on key-value cache model. Tip-Adapter [21] proposed to construct an adapter with training images as keys and one-hot label encodings as values for image recognition. Some work further enhanced the cache model by discriminative prior refinement [22] or knowledge augmentation [23]. For efficient transfer, existing methods mainly focus on how to configure tunable parameters or the leverage of external prior knowledge in tuning.

To this end, we propose to explicitly constrain the equivalence of image and text latent manifolds in transferring VLMs toward downstream tasks. We study the structure of the image and text latent manifolds from the lens of topological data analysis [24-26]. Topology encodes the connectivity of a space to describe its underlying structure, and the preservation of topology between spaces is fundamental for their structural equivalence. Topological data analysis employs homology groups to quantify the topological features of manifold structure, such as connected components, loops, voids, and higher-dimensional holes, as homology classes and tracks the survival of the topological features across multiple scales via persistent homology [26] to capture their size and position, which summaries the global shape of manifold. These topology insights provide an avenue to achieve a structural equivalence of image and text latent manifolds.

In this work, we propose a Homology Consistency (HC) constraint for efficient transfer on VLMs, which constrains the structural equivalence of image and text latent manifolds based on persistent homology in downstream tuning. Given image-text data samples, we construct simplicial complex on the top of data to mimic the topological structure of latent manifolds, and induce a nested sequence of subcomplexes called filtration. Through the filtration, we capture the persistence of homology classes from their appearance to non-existence and guide the homology persistences of image and text manifolds to be consistent. Specifically, we locate the births and deaths of homology classes in latent manifolds and track these homology persistences, then guide the directions of persistence tracks in image and text manifolds to coincide each other so as to achieve a homology-level structural equivalence. Additionally, we apply a deviating perturbation to persistence-related text samples to encourage their respective semantically related images to be distributed uniformly relative to them in embedding, in order to enhance the generalization of the track coincidence in adjusting image-text alignment. Further, we tailor the implementation of the proposed HC constraint for the main residual blending and key-value cache based paradigms of adapter tuning. Extensive experiments on few-shot learning over 11 benchmarks and domain generalization demonstrate the effectiveness and robustness of HC constraint in efficient transfer learning on VLMs.

Our main contributions are summarized as follows:

We propose to explicitly constrain the structural equivalence of image and text latent manifolds in efficient transfer on VLMs, to improve the generalization of downstream image-text alignment adjusting beyond data samples in a low-data regime.

We propose a theoretically well-founded homology consistency (HC) constraint based on persistent homology for efficient transfer on VLMs. We coincide the persistences of homology classes of topological features between image and text manifolds, and apply a deviating perturbation to generalize the persistence coincidence to unseen data.

We tailor the implementation of the proposed HC constraint for the two main paradigms of adapter tuning respectively, showing the transferability of our method.

We evaluate the proposed HC constraint on few-shot classification over 11 popular benchmarks. The extensive experiments demonstrate that HC constraint can boost the performance of baselines significantly and achieve state-of-the-art.

2 Related Work

Efficient Transfer Learning. To better transfer VLMs to downstream tasks especially with limited target domain data, a lot of research on efficient transfer learning has been done, which mainly exists in two types, prompt tuning and adapter tuning. Advanced than manual prompt that demands domain expertise to develop suitable format, prompt tuning methods design learnable prompts to adapt VLMs on downstream data. As a pioneer work, CoOp [9] for the first time composed prompts by concatenating text category embedding and learnable context vectors. The learnable prompts can be configured in text input [9, 10], image input [11] or jointly both [12, 13]. A line of work focuses on injecting semantic priors, such as category-related external knowledge [14, 15], category embedding distribution [16] and the diversity of visual concepts [17, 18], in prompt tuning. In another branch, adapter tuning methods insert a learnable lightweight adapter module into the frozen pre-trained VLMs and show excellent performance. The adapter architecture and insertion manner allow for flexible modifications. There are two main paradigms: residual blending and key-value cache based. By residual blending, CLIP-Adapter [19] appended learnable bottleneck layer to frozen encoder to tune embeddings. TaskRes [20] added a set of prior-independent parameters to frozen text category embeddings to obtain an image classifier. GraphAdapter [27] proposed to learn downstream knowledge with inter-class relationship of image and text samples. Based on the key-value cache model, Tip-Adapter [21] constructed an adapter with training images and label encodings as keyvalue to recognize query images. APE [22] refined the cache model by visual discrimination priors. CaFo [23] cascaded diverse external knowledge from DINO [28], DALL-E [29], and GPT-3 [30] to assist recognition. However, although existing methods have attained remarkable achievements in VLMs transfer, they adjust the semantic alignment [31-33] of image and text latent manifolds toward downstream tasks merely on observed samples, e.g. , data sets and external knowledge bases, and lack insight into underlying manifold structure, which may cause unguaranteed generalization beyond the data samples. In this work, we propose to explicitly constrain the structural equivalence of image and text latent manifolds in downstream tuning to facilitate the transfer of VLMs.

Topological Data Analysis in Machine Learning. The area of topological data analysis [26] infers the topological structure of data spaces using algebraic tools such as persistent homology, and has been applied in many fields of machine learning, e.g. , image segmentation [34-37], graph machine learning [38-40], molecular representation [41, 42], point cloud analysis [43, 44], etc. For instance, in image segmentation, [35] proposed to drive the segmentations to contain the specified topological features without requiring ground-truth labels. [36] used discrete Morse theory and persistent homology to learn the structural representation of images for fine-scale structure segmentation. In graph machine learning, [39] integrated vertex- and edge-level topological features into message-passing graph neural networks to boost their expressive power. In point cloud analysis, [44] developed a learnable filtration on point clouds to obtain adaptive topological features for given tasks. Besides, [45] preserved the topological structures of input space into latent space of autoencoders by aligning topologically relevant distances. [46] applied representation topology divergence [47] in dimensionality reduction to force closeness on topological structures. How to analyse the structure of data spaces for semantic alignment in vision-language tasks through topology remains under-explored in the literature. In this work, we characterize and align the structure of the image and text latent manifolds by means of persistent homology in efficient transfer learning on VLMs.

3 Methodology

3.1 Preliminaries

3.1.1 Contrastive Language-Image Pre-training (CLIP)

As a representative VLM, CLIP [1] is trained on massive image-text pairs and shows promising zero-shot performance on downstream tasks. CLIP adopts two separate encoders to embed images and texts into latent manifolds, and aligns the bi-modal embeddings by contrastive learning that forces paired image and text closer and unpaired ones away. For the transfer to image classification with N classes, CLIP obtains the class embedding w i by feeding the prompt templates, e.g. , 'A photo of a [CLASS]', filled with class name c i into text encoder, and the probability that an image x belongs to category c i can be formulated as p ( y = c i | x ) = exp ( x ⊤ w i /τ ) / ∑ N j =1 exp ( x ⊤ w j /τ ) , where the embeddings are l 2 -normalized and τ denotes a temperature hyper-parameter.

3.1.2 Persistent Homology

Discrete data points contain only observations from the latent space in which they reside and do not have interesting topology. To peek at the topological structure of latent space, we mimic its connectivity by constructing a simplicial complex with the data points as vertices. As a basic element, simplex σ with dimension p is the convex hull of a set of p +1 affinely independent points ( x 0 , . . . , x p ) . Given a finite data point set X in metric space ( M,d ) and a threshold a > 0 , the commonly used Vietoris-Rips (Rips in short) complex is defined as:

which is fully determined by the pairwise distances of X . The dimension of K a is the maximum dimension of any simplex within it. The formal sums of p -simplices added with Z 2 -additions form a chain group C p ( K a ) ( C p for brevity). Then, define a boundary operator ∂ p on p -simplex σ as a map that sends σ to the ( p -1) -chain consisting of σ 's ( p -1) -faces referred as σ 's boundary. Applying ∂ p to the chain groups can obtain a sequence of homomorphisms: C k ∂ k - → C k -1 · · · C 1 ∂ 1 - → C 0 . All p -chains whose boundaries are empty form a cycle group Z p , which is the kernel of ∂ p . The image of boundary operator ∂ p +1 on C p +1 forms a boundary group B p . Further, taking the quotient of the Z p with B p , the p -th homology group H p = Z p / B p classifies the p -cycles in Z p by collecting those cycles that differ by a boundary into the same homology class. In a topological view, the rank of homology group H p captures the number of p -dimensional holes in space.

Persistent homology offers a way to compute the quantified summary of topological structures of the latent space from sampled data. For the data point set X in space ( M,d ) , we define a function f : M → R , f (( x 0 , . . . , x p )) = max i,j ∈{ 0 ,...,p } f (( x i , x j )) on simplices. Then, given a sequence of thresholds a 1 ≤ a 2 ≤ . . . , ≤ a n , the growing sublevel sets f -1 ( -∞ , a ] at these values give rise to a nested sequence of subcomplexes, K a 1 ⊆ K a 2 ⊆ · · · ⊆ K a n , called a filtration F , as shown in Fig. 1. The inclusions in F induce:

K

K

K

a

a3

where the images of the homomorphisms h i,j p are persistent homology groups H i,j p . A non-trivial homology class ϵ ∈ H p ( K a ) is born at K i , if ϵ ∈ H i,a p but ϵ / ∈ H i -1 ,a p . Likewise, the homology class ϵ dies entering K j , if ϵ ∈ H a,j -1 p but ϵ / ∈ H a,j p . The persistence of a homology class is the lifespan from its birth to death. See Appendix A for more details.

3.2 Homology Consistency

Given a set of pre-trained image embeddings X and text embeddings T , we construct a Rips complex K a M ( X ) with the maximum pairwise distance a M of X and further derive a sublevel set filtration, F ( K ) , as the nested family of subcomplexes K a 0 ⊆ K a 1 ⊆ ·· · ⊆ K a M at the increasing scale sequence of pairwise distances { a i } i M =0 where a 0 = 0 . Then we arrive at p -th persistent homology groups that capture the survival of the homology classes of p -dimensional topological features ( e.g. , 0 -dimension: connected components, 1 -dimension: loops, 2 -dimension: voids, etc.) and pair the birth and death times of p -th homology

↑↑

unseen data

in-sample alignment driven

bymaintuningobjective

data samples

(a) Persistence Track Coincidence

(b) DeviatingPerturbation

classes, following [45]. Since the edge skeleton of Rips complex fully determines all of its simplices,

under the assumption that pairwise distances of X are unique, every birth-death time pair can be mapped back to the simplices that respectively created and destroyed the unique corresponding homology class. The schematic illustration of HC constraint is shown in Fig. 2.

Persistence Track Coincidence. Let µ be a p -th homology class that is born at a b and dies at a d , then the birth simplex β forms µ in K a b and the death simplex δ causes µ to disappear entering K a d . For example, if µ were to be 0-dimensional, it emerges at a 0 , and δ is the edge that joins it with some other point; if µ were to be 1-dimensional, then β is the edge that forms the loop corresponding to µ in K a b , and δ is the triangle that incurs the loop to be contractible in K a d . Since the Rips complex is pairwise distance based, on every p -simplex we have:

For the birth simplex β and death simplex δ , this means they will not be established until their longest edge ( x i ( β ) , x j ( β ) ) arg max f ( β ) and ( x i ( δ ) , x j ( δ ) ) arg max f ( δ ) appear at time a b and a d , which we refer to as birth edge and death edge, respectively. It can be said that the simplex β is completed by the birth edge and δ by the death edge. Such edges mark the creation and destruction of the homology class µ in the image latent manifold. Defining the direction of the edges as from i to j such that i < j in the given vertex sequence of simplices, we track the persistence direction of p -th homology class µ from its birth edge to death edge as:

Then we obtain the persistence track of µ in text latent manifold, tra p ( µ, T ) , through determining its birth and death edges directly by taking the corresponding texts of the end-points of µ 's birth and death edges in image manifold accordingly. Further, for aligning the structure of image and text latent manifolds, we guide the track coincidence (TC) of p -th homology classes between the two by:

where Γ p is the set of p -th homology classes in latent manifolds and φ is cosine similarity.

Deviating Perturbation. In a low-data regime where samples are not sufficient to fully characterize the topology of the latent manifold of interset, the sight of persistence track coinciding is confined on the given limited samples, which hinders the generalization of structural equivalence guided by track coincidence. For all end-point images of birth and death edges of p -th homology classes in Γ p and their corresponding texts, we drive every text to deviate from its semantically related images in embedding (what we consider here is that multiple images are related to the same category text in classification) without breaking the track coincidence, so as to encourage the text-related images beyond samples in latent manifold to be uniformly distributed around the text.

Specifically, since the persistence tracks of the 0 -th homology classes are exactly the death edges (birth edges are 0), their end-point samples are available from the given dataset. We quantify the degree of deviation as the similarity between the orientations of the text's embedding relative to the embeddings of its related images. Then, for 0 -th homology classes in Γ 0 , we enlarge the deviation between end-point texts and their respective semantically related images by reducing the variation in their relative orientations, by applying a deviating perturbation (DP) as:

where X Γ 0 denotes the set of all end-point images of 0 -th homology classes in Γ 0 , ( x i ) T denotes x i 's corresponding text, X ′ i ∈ X Γ 0 denotes a set of images of the same category that are semantically related to x i and φ is cosine similarity. The deviating perturbation can benefit the structural equivalence to get rid of a biased reconstruction of latent topology caused by insufficient sampling.

Homology Consistency. To constrain the structural equivalence of image and text latent manifolds, we coincide persistence tracks of homology classes along with the deviating perturbation by:

where λ is the hyper-parameter controlling the perturbation strength. We mainly focus on the 0 -th homology classes ( p = 0 ), since higher-dimensional classes significantly increase computational cost but bring almost no additional performance benefits in VLMs tuning in practice.

3.3 Implementation

To examine the structural equivalence constrained by our proposed homology consistency in the efficient transfer of VLMs and show its transferability, we tailor the implementation of homology consistency constraint to the two main paradigms of adapter tuning, residual blending based and key-value cache model based, respectively.

Residual blending based tuning methods construct an adapter to produce learnable residuals and blend them with the pre-trained features. As a representative, TaskRes [20] adds prior-independent parameters x as a residual to the pre-trained text embeddings t to form a learnable image classifier t ′ = t + αx , where α is a scaling factor, and updates the classifier by cross-entropy loss L CE .

In this paradigm, given frozen pre-trained image embeddings X and tunable text embeddings T , our method can naturally construct a filtration on X to capture the image persistence tracks and further obtain their text tunable counterparts, and then employ the persistence track coincidence with deviating perturbation through L HC (Γ , X, T ) in downstream tuning together with L CE .

Key-value cache based tuning methods build adapters via a key-value cache model with pre-trained embeddings of all training images as keys and one-hot encodings of corresponding labels as values. The image keys can be unfreezed as learnable parameters. As a representative, to recognize an image x , Tip-Adapter-F [21] first measures its affinity weights with the cached keys F train by A = exp ( -β ( 1 -xF T train )) , then aggregates the cached values L train with weights A as a prediction AL train, and further combines AL train with the similarity between image x and pre-trained text category embeddings W c as final classification logit αAL train + xW T c /τ CLIP in L CE .

In this paradigm, the cache-based adapter represents the categories as one-hot label encodings L train , that is, the L train is the only textual representation of categories in this adapter. To be comparable to label encodings, we regard the affinity weights A of an image as a sparse visual representation of this image. For implementing homology consistency, we first capture the birth and death edges of image homology classes based on pre-trained image embeddings, then replace end-point images of the edges with their corresponding affinity weights A or label encoding L train and follow Eq. 4 to construct tra p ( µ, A ) and tra p ( µ, L train ) analogically. We implement HC as L HC (Γ , A, L train ) by taking tra p ( µ, A ) and tra p ( µ, L train ) as proxies for original image and text tracks.

Optimization. Efficient transfer learning commonly adopts the cross-entropy loss L CE between labels and the predicted class probability to tune learnable parameters, e.g. , θ , on downstream classification tasks. We have two ways to integrate the gradient from our proposed HC constraint into parameter tuning. (1) We alter the cross-entropy gradient ∇ θ L CE toward the direction of HC by adding the gradient of HC constraint to ∇ θ L CE with a constant factor η by: ∇ θ L = ∇ θ L CE + η ∇ θ L HC . (2) For Tip-Adapter-F, due to its slightly complex hyper-parameter configuration, fixing a constant factor is difficult to control the contribution of each gradient. We adaptively keep the gradients at the same order of magnitude through scaling the gradient of homology consistency constraint based on gradient norm by ∇ θ L = ∇ θ L CE + ω ∥∇ θ L CE ∥ 2 ∥∇ θ L HC ∥ 2 ∇ θ L HC in optimization.

4 Experiments

4.1 Experimental Settings

Datasets. Following previous efficient transfer learning works, we conduct the few-shot learning evaluation on 11 benchmark datasets including Caltech101 [48], DTD [49], EuroSAT [50], FGVCAircraft [51], Flowers102 [52], Food101 [53], ImageNet [54], OxfordPets [55], StanfordCars [56], SUN397 [57] and UCF101 [58]. These datasets cover a wide range of visual recognition on generic objects, fine-grained categories, scenes, actions, etc. We sample 1, 2, 4, 8 and 16 shots per class, respectively, for model training and evaluate on full test sets. In addition, we evaluate the domain generalization performance of our method with the ImageNet [54] as source and its variants ImageNetV2 [59], ImageNet-Sketch [60], ImageNet-A [61] and ImageNet-R [62] as targets.

Implementation Details. In addition to HC, following TaskRes, we augment the original pre-trained text features into ones tuned on downstream tasks in HC ∗ . In implementing HC / HC ∗ on TaskRes (HC-TR / HC ∗ -TR), we set the values of η and λ on different datasets according to the procedure of factor determination in ablation studies. We set the scaling factor α to 1 for all datasets. The training batch size is 256 . We employ the Adam optimizer with an initial learning rate of 1 e -4

641

ImageNet

Caltech101

DTD

94

66

6

65

65

Score (%)

92

(%)

60

CLIP-Adapter

CLIP-Adapter

CLIP-Adapter

63

+HC-TR

TaskRes

HC-TR

TaskRes

TaskRes

HC-TR

+ HC'-TR

90

HC*-TR

1-,H

62

Tip-Adapter-F

Tip-Adapter-F

50

Tip-Adapter-F

HC-TAF

89

HC-TAF

HC-TAF

61

45

12

4

8

16

12

4

8

16

2

4

8

16

Number of labeled training examples per class

Number of labeled training examples per class

Number of labeled training examples per class

EuroSAT

FGVCAircraft

Flowers102

85

95

35

80

06

75-

CLIP-Adapter

CLIP-Adapter

CLIP-Adapter

70

TaskRes

25

TaskRes

TaskRes

HC-TR

HC-TR

80

HC-TR

65

H HC'-TR

HC′-TR

1-,H

+

+

Tip-Adapter-F

HC-TAF

20

Tip-Adapter-F

HC-TAF

75

Tip-Adapter-F

HC-TAF

60

+

+

4

8

16

12

4

8

16

4

8

16

Number of labeled training examples per class

Number of labeled training examples per class

Number of labeled training examples per class

Food101

OxfordPets

StanfordCars

79

90

75

89

78

70

Score (%)

77

CLIP-Adapter

CLIP-Adapter

CLIP-Adapter

TaskRes

TaskRes

TaskRes

T

HC-TR

HC-TR

HC-TR

76

HC′-TR

Tip-Adapter-F

86

HC′-TR

Tip-Adapter-F

60

HC*-TR

Tip-Adapter-F

1

HC-TAF

85

HC-TAF

HC-TAF

75

55

2

4

8

16

2

4

8

16

2

4

8

16

Number of labeled training examples per class

Number of labeled training examples per class

Number of labeled training examples per class

SUN397

UCF101

Averageover11datasets

72

80.0

77.5

76

70

74

75.0

Score (%)

68

72.5

72

66

CLIP-Adapter

CLIP-Adapter

CLIP-Adapter

TaskRes

TaskRes

68

TaskRes

64

HC-TR

HC*-TR

67.5

+T

T

HC-TR

HC*-TR

66

HC-TR

1-,H

Tip-Adapter-F

65.0

Tip-Adapter-F

Tip-Adapter-F

62

HC-TAF

HC-TAF

64

HC-TAF

?

62.5

2

4

8

16

12

4

8

16

62

2

4

8

16

Number of labeled training examples per class

Number of labeled training examples per class

Number of labeled training examples per class

on ImageNet and 1 e -3 on others, and the learning rates decay with cosine learning rate schedule following TaskRes. In implementing HC / HC ∗ on Tip-Adapter-F (HC-TAF / HC ∗ -TAF), ω and λ are also set according to the factor determination procedure in ablation studies. The λ here is significantly larger than the above because the magnitude difference between the ∇ θ L TC and ∇ θ L DP here is relatively large. The training batch size is 256 . Following Tip-Adapter-F, we employ the AdamW optimizer with a cosine annealing scheduler. We set initial learning rate as 1 e -3 . All experiments are conducted on a single NVIDIA A40 GPU. Note that the experiments on baseline and with the HC / HC ∗ are conducted in the same setting for fair comparison. The code is publicly available 2 .

4.2 Performance Analysis

Few-shot Learning. We validate the effectiveness of our proposed HC constraint on 11 benchmark datasets using representative adapter tuning methods, the residual blending based TaskRes and key-value cache based Tip-Adapter-F, as baselines. The experimental results are shown in Fig. 3, from which it can be observed that our method brings performance improvements for 1 to 16 shots

consistently. Taking the 16 shots average accuracy as an example, HC-TR exceeds TaskRes by 0 . 40% and HC-TAF exceeds Tip-Adapter-F by 1 . 04% . At 16-shot, HC-TAF outperforms Tip-Adapter-F by 0 . 61% on ImageNet. For the challenging fine-grained classification dataset FGVCAircraft, the proposed HC constraint gains 1 . 38% and 1 . 92% on TaskRes and Tip-Adapter-F, respectively. The further improvements of HC ∗ -TR and HC ∗ -TAF yielded by pre-tuned text classifier suggest that the HC-equipped tuning can still significantly benefit from representation enhancement.

The performance comparison of our HC constraint with the state-of-the-art adapter tuning methods on ImageNet is shown in Tab. 1. APE-T performs markedly best at 1 and 2 shots, whereas the models constrained by homology consistency, e.g. , HC ∗ -TAF, exceed it and other state-of-thearts on the 8-/16-shot setting in more sample cases. This arises from the fact that the efficacy of homology consistency in visionlanguage aligning depends primarily on the modeling capability of the simplicial complex builded on training data towards the topological structure of latent space. The

denser the data samples, the more sufficiently they summarize the structure, and the more effective the homology consistency constraint is. It is worth noting that HC boosts baselines to achieve stateof-the-art without introducing any additional training parameters. We respectively take TaskRes and Tip-Adapter-F as representative methods of residual blending based ( e.g. , CLIP-Adapter, TaskRes, GraphAdapter) and key-value cache based ( e.g. , Tip-Adapter-F, APE-T) adapter tuning for applying HC constraint. The HC constraint is verified to be effective on these two baselines by extensive experiments, and can theoretically be extended to other VLMs efficient transfer learning methods.

Domain Generalization. We investigate the generalization ability of the models constrained by HC under domain shift. We train the models on ImageNet with 16 shots and test the trained models on ImageNet variant datasets ImageNet-V2, ImageNet-Sketch, ImageNet-A and ImageNet-R. As

shown in Tab. 2, HC-TR and HC-TAF outperform their respective baselines on the source dataset ImageNet across four different visual backbones ResNet-50, ResNet-101, ViT-B/32 and ViT-B/16. The performance of HC-TR is slightly inferior to that of TaskRes on ImageNet-Sketch, -A and -R with backbones ResNet-50, ResNet-101 and ViT-B/32, which concurs with TaskRes's reported source-overfitting pitfall. Nonetheless, HC-TR outperforms TaskRes with the more representative ViT-B/16. HC-TAF shows better generalization than Tip-Adapter-F in almost all target domains with various backbones. Experiments demonstrate that the performance improvement is not reliant on the shortcut to overfit the seen data domain.

4.3 Ablation Studies

Constraint terms: track coincidence and deviating perturbation. The homology consistency constraint consists of track coincidence and deviating perturbation, and the effect of homology consistency constraint in efficient transfer comes from their collaboration. Here, we investigate the individual roles of track coincidence and deviating perturbation in tuning. The ablation on ImageNet dataset is shown in Tab. 3, from which we can observe that (1) Without the generalization enhancement in latent spaces brought by DP, the performance

of TC alone decreases relative to full homology consistency constraint. (2) Conversely, if we do not guide the coincidence of persistence tracks and only apply DP to track end-point samples, the original performance of baselines will be damaged. This is because DP plays a role of regularization term for TC. Without coinciding persistence tracks by TC, not only the direct constraint on the structural equivalence of latent manifolds is lost, but also the only DP will cause the track end-point samples to randomly deviate from the hetero-modal training samples, which interferes with the downstream tuning and thus performance drop.

Scaling factors of HC gradients in tuning. The hyper-parameters η and ω scale the weight of homology consistency constraint in tuning and λ controls the strength of deviating perturbation within the constraint (Sec. 3.3). We adopt the constant scaling (with η , λ ) for TaskRes and adaptive scaling (with ω , λ ) for Tip-Adapter-F. In general, performance first increases and then decreases as η (or ω ) and λ increase, and the optimal performance is achieved by a specific combination of their values. Taking 16-shot ImageNet as an example, as shown in Tab. 4, the optimal η and λ are 15 ( i.e. , 0 . 15 /τ CLIP) and 2 . 5 on HC-TR, the optimal ω and λ are 0 . 4 and

100 on HC-TAF. In implementation, we first determine the values of scaling factors at 16-shot, and then migrate them to HC ∗ -TR / HC ∗ -TAF and other few-shot settings. The setting of hyper-parameter factors on other datasets follows a similar procedure.

5 Conclusions, Limitations and Future Work

In this work, we study the generalizability of image-text alignment adjusting in the efficient transfer of VLMs under a low-data regime. We propose to explicitly constrain the structural equivalence of image and text latent manifolds in downstream tuning and design a theoretically well-founded homology consistency constraint based on persistent homology for VLMs transfer. Our method constraint coincides the persistences of homology classes of topological features between image and text manifolds and applies a deviating perturbation to generalize the persistence coincidence to unseen data. Extensive experiments demonstrate the effectiveness and robustness of our method.

Limitations and Future Work. In structural equivalence constraint, we do not explore the effects of higher-dimensional homology classes in depth. Besides, following previous work on efficient transfer learning for VLMs, we only apply the proposed homology consistency constraint on a series of few-shot recognition tasks. As a next step, we will further extend the application scenarios of our method to other VLMs downstream tasks.

Acknowledgments and Disclosure of Funding

This research is supported by Artificial Intelligence-National Science and Technology Major Project 2023ZD0121200, National Science Fund for Excellent Young Scholars No. 62222212 and National Natural Science Foundation of China No. 62336001.

References

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning , pages 4904-4916. PMLR, 2021.

Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6995-7004, 2021.

Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8552-8562, 2022.

Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16793-16803, 2022.

Hengcan Shi, Munawar Hayat, Yicheng Wu, and Jianfei Cai. Proposalclip: Unsupervised opencategory object proposal generation via exploiting clip cues. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9611-9620, 2022.

Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 18082-18091, 2022.

Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11686-11695, 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16816-16825, 2022.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision , pages 709-727. Springer, 2022.

Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19113-19122, 2023.

Sheng Shen, Shijia Yang, Tianjun Zhang, Bohan Zhai, Joseph E. Gonzalez, Kurt Keutzer, and Trevor Darrell. Multitask vision-language prompt tuning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 5656-5667, 2024.

Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledgeguided context optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6757-6767, 2023.

Baoshuo Kan, Teng Wang, Wenpeng Lu, Xiantong Zhen, Weili Guan, and Feng Zheng. Knowledge-aware prompt tuning for generalizable vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15670-15680, 2023.

Eulrang Cho, Jooyeon Kim, and Hyunwoo J Kim. Distribution-aware prompt tuning for visionlanguage models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22004-22013, 2023.

Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5206-5215, 2022.

Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Plot: Prompt learning with optimal transport for vision-language models. In International Conference on Learning Representations , 2023.

Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision , pages 1-15, 2023.

Tao Yu, Zhihe Lu, Xin Jin, Zhibo Chen, and Xinchao Wang. Task residual for tuning visionlanguage models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10899-10909, 2023.

Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In European Conference on Computer Vision , pages 493-510. Springer, 2022.

Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, and Peng Gao. Not all features matter: Enhancing few-shot clip with adaptive prior refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2605-2615, 2023.

Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Yu Qiao, Peng Gao, and Hongsheng Li. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15211-15222, 2023.

Allen Hatcher. Algebraic Topology . Cambridge University Press, 2002.

James R Munkres. Elements of algebraic topology . CRC press, 2018.

Tamal Krishna Dey and Yusu Wang. Computational topology for data analysis . Cambridge University Press, 2022.

Xin Li, Dongze Lian, Zhihe Lu, Jiawang Bai, Zhibo Chen, and Xinchao Wang. Graphadapter: Tuning vision-language models with dual knowledge graph. Advances in Neural Information Processing Systems , 36, 2023.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning , pages 8821-8831, 2021.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Zheren Fu, Zhendong Mao, Yan Song, and Yongdong Zhang. Learning semantic relationship among instances for image-text matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15159-15168, 2023.

Huatian Zhang, Lei Zhang, Kun Zhang, and Zhendong Mao. Identification of necessary semantic undertakers in the causal view for image-text matching. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 7105-7114, 2024.

Zheren Fu, Lei Zhang, Hou Xia, and Zhendong Mao. Linguistic-aware patch slimming framework for fine-grained cross-modal alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 26307-26316, 2024.

Xiaoling Hu, Fuxin Li, Dimitris Samaras, and Chao Chen. Topology-preserving deep image segmentation. Advances in neural information processing systems , 32, 2019.

James R Clough, Nicholas Byrne, Ilkay Oksuz, Veronika A Zimmer, Julia A Schnabel, and Andrew P King. A topological loss function for deep-learning based image segmentation using persistent homology. IEEE transactions on pattern analysis and machine intelligence , 44(12):8766-8778, 2020.

Xiaoling Hu, Dimitris Samaras, and Chao Chen. Learning probabilistic topological representations using discrete morse theory. In The Eleventh International Conference on Learning Representations , 2022.

Nico Stucki, Johannes C Paetzold, Suprosanna Shit, Bjoern Menze, and Ulrich Bauer. Topologically faithful image segmentation via induced matching of persistence barcodes. In International Conference on Machine Learning , pages 32698-32727. PMLR, 2023.

Zuoyu Yan, Tengfei Ma, Liangcai Gao, Zhi Tang, and Chao Chen. Link prediction with persistent homology: An interactive view. In International conference on machine learning , pages 11659-11669. PMLR, 2021.

Johanna Immonen, Amauri Souza, and Vikas Garg. Going beyond persistent homology using persistent homology. Advances in Neural Information Processing Systems , 36, 2023.

Joshua Southern, Jeremy Wayland, Michael Bronstein, and Bastian Rieck. Curvature filtrations for graph generative model evaluation. Advances in Neural Information Processing Systems , 36, 2023.

Jacob Townsend, Cassie Putman Micucci, John H Hymel, Vasileios Maroulas, and Konstantinos D Vogiatzis. Representation of molecular structures with persistent homology for machine learning applications in chemistry. Nature communications , 11(1):3230, 2020.

Yuankai Luo, Lei Shi, and Veronika Thost. Improving self-supervised molecular representation learning using persistent homology. Advances in Neural Information Processing Systems , 36, 2023.

Vincent Peter Grande and Michael T Schaub. Topological point cloud clustering. In International Conference on Machine Learning , pages 11683-11697. PMLR, 2023.

Naoki Nishikawa, Yuichi Ike, and Kenji Yamanishi. Adaptive topological feature via persistent homology: Filtration learning for point clouds. Advances in Neural Information Processing Systems , 36, 2023.

Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In International conference on machine learning , pages 7045-7054. PMLR, 2020.

Ilya Trofimov, Daniil Cherniavskii, Eduard Tulchinskii, Nikita Balabin, Evgeny Burnaev, and Serguei Barannikov. Learning topology-preserving data representations. In The Eleventh International Conference on Learning Representations , 2022.

Serguei Barannikov, Ilya Trofimov, Nikita Balabin, and Evgeny Burnaev. Representation topology divergence: A method for comparing neural network representations. In International Conference on Machine Learning , pages 1607-1626. PMLR, 2022.

Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop , pages 178-178. IEEE, 2004.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3606-3613, 2014.

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 12(7):2217-2226, 2019.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Finegrained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing , pages 722-729. IEEE, 2008.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13 , pages 446-461. Springer, 2014.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition , pages 3498-3505. IEEE, 2012.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for finegrained categorization. In Proceedings of the IEEE international conference on computer vision workshops , pages 554-561, 2013.

Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition , pages 3485-3492. IEEE, 2010.

Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 , 2012.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning , pages 5389-5400. PMLR, 2019.

Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems , 32, 2019.

Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15262-15271, 2021.

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision , pages 8340-8349, 2021.

A Simplicial Complex, Homology Class, Persistent Homology and Homotopy Equivalent Assumption

A.1 Simplicial Complex

A simplicial complex consists of a set of simplices, such as points, line segments, triangles, and p -dimensional analogues. A simplex σ with dimension p is the convex hull of a set of p +1 affinely independent points ( x 0 , . . . , x p ) . For 0 ≤ p ′ ≤ p , a p ′ -face of σ is a p ′ -simplex that is the convex hull of a non-empty subset. A simplicial complex K consists of a set of finite simplices that satisfy: K contains every face of each simplex in K and for any two simplices σ, τ ∈ K , their intersection σ ∩ τ is either empty or a face of both σ and τ . The dimension of K is the maximum dimension of any simplex in K . Given a finite data point set X in metric space ( M,d ) and a real a > 0 , the Vietoris-Rips complex K Rips a ( X ) is the set of simplices σ such that d ( x, x ′ ) ≤ a for every pair of vertices of σ , and the ˇ Cech complex K Cech a ( X ) is the set of simplices σ such that the closed balls centering its vertices have a non-empty interaction.

A.2 Homology Class

In complex K , let the number of p -simplices be m p , a p -chain c is a formal sum of p -simplices added with some coefficients, i.e. , c = ∑ m p i =1 α i σ i . In particular, the p -chains with Z 2 -additions form a chain group C p ( K ) where the identity is the chain 0 = ∑ m p i =1 0 σ i , and since c + c = 0 , the inverse of a chain c is itself. When K is clear from the context, C p ( K ) can also be notated as C p . Define a boundary operator ∂ p on p -simplex σ as a map that sends σ to the ( p -1) -chain consisting of σ 's ( p -1) -faces referred as σ 's boundary, i.e. , ∂ p σ = ∑ p i =0 ( x 0 , . . . , ˆ x i , . . . , x p ) where ˆ x i indicates that x i is omitted. The boundary of a vertex is empty. The ∂ p induces a homomorphism ∂ p : C p → C p -1 that produces a ( p -1) -chain when extended to a p -chain c through ∂ p c = ∑ m p i =1 α i ( ∂ p σ i ) . Applying ∂ p to the chain groups, a sequence of homomorphism, 0 = C k +1 ∂ k +1 - - - → C k ∂ k - → C k -1 · · · C 1 ∂ 1 - → C 0 ∂ 0 - → C -1 = 0 , where 0 denotes a trivial group, is obtained. A p -chain c is a p -cycle if c has empty boundary. All p -cycles together form a cycle group Z p under the Z 2 -addition operation inherited from C p . Z p is the subgroup of C p that is mapped to the 0 of C p -1 by ∂ p , i.e. , Z p is the kernel of ∂ p . The image of the boundary operator ∂ p is a subgroup of ( p -1) -chains, called boundary group B p -1 = ∂ p (C p ) . For any p -simplex σ , every ( p -2) -faces is involved in exactly two ( p -1) -faces in ∂ p σ , thus ∂ p -1 B p -1 = 0 for p > 0 and B p -1 ⊆ Z p -1 . The homology group H p is defined to classify the cycles in Z p by collecting those cycles that differ by a boundary into the same class. This is achieved by H p = Z p / B p , taking the quotient of the Z p with boundary group B p . By definition, the elements of H p are cosets of B p in Z p , i.e. , { c +B p | c ∈ Z p } . For a cycle c , c +B p in H p forms its homology class [ c ] . Two cycles c and c ′ in the same class [ c ] = [ c ′ ] are homologous. In fact, [ c ] = [ c ′ ] holds if and only if c ∈ c ′ +B p .

A.3 Persistent Homology

For data point set X in metric space ( M,d ) , define a function f : M → R , f (( x 0 , . . . , x p )) = max i,j ∈{ 0 ,...,p } f (( x i , x j )) on simplices. Then, given a sequence of thresholds a 1 ≤ a 2 ≤ . . . , ≤ a n , the growing sublevel sets f -1 ( -∞ , a ] at these values give rise to a nested sequence of subcomplexes, K a 1 ⊆ K a 2 ⊆ · · · ⊆ K a n , called a filtration F . The inclusions in filtration F induce a sequence i,j

of homomorphisms h i,j p , H p F : 0 = H p ( K 0 ) → H p ( K 1 ) →··· → H p ( K i ) → h p · · · → H p ( K j ) → · · · → H p ( K n ) = H p ( K ) , called a homology module. The images of the homomorphisms h i,j p are persistent homology groups H i,j p for the module, H i,j p = im h i,j p , for 0 ≤ i ≤ j ≤ n . The non trivial elements of persistent homology groups H i,j p consist of homology classes that survive from K i to K j , i.e. , the homology classes that do not quotient out by boundaries, which implies that H i,j p = Z p ( K i ) / (B p ( K j ) ∩ Z p ( K i )) . A non-trivial homology class ϵ ∈ H p ( K a ) is born at K i , if ϵ ∈ H i,a p but ϵ / ∈ H i -1 ,a p . Likewise, the ϵ dies entering K j , if ϵ ∈ H a,j -1 p but ϵ / ∈ H a,j p . When a class dies, it may be merged with several classes and raises a new birth. The persistence of homology class that is created at K i and destoryed at K j is defined as a i -a j . The homology classes that never die are set to remain alive till a n +1 = a ∞ = ∞ . As a visual representation, the persistence diagram

Dgm p ( F ) of filtration F draws the paired birth at a i and death at a j that bound the survival interval of one or more homology classes as a point ( a i , a j ) on the extended plane R 2 := ( R ∪ {±∞} ) 2 .

A.4 Assumption on Homotopy Equivalent of Image and Text Latent Manifolds

Topology specifies how points are connected and is a key geometric signature of metric spaces. Topological equivalence can be formalized by continuous functions that map points from one space to the other while preserving the connectivity. A straightforward equivalence is that the points as well as their neighborhoods in two spaces are in one-to-one correspondence, called being homeomorphic [24, 26], which requires a bijecitve map with continuous inverse. Two homeomorphic spaces share the exact same topological properties, i.e. , from the topological point of view, the two are completely consistent. Another less rigid relation is homotopy equivalent [24, 26]. Formally, given spaces X and Y , two maps f 0 , f 1 : X → Y are homotopic if they can be joined by a continuous function F : X × [0 , 1] → Y such that F ( x, 0) = f 0 ( x ) and F ( x, 1) = f 1 ( x ) for every x ∈ X , denoted as f 0 ≃ f 1 . The spaces X and Y are homotopy equivalent if there exist two maps f : X → Y and g : Y → X such that f ◦ g ≃ id Y and g ◦ f ≃ id X , which suggests an intuitive fact that space X is homotopy equivalent to Y if and only if both X and Y are deformation retracts of a common space Z .

Vision and language are key information modalities for human cognition and have intrinsic correspondence on semantics. Observe that similar image scenes tend to be described by synonymic texts, that is, the connectivity of image data samples is preserved after mapping them to the text latent manifold, and vice versa. This preservation of connectivity suggests that the two manifolds have similar topological structures. Besides, image and text data are both relevant and complementary concrete expressions of the real world. The latent manifolds in which they reside can be seen as deformation retracts of the general world knowledge space. Thus, it is rational to consider that the image latent manifold and text latent manifold are homotogy equivalent. As a topological invariant of homotogy equivalent spaces, the homology is a viable tool for quantifying and aligning the structure of image and text latent manifolds.

B Full numerical results on few-shot learning

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly express our motivation and insight in the Abstract and Introduction. The contributions are summarized in the Introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We clearly express our limitations in the Sec. 5, Conclusions, Limitations and Future Work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We discuss the homotopy equivalent assumption underpinning our method in Appendix A.4 and explain the rationale for such an assumption.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our paper provides a detailed account of implementation details and hyperparameter settings of our method for reproducing.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will make all code avaliable when paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide implementation details in Sec. 4.1, Experimental Settings.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We fix the random seeds in experiments for fair comparison.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We specify the type of compute workers in implementation details in Sec. 4.1. Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We believe that our study will not pose any negative societal due to its theoretical nature.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper does not contain data or models that are at high risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the datasets used in our paper and give a brief introduction.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper has nothing to do with crowdsourcing and human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
A Homology Consistency (HC) constraint for efficient transfer on VLMs is proposed in this paper, which explicitly constrains the correspondence of image and text latent manifolds through structural equivalence based on persistent homology in downstream tuning.
The proposed method tracks the persistence of the homology classes of topological features across multiple scales and guide the directions of persistence tracks in image and text manifolds to coincide each other. Additionally, a deviating perturbation is applied to generalize the persistence coincidence to unseen data. Experiments on recognition and generalization tasks show the superior performance.

STRENGTHS:
1. The paper is well-written with a straightforward motivation.
2. A theoretically well-founded homology consistency (HC) constraint based on persistent homology is proposed for efficient transfer on VLMs.
3. Experiments on recognition tasks show the superior performance.

WEAKNESSES:
The hyper-parameters η, λ, ω should be determined at 16 shots and then migrated to other few-shot settings. If the number of samples is less than 16, how should the aforementioned hyper-parameters be set, and will there be a significant difference in performance?

QUESTIONS:
The hyper-parameters η, λ, ω should be determined at 16 shots and then migrated to other few-shot settings. If the number of samples is less than 16, how should the aforementioned hyper-parameters be set, and will there be a significant difference in performance?

LIMITATIONS:
The experiments only conducted on recognition tasks.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper identifies a key issue with existing methods for tuning pre-trained vision-language models to downstream tasks with limited data: they adjust the alignment between image and text based solely on observed samples, which may not generalize well beyond the training data. To address this issue, the paper proposes a novel constraint from the perspective of topological data analysis.\n\nThis constraint employs persistent homology to ensure the structural equivalence of image and text latent manifolds during tuning.\n\nSTRENGTHS:\n1.  The paper offers a new way of looking at model tuning through the lens of topological analysis, with a focus on understanding the structure of data spaces for better semantic alignment in vision-language tasks. I appreciate this perspective on the issue.\n\n2.  The proposed method exhibits a thoughtful theoretical underpinning, using persistent homology to enhance the generalizability of image-text alignment adjusting.&#x20;\n\n3.  The paper is well-written and the reason for leveraging topological data analysis to enhance semantic alignment during the tuning process is reasonable and easy to follow up.\n\nWEAKNESSES:\nThe paper does not adequately discuss how it relates to existing image and text alignment techniques, including those based on distance metrics, mutual information, adversarial training, and attention mechanisms. This lack of comparative analysis creates a gap in fully appreciating the distinctive contributions and potential advantages.\n\nQUESTIONS:\n1.  Could you please provide insights into the fundamental differences and advantages of topological data analysis in your method over other alignment methods?\n\n2.  Where do you foresee potential challenges in extending your proposed method to tasks outside the few-shot learning domain, particularly in scenarios such as zero-shot learning, or applications involving detection and segmentation?\n\n3.  Could you elaborate on how incorporating higher-dimensional homology classes into the tuning process might impact the model's performance and behavior, beyond the computational cost?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
The paper introduces a Homology Consistency (HC) constraint for efficient transfer learning on vision-language models (VLMs), ensuring task-specific image-text alignment while preserving general knowledge by using structural equivalence based on persistent homology. This approach mimics the topology of latent manifolds and tracks the persistence of topological features.

STRENGTHS:
1. This paper is well motivated, and the motivation of using homology consistency is interesting. 
2. This paper has a good theoretic support.

WEAKNESSES:
1. The performance of the proposed method is worse than the baseline method in low-shot (1-shot and 2-shot) tasks. 
2. The improvement in Table 2 is marginal. Is the comparison fair with the same random seed? How many runs did you conduct? Could the authors also report the standard deviation of the score? 
3. Moreover, is 16-shot common in this benchmark? 16 shot seems a lot in few-shot learning. 
4. Can you also elaborate more why with only DP, the performance drops in Table 3? 
5. In addition, could you elaborate more why choosing 0-th homology classes? What are the potential effects of using other homology classes?

QUESTIONS:
1. Can you visualise the topology of the data before and after adaptation with the HC constraint? It will be very interesting to see how actually the HC constraint can preserve the topology of the manifold during transfer learning.

LIMITATIONS:
My concern is the performance improvement is marginal and limited to more shots setting (16-shot).
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14

comments: SUMMARY:
This paper proposes Homology Consistency (HC) constraint for transfer learning on VLMs, and it explicitly constrains the correspondence of image and text latent manifolds by structural equivalence based on persistent homology in downstream tuning.

STRENGTHS:
1. The proposed method is well-founded and clearly explains the proposed homology consistency (HC) constraint.

2. Extensive experiments are performed on 11 benchmark datasets.

WEAKNESSES:
1. The paper lacks discussions on the computational cost of the proposed techniques.

2. The proposed method for constraining the structural equivalence of image and text latent manifolds seems generalizable to other learning tasks for vision-language models. However, the proposed method is only evaluated for few-shot learning of vision language models.

3. Although the model outperforms other methods in most cases, the improvements are relatively marginal.

4. The paper only applies the method to a limited number of adapter models (TaskRes and Tip-Adapter-F).

QUESTIONS:
see weaknesses.

LIMITATIONS:
The limitations are not discussed sufficiently (see weaknesses).
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-17