Title: With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You
Venue: NeurIPS 2025
Accepted: yes
Keywords: multimodal learning

vision language models

representation learning
Abstract: Multimodal models have demonstrated powerful capabilities in complex tasks requiring multimodal alignment, including zero-shot classification and cross-modal retrieval. However, existing models typically rely on millions of paired multimodal samples, which are prohibitively expensive or infeasible to obtain in many domains. In this work, we explore the feasibility of building multimodal models with limited amount of paired data by aligning pretrained unimodal foundation models. We show that high-quality alignment is possible with as few as tens of thousands of paired samples$\unicode{x2013}$less than 1\% of the data typically used in the field. To achieve this, we introduce STRUCTURE, an effective regularization technique that preserves the neighborhood geometry of the latent space of unimodal encoders. Additionally, we show that aligning last layers is often suboptimal and demonstrate the benefits of aligning the layers with the highest representational similarity across modalities. These two components can be readily incorporated into existing alignment methods, yielding substantial gains across 24 zero-shot image classification and retrieval benchmarks, with average relative improvement of 51.6\% in classification and 91.8\% in retrieval tasks. Our results highlight the effectiveness and broad applicability of our framework for limited-sample multimodal learning and offer a promising path forward for resource-constrained domains.

With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You

Abstract

Multimodal models have demonstrated powerful capabilities in complex tasks requiring multimodal alignment, including zero-shot classification and cross-modal retrieval. However, existing models typically rely on millions of paired multimodal samples, which are prohibitively expensive or infeasible to obtain in many domains. In this work, we explore the feasibility of building multimodal models with limited amount of paired data by aligning pretrained unimodal foundation models. We show that high-quality alignment is possible with as few as tens of thousands of paired samples-less than 1% of the data typically used in the field. To achieve this, we introduce STRUCTURE, an effective regularization technique that preserves the neighborhood geometry of the latent space of unimodal encoders. Additionally, we show that aligning last layers is often suboptimal and demonstrate the benefits of aligning the layers with the highest representational similarity across modalities. These two components can be readily incorporated into existing alignment methods, yielding substantial gains across 24 zero-shot image classification and retrieval benchmarks, with average relative improvement of 51 . 6% in classification and 91 . 8% in retrieval tasks. Our results highlight the effectiveness and broad applicability of our framework for limited-sample multimodal learning and offer a promising path forward for resource-constrained domains.

1 Introduction

Unimodal foundation models (FMs) have achieved remarkable progress in recent years, demonstrating high performance on a variety of complex tasks across diverse domains. Large language models (LLMs) [1, 2] have shown unparalleled abilities across a wide range of natural language understanding and generation tasks, reaching human-level performance on many complex reasoning and comprehension tasks. In parallel, vision FMs [3, 4] have delivered similar breakthroughs in visual perception, enabling systems to perform tasks such as object recognition and segmentation with minimal supervision. In scientific domains, models such as AlphaFold [5] and ESM2 [6] have demonstrated the potential of large-scale models to solve highly specialized and complex problems, including accurate protein structure prediction and sequence-based modeling of biomolecular functions.

While these models are highly effective within their respective modalities, many applications involve multimodal data, where it is important to map different modalities into a shared representation space to enable meaningful cross-modal comparison, retrieval, and classification. Pioneering multimodal models like CLIP [7] and CLAP [8] align visual and auditory inputs with language through contrastive learning and achieve strong zero-shot performance across a broad spectrum of multimodal tasks. However, these powerful multimodal models rely on vast amounts of paired training data ( e.g. , CLIP uses 400M pairs), which are often unavailable in many domains like healthcare and biology, where collecting high-quality multimodal data is expensive and labor-intensive.

1

Modality1

Modality 2

Goal:

Alignment

Enc1

f1

Modality 1

Pretrainedspace

Canwe align differentmodalities

usingonlylimitedsamples?

TEXT

Enc2

f2

Shared space

Modality 2

samplesofmodality1

Frozen

Enc: Encoder

correspond

aligned

misaligned

samplesofmodality2

Trainable

f:Alignment function

A promising direction to address this limitation is to leverage powerful representations from pretrained unimodal FMs for multimodal alignment. However, existing alignment methods [9-11] still require large quantities of paired examples, often tens of millions, to learn a shared embedding space effectively. On the other hand, unsupervised techniques [12, 13] do not use labeled data but primarily perform sample-level matching, thus failing to construct a shared embedding space. This opens a fundamental question: Can we align FMs from different modalities into a shared representation space using a limited number of multimodal samples?

In this work, we answer this question affirmatively by introducing a modular alignment strategy that requires as few as tens of thousands of paired examples, less than 1% of the data used by existing modalities alignment methods [9-11]. Specifically, we introduce STRUCTURE, a novel regularization technique designed to preserve the multiscale neighborhood structure of each unimodal latent space during alignment, ensuring that the relationships between samples captured by the unimodal encoders are retained. In addition, based on the observation that higher similarity correlates with better alignment performance, we propose to align those layers with the highest representational similarity. Incorporating both components into existing alignment methods leads to consistent relative performance gains across 22 zero-shot classification and two retrieval benchmarks, averaging 51 . 6% in classification and 91 . 8% in retrieval, demonstrating superior performance in low-data settings. Finally, we show that incorporating just a few labeled examples per class from the target domain into the training set can effectively bridge the performance gap to large-scale multimodal models trained on hundreds of millions of multimodal data, highlighting the importance of domain coverage over sheer data volume in multimodal alignment.

2 Related work

Multimodal foundation models. Recent advances in multimodal FMs have produced unified architectures capable of understanding and generating content across vision and language. Early models such as CLIP [7] and ALIGN [14] use contrastive learning to align image and text embeddings, achieving strong zero-shot performance in classification and retrieval. In contrast, models like BLIP [15] and GIT [16] adopt generative pretraining to support tasks such as image captioning and visual question answering. More recent efforts, including FLAVA [17], PaLI [18], Kosmos-1 [19], and Gemini [20], aim for broad task coverage through unified multimodal training, reflecting a shift toward scalable, general-purpose multimodal systems. Despite their impressive capabilities, existing multimodal models rely heavily on massive paired datasets, often hundreds of millions of samples. In contrast, in this work we explore how to align powerful unimodal models that are readily available in many domains using only limited multimodal supervision, reducing reliance on large-scale paired data and enabling multimodal learning in low-data regimes.

Parameter-frozen modality alignment. Parameter-frozen modality alignment is an efficient strategy for building multimodal systems by aligning pretrained unimodal encoders without updating their internal weights. Unsupervised alignment methods seek to align representations without relying on paired data. Techniques such as Centered Kernel Alignment [12] focus on maximizing representational similarity between modalities, leveraging shared structural patterns. However, a key limitation of these methods [13, 12, 21] is that they primarily perform sample-level matching, thus failing to construct a shared embedding space. Furthermore, they are not able to utilize paired data even when available, missing out on rich alignment signals that could significantly enhance performance. In contrast, supervised alignment methods assume access to abundant paired data, such as millions of paired samples, and train lightweight alignment modules ( e.g. , linear projections or MLPs) while keeping the encoders frozen. These approaches [9, 10] have achieved strong performance in tasks like zero-shot classification and retrieval. However, these methods are limited by their dependence on large-scale multimodal datasets, requiring tens of millions of paired samples to effectively train the models. In contrast, our work explores parameter-frozen alignment in a low-data regime, with tens of thousands of paired samples, which is 0 . 02% of the paired samples used by previous works. To address the challenges of low-data regimes, we propose an effective strategy that can be easily integrated into existing methods [9, 10] to make them more effective under low-data conditions.

Other works have also focused on data-efficient alignment. FuseMix [22] shares our motivation of using limited paired data but focuses on improving training efficiency via latent space augmentations. Our geometric regularization and layer selection are complementary to their approach, and as shown in our experiments, combining them yields further gains. Learning-free methods like ASIF [23] also leverage neighborhood structure, but for direct matching without training an alignment function.

Platonic representation hypothesis. The Platonic representation hypothesis [24] suggests that models from different modalities can converge to similar internal representations. This insight motivates the possibility of aligning separate representation spaces. While earlier multimodal FMs like CLIP [7] depend on end-to-end training with vast amounts of paired data, the Platonic view offers a more efficient alternative: aligning independently trained unimodal models within a shared embedding space. This approach opens the door to scalable and flexible multimodal learning without the need for joint training, and it has recently spurred interest in supervised and unsupervised alignment techniques [9, 12]. Building on this direction, which is related to the broader field of geometric priors in representation alignment [25-28], our work explores its potential in the challenging low-data setting.

3 Problem setup

We consider the task of aligning representations from independently pretrained encoders across different modalities. As illustrated in Figure 1, we keep the encoders frozen and learn lightweight alignment functions that map from each modality's latent space into a shared space where semantically related samples are close. We next formalize this setup.

Modality alignment with pretrained representations. Let X 1 ⊆ R d 1 and X 2 ⊆ R d 2 be the latent spaces of two pretrained unimodal encoders, which correspond to outputs from either the last or intermediate layers of the encoders, and d 1 and d 2 be their respective dimensions, which do not need to be equal. Given a set of N paired multimodal samples D = { ( x i 1 , x i 2 ) } N i =1 , where x i 1 ∈ X 1 and x i 2 ∈ X 2 , our goal is to learn two alignment functions f 1 : R d 1 → A and f 2 : R d 2 → A , which map the modality-specific spaces X 1 and X 2 into a shared embedding space A ⊆ R k of dimension k . Alignment in the shared space is achieved when the paired samples ( x i 1 , x i 2 ) are closer to each other than to any non-paired sample. Specifically,

̸

where sim ( · ) denotes a similarity function, such as cosine similarity.

Different from previous works, we focus on this alignment problem under the challenging condition of limited paired data . Specifically, we focus on the scenario when N is relatively small ( i.e. , tens of thousands of samples) compared to tens of millions of paired samples considered in previous works [9, 10].

Weunify different alignment methods under a joint framework that consists of three main components: (i) modality-specific latent spaces X 1 and X 2 , (ii) alignment functions f 1 and f 2 , and (iii) the objective

function L A , which guides the construction of the shared space A . Table 1 summarizes how existing methods instantiate each of these components. In our work, we propose a general framework that can work with any alignment function and any objective function by regularizing the objective using the STRUCTURE regularization R S , and using as modality-specific latent layers those layers that have highest representational similarity. We introduce both components in the following section.

4 Multimodal alignment with limited data

To align pretrained unimodal encoders using a limited number of paired multimodal samples, we propose two key components: (i) STRUCTURE, a regularization that preserves the intrinsic geometry of each modality's latent space, and (ii) a strategy for selecting modality-specific latent space, those layer pairs that have highest representational similarity. Both components can be seamlessly integrated into existing alignment methods [9, 10, 29] (see Table 1).

Preserving neighborhood via STRUCTURE regularization. With a limited number of paired samples, it is crucial to preserve the latent structure of pretrained unimodal encoders, trained on millions or even billions of examples, as they encode meaningful relationships between samples. To achieve this, we introduce STRUCTURE, a regularization term that preserves the neighborhood relationships of the pretrained space of unimodal encoders within the aligned space.

Given a modality-specific space X ⊆ R d and its corresponding shared space A ⊆ R k , where X can be either X 1 or X 2 , the regularization term aims to ensure the hierarchical ( i.e. , multi-scale) consistency between the relationships expressed by X and A . Specifically, each sample x i ∈ X and a i ∈ A ( i.e. , a i = f ( x i ) ) is first ℓ 2 -normalised, ̂ x i = x i / ∥ x i ∥ 2 , ̂ a i = a i / ∥ a i ∥ 2 , and then centered to remove any global translation bias:

We denote the normalized and centered matrices as ˜ X = [ ˜ x 1 , ˜ x 2 , · · · , ˜ x N ] ∈ R N × d and ˜ A = [ ˜ a 1 , ˜ a 2 , · · · , ˜ a N ] ∈ R N × k .

The (scaled) similarity matrices are computed with temperature τ > 0 as:

To interpret the similarities as probability distributions, we apply a softmax function row-wise. Specifically, we define P X = softmax( S X ) , P A = softmax( S A ) as the corresponding matrices with the similarity distributions of the respective spaces.

To capture relationships reachable by exactly l hops on the similarity graph, we define for each hierarchical level l = 1 , . . . , L , where L ∈ N is the total number of levels 2 :

Intuitively, P X can be seen as the transition matrix of a random walk on the data manifold. The l -hop matrices thus capture multi-step relationships and progressively more global structure.

̸

The key idea of our regularization is to enforce consistency between the structural relationships, i.e. , the relative positions and neighborhood structure in the embedding space, captured by X and A . Thus, we employ Jensen-Shannon divergence because of its symmetric nature to measure the divergence between similarity distributions. Specifically, at each level l , we define the level-specific divergence as:

where

and D KL is the Kullback-Leibler divergence. In practice, a small constant ε is added inside the logarithm for numerical stability, i.e. , P ( l ) X + ε and P ( l ) A + ε .

The final formulation of the STRUCTURE regularizer is a weighted average of the divergences across levels, where lower levels are weighted more heavily to counteract the more concentrated distributions of the higher levels:

Wedenote R ( L ) S as the regularization that operates on L levels, and set it to 1 if not otherwise specified. Together with any objective function L A used for representation alignment (e.g., L C in work [9, 10]), the combined loss is defined as

where λ is the regularization weight.

Each JS ( · ) is non-negative and vanishes iff its inputs coincide, with equality exactly when P X = P A for all l , resulting from perfect multi-scale alignment. Moreover, by virtue R S is invariant to global scaling, translations, and orthonormal rotations, depending solely on the intrinsic, hierarchical relational structure of the embeddings. Since R S operates in sample space, and thus is influenced by the size of N , we investigate the generalization gap between the empirical and expected values of the regularization.

Lemma 1 (Generalization bound) . The generalization gap between the empirical and the expected values of R S is bounded by

where ˆ R N is the empirical STRUCTURE regularizer and R ⋆ is its expectation over the data distribution. This shows that the empirical regularizer faithfully approximates its population expectation as the number of samples increases.

Proof. Given in Appendix C.

Similarity-based layer selection. In parameter-frozen alignment, the quality of alignment is closely linked to the representational similarity between the unimodal representation spaces X 1 and X 2 . Given two unimodal FMs, these spaces typically correspond to different layers of the models. Therefore, selecting the appropriate layers for alignment is critical. To identify the layers with the greatest potential for effective alignment, a metric is required to quantify the similarity between these representation spaces. Examples include Centered Kernel Alignment (CKA) [30], which measures the overall correspondence of pairwise relationships by comparing centered Gram matrices, unbiased CKA [31], which further removes the systematic overestimation inherent in standard CKA statistics when sample sizes are small, and mutual k -nearest-neighbor (kNN) [24], which identifies the k closest samples in each modality's feature space and records the fraction of neighbors they share. Prior work has, however, solely relied on aligning models at their last layers [9], ignoring layer-based similarity. We instead challenge this approach and show in Figure 2 a strong correlation between the representational

0.2

0.4

Similarity

0

20

40

60

80

100

Performance

= 0.94

RoBERTa + ViT-L

0.2

0.4

Similarity

0

20

40

60

80

100

= 0.97

RoBERTa + ViT-G

0.05

0.10

0.15

Similarity

0

20

40

60

80

= 0.95

Llama3-8B + ViT-G

0.05 0.10 0.15

Similarity

0

20

40

60

80

100

= 0.95

Llama-13B + ViT-G

STL10

CIFAR10

Caltech101

CIFAR100

ImageNet

Pets

similarity measured in terms of mutual kNN of the layers of pretrained unimodal encoders and downstream zero-shot performance once aligned using a single linear layer with our regularization.

We thus propose the following procedure for layer selection: (i) compute the representational similarity between all pairs of layers in terms of mutual kNN on a small set of paired samples (in the order of 5 , 000 pairs), typically randomly selected from the training set, and (ii) choose the layers with the highest similarity for alignment. Throughout our work, we compute representational similarity according to mutual kNN with k chosen according to Rice's criterion 3 , similar to what has been used in prior work [24] to measure similarities of different latent spaces. We show that this selection procedure leads to consistent results with different subset sizes and repeats in Appendix H.2.

5 Experiments

Experimental setup. In our experiments, we align frozen pretrained unimodal encoders using a limited number of image-text pairs. To align models, we use the MS COCO train split consisting of 80 , 000 paired samples [33]. To demonstrate versatility of the STRUCTURE regularization, we apply it to different alignment strategies shown in Table 1, including linear mapping (Linear) [10], non-linear mapping (MLP) [9], and CSA [29]. In addition, to evaluate the benefits of selecting layers based on the similarity of representations rather than the last layer, we trained the models on either the final layers (' Last ') or the similarity-based selected layers (' Similar '). Experiments are performed with different combinations of models. By default, we consider RoBERTa [34] and DINOv2 ViT Giant [3] as our unimodal encoders.

We evaluate model performance on zero-shot classification and cross-modal retrieval tasks. For zero-shot classification, we consider 22 datasets from the CLIP benchmark [7]. For cross-modal retrieval, we consider the Flickr30 and MS COCO test splits to evaluate both text-to-image and image-to-text performance. Detailed dataset descriptions and training configurations are described in Appendices D and E, respectively.

Performance comparison. We conduct a comprehensive evaluation of zero-shot classification and cross-modal retrieval performance across diverse datasets and alignment techniques. Table 2 shows that adding STRUCTURE regularization helps improve generalization performance across all three alignment techniques. On average, STRUCTURE yields substantial relative gains in both zero-shot classification and cross-modal retrieval. Specifically, for zero-shot classification, the average relative improvement from using STRUCTURE (compared to the baseline using the most similar layers without regularization) is 74 . 0% for MLP, 68 . 4% for Linear, and 26 . 8% for CSA. In retrieval tasks, the corresponding relative improvements are 137 . 0% , 122 . 8% , and 15 . 9% , respectively. These results highlight that MLP and Linear alignment strategies benefit the most from STRUCTURE, while CSA shows more moderate but still substantial gains, indicating the broad applicability of our STRUCTURE across different alignment methods.

In addition to regularization, selecting layers based on representational similarity instead of defaulting to the final layer also leads to consistent improvements across all methods. On average, this strategy yields relative gains in classification accuracy of 2 . 5% for Linear, 4 . 8% for MLP, and 2 . 0% for CSA. For retrieval tasks, the relative improvements are even more pronounced, 8 . 6% for Linear, 18 . 3% for MLP, and 2 . 7% for CSA. This strategy alone provides a significant boost in performance and, when combined with STRUCTURE, results in the best overall generalization. The complete results across all 24 datasets are shown in Appendix H.8, showing consistent trends on the extended benchmark: average improvements from baseline to our approach are 65 . 0% for Linear in classification and 122 . 7% in retrieval, 61 . 5% and 136 . 8% for MLP, and 28 . 3% and 15 . 9% for CSA, respectively. Interestingly, on the CIFAR10 dataset, our alignment strategy even outperforms CLIP [7] by around 2% while using only 0 . 02% of the data used to train CLIP.

1k

5k

10k

50k

80k

50

75

U

= 10.9

STL10

1k

5k

10k

50k

80k

50

75

U

= 17.3

CIFAR10

1k

5k

10k

50k

80k

0

10

20

U

= 35.8

ImageNet

1k

5k

10k

50k

80k

0

20

40

U

= 22.4

Flickr30 T2I

1k

5k

10k

50k

80k

Sample Size

10

20

30

U

= 9.6

Food101

1k

5k

10k

50k

80k

Sample Size

20

40

U

= 23.1

CIFAR100

1k

5k

10k

50k

80k

Sample Size

5

10

15

U

= 32.5

Pets

1k

5k

10k

50k

80k

Sample Size

0

25

50

U

= 5.7

Flickr30 I2T

Zero

-

Shot

Classification

Retrieval

(R@1)

Last

Similar +

S

Scaling down the training data. To measure the effect of the size of the training dataset on the performance, we subsampled the COCO training set and evaluated the effectiveness of the proposed method across varying dataset sizes for both zero-shot classification and retrieval tasks. We observe that across different tasks and sample sizes, STRUCTURE regularization combined with layer selection significantly improves performance, even in extreme resource-constrained settings of only 1 , 000 samples (see Figure 3). Similarly, we find that the layer selection strategy continues to provide improvements even with less data (Appendix H.3). These results demonstrate that combining

latent-geometry preservation with strategic layer selection is highly performant under severe data scarcity, making multimodal alignment practical even in the most constrained low-data regimes.

To quantify label efficiency, we measure the utility [35] U ( N ) = ( ˆ N/N ) -1 for different-sized subsets and report its average value. Here, N is the number of labeled examples used by the regularized model, and ˆ N is the minimal number of labeled examples required when naively relying on the last layer for alignment without any regularization to match that same accuracy. This metric tracks the reduction in samples needed to achieve equivalent performance. For example, for zero-shot classification on the CIFAR100 dataset and text-to-image retrieval performance on the Flickr30 dataset, the outlined method yields an average utility of 23 . 1 × and 22 . 4 × , respectively, indicating that roughly 23 × fewer samples are needed to achieve the same performance as the baseline. These substantial label savings demonstrate that our approach significantly reduces annotation requirements while preserving high performance, thereby enhancing practicality for real-world multimodal applications.

Scaling up the training data. We next explored the effect of increasing the size of the multimodal dataset used for alignment. We gradually incorporate increasingly larger subsets of the LAION15 Mdataset [36] into the COCO dataset of 80 K samples. Following the approach of Gadre et al. [37], we filter out low-quality LAION samples by computing CLIP scores [38] for each image-text pair and discarding those with a score below 0 . 15 . From the remaining 847 K samples, we randomly draw subsets and incrementally add them to the 80 K COCO training set in steps of 80 K. Figure 4 presents the results of incrementally

COCO

80k

160k

240k

320k

400k

480k

560k

Added LAION Samples

40

50

60

70

Zero-Shot

CIFAR100

COCO

80k

160k

240k

320k

400k

480k

560k

Added LAION Samples

10

20

30

40

ImageNet

Similar

Similar +

S

adding different numbers of paired samples for CIFAR100 and ImageNet zero-shot classification, and we compare performance with and without applying STRUCTURE regularization on layers selected based on representational similarity. The steepest performance gain occurs with the initial addition of 80K samples, while further increases yield smaller but steady improvements. Notably, the impact of STRUCTURE regularization diminishes as more data becomes available, highlighting its particular effectiveness in low-data regimes, where preserving pretrained structure is most beneficial.

Train-test data distribution shift. Despite the advantages of our alignment approach in low-data regimes, performance remains low on certain datasets. For example, accuracy on the Pets dataset reaches only around 15% . We attribute this to distribution shifts between the COCO-based training set and the target evaluation domains, including differences in image content, label granularity, and overall dataset coverage and scale. To understand the impact of this shift, we augmented the training set ( 80 K COCO pairs) with a small number of k in-domain examples per class drawn from different image classification benchmarks. These k samples per class are added directly to the 80,000-sample COCO training set (e.g., 80 , 000 + 10 × k total samples for CIFAR10). We find that mixing a small number of samples from the target domain into the training set can effectively close distribution gaps and achieve strong performance. Figure 5 plots zero-shot accuracy as a function of number of samples per class k across five datasets, including Flowers, Food101, CIFAR100, Pets, and ImageNet. Remarkably, we can achieve or even outperform CLIP trained on hundreds of millions of multimodal samples by adding a few in-distribution labeled samples. Specifically, with just three samples per class, accu-

0

5

10

15

20

Samples per Class

0

20

40

60

80

100

In-Domain

CIFAR100

Pets

Flowers

Food101

ImageNet

Last

Similar

Similar +

S

RoBERTa

Llama3-8B

Llama-13B

75.6

79.7

92.6

68.9

85.3

92.8

70.8

79.2

93.8

STL10

Last

Similar

Similar +

S

85.5

89.0

96.3

74.3

80.8

92.5

69.5

69.9

87.2

CIFAR10

Last

Similar

Similar +

S

14.8

14.6

30.6

22.7

26.5

41.7

20.6

18.1

37.1

FOOD101

Last

Similar

Similar +

S

34.0

33.1

51.3

27.0

29.4

47.8

26.4

23.1

48.2

CIFAR100

0

20

40

60

80

100

racy on the Flowers dataset rises from around 24% to over 95% , outperforming CLIP's 93% accuracy. Similarly, on the CIFAR100 dataset, using only four samples per class yields a 3% improvement over CLIP. On the ImageNet dataset, our alignment strategy matches CLIP's performance with only 17 samples per class. On both Pets and Flowers datasets, we achieve comparable results to CLIP using approximately 20 samples per class. These results suggest that in a limited data regime, labeling just a few in-domain samples can yield comparable or even superior performance to large models trained on millions of paired samples.

Different pairs of models. We explore the performance across different language models, each combined with a DINOv2 ViT-G encoder. Figure 7 shows that our regularization term and layer selection strategy consistently improve results across all combinations of models. On average, our method yields improvements of 15 . 2% for RoBERTa, 20 . 5% for Llama3-8B, and 19 . 8% for Llama13B. Notably, the RoBERTa-based combination achieves the highest overall performance, supporting previous findings [9] that RoBERTa aligns particularly well with vision FMs.

Neighborhood preservation. To empirically verify that the STRUCTURE regularization preserves pretrained neighborhood, we monitored Trustworthiness 100 and Continuity 100 [39] on a fixed random subset of 5 , 000 embeddings from the training and the held-out validation set at the end of each epoch with and without the STRUCTURE regularization. Trustworthiness measures the fraction of k -nearest neighbors in the aligned space that were already neighbors in the original pretrained embedding, while Continuity measures the fraction of original neighbors that remain in the aligned space. As shown in Figure 6, over 2 , 000 epochs, both the training Trustworthiness and Continuity steadily decline under the standard alignment objective, while the gap between training and validation values steadily widens. In contrast, with STRUCTURE regularization, Trustworthiness and Continuity curves on training and validation remain between 0 . 99

0

1000

2000

Epoch

0.93

0.95

0.98

1.00

Trustworthiness

0

1000

2000

Epoch

0.96

0.98

1.00

Continuity

Similar (Train)

Similar (Val)

Similar +

S (Train)

Similar +

S (Val)

and 1 . 00 (gap < 0 . 002 ) with no drift. This demonstrates that our regularizer enforces consistent, geometry-respecting alignments throughout optimization, avoiding the over-warping seen in unregularized alignment.

Further experiments. Additional results are provided in the Appendix H, including experiments with text-audio alignment and application to the biological domain (see H.4), more ablations such as the robustness to the parameter λ , number of regularization levels L , layer selection metric,

normalization schemes, and distance functions (see H.6), as well as comparisons to unsupervised alignment methods (see H.7).

6 Conclusion

We present a simple, yet effective, framework for parameter-frozen modality alignment in low-data regimes, leveraging two key components: (i) a novel STRUCTURE regularizer that preserves the multi-scale neighborhood geometry of each modality's pretrained latent space, and (ii) an automatic layer-selection procedure that identifies and aligns the pair of intermediate layers with the highest representation similarity. Both components integrate seamlessly with existing alignment pipelines, be they linear projections, MLPs, or advanced matrix-decomposition methods, offering a plug-and-play solution for any modality alignment method. We evaluate our strategy by incorporating it into three existing modality alignment methods and observe consistent performance improvements across 24 benchmark datasets covering zero-shot classification and retrieval tasks, with average relative improvements of 51 . 6% in classification and 91 . 8% in retrieval tasks. This stems from a faithful preservation of pretrained geometry throughout training, enforced by the proposed regularization. We further show that injecting only a handful of in-domain examples per class achieves comparable performance to multimodal models trained on hundreds of millions of samples.

Limitations

While our method performs competitively in the low-data regime, there remains a performance gap on more challenging tasks compared to models like CLIP that are trained on hundreds of millions of paired samples. That said, we show that this gap can be significantly reduced by incorporating only a few in-domain samples in Figure 5, which highlights the potential of few-shot adaptation. Additionally, this prompted us to consider which data is most effective for aligning the spaces of different modalities. A promising future direction is to systematically investigate how different types of data influence cross-modal alignment quality in a low-data regime.

Currently, we have only investigated aligning two modalities. However, our framework can be straightforwardly extended to three or more modalities. Since the STRUCTURE regularization is computed independently for each modality to preserve its own pretrained structure, this extension only requires adding an additional regularization term to the loss function for each new modality. For instance, aligning three modalities ( X 1 , X 2 , X 3 ) would involve the following objective:

Since aligning more than two modalities is still a very active research topic, we leave these investigations to future efforts.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research , 2023.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment Anything. In International Conference on Computer Vision , 2023.

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with AlphaFold. Nature , 2021.

Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , 2021.

Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. CLAP: Learning Audio Concepts from Natural Language Supervision. In International Conference on Acoustics, Speech and Signal Processing , 2023.

Mayug Maniparambil, Raiymbek Akshulakov, Yasser Abdelaziz Dahou Djilali, Sanath Narayan, Ankit Singh, and Noel E O'Connor. Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment. Conference on Computer Vision and Pattern Recognition , 2024.

Zaid Khan and Yun Fu. Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learning. In International Conference on Learning Representations , 2023.

Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly Mapping from Image to Text Space. In The Eleventh International Conference on Learning Representations , 2023.

Mayug Maniparambil, Raiymbek Akshulakov, Yasser Abdelaziz Dahou Djilali, Mohamed El Amine Seddik, Sanath Narayan, Karttikeya Mangalam, and Noel E O'Connor. Do Vision and Language Encoders Represent the World Similarly? In Conference on Computer Vision and Pattern Recognition , 2024.

Laure Ciernik, Lorenz Linhardt, Marco Morik, Jonas Dippel, Simon Kornblith, and Lukas Muttenthaler. Training objective drives the consistency of representational similarity across datasets. CoRR , 2024.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning , 2021.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , 2022.

Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A Generative Image-to-text Transformer for Vision and Language. Transactions on Machine Learning Research , 2022.

Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Conference on Computer Vision and Pattern Recognition , 2022.

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. PaLI: A Jointly-Scaled Multilingual Language-Image Model. In International Conference on Learning Representations , 2022.

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding Multimodal Large Language Models to the World. arXiv e-prints , pages arXiv-2306, 2023.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A Family of Highly Capable Multimodal Models. arXiv e-prints , pages arXiv-2312, 2023.

Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in Neural Information Processing Systems , 2017.

Noël Vouitsis, Zhaoyan Liu, Satya Krishna Gorti, Valentin Villecroze, Jesse C Cresswell, Guangwei Yu, Gabriel Loaiza-Ganem, and Maksims Volkovs. Data-efficient multimodal fusion on a single gpu. In Conference on Computer Vision and Pattern Recognition , 2024.

Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodola, and Francesco Locatello. Asif: Coupled data turns unimodal models to multimodal without training. Advances in Neural Information Processing Systems , 2023.

Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The Platonic Representation Hypothesis. In International Conference on Machine Learning , 2024.

Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. In International Conference on Learning Representations , 2023.

Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and Emanuele Rodolà. Latent space translation via semantic alignment. Advances in Neural Information Processing Systems , 2023.

Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In International Conference on Learning Representations , 2017.

Mikel Artetxe, Gorka Labaka, and Eneko Agirre. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Annual Meeting of the Association for Computational Linguistics , 2018.

Po-han Li, Sandeep P. Chinchali, and Ufuk Topcu. CSA: Data-efficient Mapping of Unimodal Features to Multimodal Features. In International Conference on Learning Representations , 2025.

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning , 2019.

Le Song, Alex Smola, Arthur Gretton, Justin Bedo, and Karsten Borgwardt. Feature selection via dependence maximization. Journal of Machine Learning Research , 2012.

George R Terrell and David W Scott. Oversmoothed nonparametric density estimates. Journal of the American Statistical Association , 1985.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision , 2014.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Alejandro Newell and Jia Deng. How Useful Is Self-Supervised Pretraining for Visual Tasks? In Conference on Computer Vision and Pattern Recognition , June 2020.

Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. Advances in Neural Information Processing Systems , 2022.

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei W Koh, Olga Saukh, Alexander J Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. DataComp: In search of the next generation of multimodal datasets. In Advances in Neural Information Processing Systems , 2023.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. In Empirical Methods in Natural Language Processing , 2021.

Jarkko Venna and Samuel Kaski. Neighborhood preservation in nonlinear projection methods: An experimental study. In International Conference on Artificial Neural Networks , 2001.

Leslie N Smith. Cyclical learning rates for training neural networks. In Winter Conference on Applications of Computer Vision , 2017.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical Report, University of Toronto , 2009.

Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Artificial Intelligence and Statistics , 2011.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A Large-scale Hierarchical Image Database. In Computer Vision and Pattern Recognition , 2009.

Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In Conference on Computer Vision and Pattern Recognition Workshop , 2004.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In European Conference on Computer Vision , 2014.

Maria-Elena Nilsback and Andrew Zisserman. Automated Flower Classification over a Large Number of Classes. In Sixth Indian Conference on Computer Vision, Graphics & Image Processing , 2008.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D Object Representations for Fine-grained Categorization. In International Conference on Computer Vision Workshops , 2013.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained Visual Classification of Aircraft. arXiv preprint arXiv:1306.5151 , 2013.

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and Dogs. In Computer Vision and Pattern Recognition , 2012.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based Learning Applied to Document Recognition. In Proceedings of the IEEE , 1998.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing Textures in the Wild. In Computer Vision and Pattern Recognition , 2014.

Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun Database: Exploring a Large Collection of Scene Categories. In International Journal of Computer Vision , 2016.

Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. Challenges in Representation Learning: A Report on Three Machine Learning Contests. In Neural Networks , 2015.

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification. In IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2019.

Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote Sensing Image Scene Classification: Benchmark and State of the Art. In Proceedings of the IEEE , 2017.

Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. Man vs. Computer: Benchmarking Machine Learning Algorithms for Traffic Sign Recognition. In Neural Networks , 2012.

Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are We Ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In Computer Vision and Pattern Recognition , 2012.

Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation Equivariant CNNs for Digital Pathology. In Medical Image Computing and Computer Assisted Intervention , 2018.

Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A Dataset of 101 Human Actions Classes from Videos in the Wild. arXiv preprint arXiv:1212.0402 , 2012.

Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A Short Note on the Kinetics-700 Human Action Dataset. arXiv preprint arXiv:1907.06987 , 2019.

Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In International Conference on Computer Vision , 2015.

Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. BEATs: Audio Pre-Training with Acoustic Tokenizers. In International Conference on Machine Learning , 2023.

Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In International Conference on Acoustics, Speech and Signal Processing , 2020.

Yanay Rosen, Yusuf Roohani, Ayush Agarwal, Leon Samotorˇ can, Tabula Sapiens Consortium, Stephen R Quake, and Jure Leskovec. Universal cell embeddings: A foundation model for cell biology. bioRxiv , 2023.

Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems , 2022.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, all claims are backed by careful evaluation of related work (see Section 2) and all claims regarding methods' performance are backed up by empirical results (see Section 5).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, the paper discusses limitations in Section 6 of the main paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Yes, for all theoretical results the paper provides a full set of assumptions and complete proof (see Appendix C).

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes, experimental details are described in the experimental setup (see Section 5) and all hyperparameters are given in Appendix D. Furthermore, Appendix B provides a PyTorch implementation for the proposed STRUCTURE regularization.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closedsource models, it may be that access to the model is limited in some way (e.g., to registered

users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Yes, code will be made public once the paper is de-anonymized.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes, experimental details are described in the experimental setup (see Section 5) and all hyperparameters are given in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Yes, the paper reports error bars whenever training was not fully deterministic, such as in Figures 3, 4, 10. However, since our default training and evaluation setup are fully deterministic (backbones frozen, identity-initialized projections, fixed seed and data order), repeating the experiment yields identical results. All standard deviations are therefore zero and have been omitted, as can be seen in Figures 3, 10 where the last dot corresponds to 5 repeats of the training, leading to zero standard deviation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, we specify the computational resources of the paper in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, we have carefully reviewed and adhered to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Yes, the positive societal impact of the paper is discussed in the introduction and conclusion. Negative societal impacts are not expected as the outlined methodology enables multimodal alignment even for limited data scenarios while being computationally efficient.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks as beyond code we are not releasing any new artifacts such as data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes, the authors are the original owners of assets and have specified the referenced assets in the experimental details (see Section 5), dataset descriptions (see Appendix E), and model descriptions (see Appendix F).

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release any new assets beyond code, which is adequately documented in order to reproduce all results from the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The paper does not involve the development of any LLMs, nor their use beyond minor editing.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Computational Resources

Our experiments used a cluster of 8 NVIDIA GeForce RTX 3090 GPUs, but each individual training run required only a single GPU and less than 4GB of VRAM and took at most 2 hours. The most computationally demanding part was obtaining the embeddings from the pretrained unimodal encoders, which on a single GPU with a batch size of 16 has a speed of 3 sec/iter.

B Implementation

C Proof of the generalization gap of STRUCTURE regularization

Unlike classical weight-space penalties such as the ℓ 1 norm, which impose sparsity directly on the model parameters and are agnostic to the number of training examples available, the proposed STRUCTURE regularizer is constructed in the sample space, via pairwise similarities among the N input embeddings. Consequently, its numerical value and statistical behavior depend explicitly on N , the size of the dataset or batch size, respectively. To ensure that this penalty remains well-behaved as the number of samples changes, we introduce its population counterpart , defined as the expectation of the empirical similarity-based regularizer under the true data distribution. In the following section, we will show that the empirical estimator is unbiased and concentrates around its population expectation at a rate of O (1 / √ N ) , thereby providing a computationally efficient yet statistically reliable approximation to the ideal regularization term.

For a fixed number of hierarchical levels L , we define the STRUCTURE regularizer

where P X = softmax ( S X ) = softmax ( ˜ X ˜ X ⊤ τ ) , P A is defined analogously, and P ( l ) X = ( P X ) l , P ( l ) A = ( P A ) l .

We define the empirical and expected values as

In order to prove the generalization gap of the regularization, we investigate how much each sample pair can influence the output and use this to derive the bound using McDiarmid's inequality.

☎

✆

Lemma 2 (Per-sample sensitivity) . Replacing a single pair ( x i 1 , x i 2 ) by an arbitrary (˜ x i 1 , ˜ x i 2 ) , while keeping the other N -1 pairs fixed, changes the value of (10) by at most

Proof. Replacing a single sample ( x i 1 , x i 2 ) alters exactly one row and one column of each similarity matrix S X , S A , which after row-wise softmax means exactly two rows of P X (and two of P A ) can change. We measure the matrix-level deviation by averaging the total variation (TV) distance of corresponding rows,

so changing two rows gives d ( P X , P ′ X ) ≤ 2 N and similarly d ( P A , P ′ A ) ≤ 2 N . Next, we check the influence of the sample perturbation for the Jensen-Shannon divergence

where M = 1 2 ( P X + P A ) . Each KL term is bounded by log 2 , and more generally if TV ( p, p ′ ) ≤ δ and TV ( q, q ′ ) ≤ δ then ∣ ∣ JS ( p, q ) -JS ( p ′ , q ′ ) ∣ ∣ ≤ log 2[ TV ( p, p ′ ) + TV ( q, q ′ )] . Hence, at any fixed level L , the two-matrix, two-row perturbation shifts the JS divergence by at most log 2( 2 N + 2 N ) = 4 log 2 N , and averaging over l = 1 , . . . , L with weights 1 /l cannot increase this bound.

Lemma 3 (Generalization bound) . The generalization gap between the empirical and the expected values of R S is bounded by

Proof. McDiarmid's bounded-difference inequality states that for any function f of independent variables X 1 , X 2 , . . . , X N ,

where c i bounds the change in f when only the i th input is perturbed. Using c i = ∆ N from (11),

Next, we fix a confidence level 1 -δ with 0 < δ < 1 and set the right-hand side of (13) to δ . Solving for ε gives

Hence, for every ε > 0 ,

Thus, with probability at least 1 -δ ,

showing O ( 1 √ N ) convergence.

D Hyperparameters

Table 3 lists all default hyperparameters that were used throughout the paper if not otherwise specified.

E Description of evaluation datasets

We use 22 vision datasets for evaluation of zero-shot classification and two retrieval datasets for evaluation on retrieval tasks, similar to those studied in Radford et al. [7], except for four tasks because of either unavailability of the dataset or lack of diversity in the modality-specific pretrained spaces, leading to random performance. These datasets cover a wide range of vision tasks, including general object classification datasets CIFAR10 [41], CIFAR100 [41], STL10 [42], ImageNet [43], Caltech101 [44]; fine-grained object classification datasets Food101 [45], Flowers [46], Cars [47], FGVC Aircraft [48], Pets [49]; handwritten digits classification dataset MNIST [50]; texture classification dataset DTD [51]; scene classification dataset SUN397 [52]; the facial emotion recognition dataset FER2013 [53]; the satellite image classification datasets EuroSAT [54], Resisc45 [55]; the German Traffic Sign Recognition Benchmark (GTSRB) [56]; the KITTI Distance dataset [57]; the metastatic tissue classification dataset PatchCamelyon (PCam) [58]; action recognition datasets UCF101 [59], Kinetics700 [60]; the country classification dataset Country211 [7]. Furthermore, we include two standard image-text retrieval benchmarks: MS COCO [33] and Flickr30 [61]. For the two video datasets, UCF101 and Kinetics700, we take the middle frame of each video clip as the input of the pre-trained models. We use accuracy for zero-shot classification evaluation and recall@1 (R@1) for retrieval evaluation. For zero-shot classification evaluation, we follow the same setup and use the same prompts as in Radford et al. [7].

F Description of pretrained unimodal encoders

We compare performance when using three self-supervised vision encoders from the DINOv2 family for alignment. Namely, a Vision Transformer Base (ViT-B), Large (ViT-L), and Giant (ViT-G), where each is pretrained on massive unlabeled image corpora using distilled masked prediction objectives to produce rich, high-dimensional patch-level embeddings [3]. For language, we compare RoBERTa, a transformer-based encoder optimized on large-scale English text, which outputs contextualized token representations that are aggregated into a fixed-length sentence embedding [34]. In addition, we leverage two sizes of the Llama3 decoder-only transformer family, namely an 8B and 13B parameter version, which are pretrained on web-scale text data for generative tasks [2]. In all experiments, we freeze the unimodal encoders and learn only lightweight projection layers atop their latent outputs to align them into a shared multimodal embedding space.

G Layers with the highest similarity

Table 5 lists the model combinations in this work along with the layers of each encoder that have the highest representational similarity measured in terms of mutual kNN with k chosen according to Rice's criterion.

H Further results

In this section, we provide a detailed results of the experiments presented in the main paper. In Section H.7, we compare our supervised alignment method to an unsupervised approach, Section H.6 presents details on the ablation study that isolates the effects of regularization and layer selection, Section H.2 analyzes the consistency of our layer selection procedure across different validation set sizes, and Section H.8 presents detailed results of the downstream performance on zero-shot classification and retrieval tasks.

H.1 Layer-wise performance

In Section 4, we propose selecting layers for alignment based on their representational similarity rather than defaulting to the final layers. To provide a more comprehensive view, Figure 8 shows a detailed layer-wise analysis between the last five layers of a RoBERTa encoder and a DINOv2 ViT-L encoder in terms of the respective downstream performance these combinations achieve after alignment. Results show that the most suitable layers (here 24/22 for the language and vision encoder pair) yield strong performance across all datasets. Additionally, we can see that for CIFAR10 and UCF101, there is a tendency that later vision layers lead to better performance while the choice of the language layers is less influential.

20

21

22

23

24

Language

model layer index

19

20

21

22

23

Vision

model layer index

88.2

87.2

86.8

87.3

88.6

84.4

86.0

82.3

88.4

91.7

80.5

80.2

93.6

83.6

93.4

95.1

95.9

95.2

93.3

96.2

83.5

83.4

83.7

83.4

86.1

stl10

82.5

85.0

87.5

90.0

92.5

95.0

20

21

22

23

24

Language

model layer index

19

20

21

22

23

Vision

model layer index

80.1

80.3

84.0

84.4

76.8

82.5

79.0

84.1

79.8

81.1

76.5

81.1

90.7

76.8

86.3

94.6

92.0

95.9

94.6

95.6

87.9

89.6

90.4

91.5

92.7

cifar10

80.0

85.0

90.0

95.0

20

21

22

23

24

Language

model layer index

19

20

21

22

23

Vision

model layer index

46.8

50.4

49.9

49.5

51.2

45.7

50.5

45.0

50.0

50.9

40.6

47.6

48.9

49.3

55.2

54.4

53.9

53.8

54.5

56.8

46.1

48.5

48.5

48.0

51.2

caltech101

42.5

45.0

47.5

50.0

52.5

55.0

20

21

22

23

24

Language

model layer index

19

20

21

22

23

Vision

model layer index

20.8

21.3

20.2

22.0

20.9

21.9

22.9

17.2

25.7

23.9

19.4

18.2

23.0

22.5

23.1

29.7

31.0

28.3

30.6

32.5

31.0

30.8

31.7

32.8

32.0

ucf101

17.5

20.0

22.5

25.0

27.5

30.0

32.5

H.2 Layer selection consistency

Figure 9 evaluates the stability of layer selection by repeating the process 100 times on random validation subsets of varying sizes ( N = 100 , 500 , 1000 , and 5000 ) for mutual kNN with k=rice. All metrics illustrate high robustness against repeated selection on different subsets, justifying our choice of a 5000-sample set to ensure reliable layer pairing across runs.

0

6

12

18

24

Language

model layer index

0

5

11

17

23

Vision

model layer index

Mutual kNN (k=rice)

0

10

20

30

40

Selection %

(a)

N

= 100

0

6

12

18

24

Language

model layer index

0

5

11

17

23

Vision

model layer index

Mutual kNN (k=rice)

0

20

40

60

80

100

Selection %

(b)

N

= 500

0

6

12

18

24

Language

model layer index

0

5

11

17

23

Vision

model layer index

Mutual kNN (k=rice)

0

20

40

60

80

100

Selection %

(c)

N

= 1

,

000

0

6

12

18

24

Language

model layer index

0

5

11

17

23

Vision

model layer index

Mutual kNN (k=rice)

0

20

40

60

80

100

Selection %

(d)

N

= 5

,

000

H.3 Low-data comparison layer selection

Figure 10 compares the layer selection strategy when scaling down the training data using the same setup as for Figure 3 of the main paper. Results show a minor difference in performance and utility when using the most similar layers for alignment. However, results show that the proposed layer selection strategy is a simple addition to existing alignment methods as the method includes choosing the last layers to align if they indeed are the most similar. This can be clearly seen as the performance across tasks is positive. Thus the selection process can additionally boost performance at minimal additional overhead.

1k

5k

10k 50k 80k

40

60

80

Zero-Shot

U

= 0.6

STL10

1k

5k

10k 50k 80k

40

60

80

U

= 0.4

CIFAR10

1k

5k

10k 50k 80k

10

20

30

U

= 0.9

Caltech101

1k

5k

10k 50k 80k

10

20

U

= 0.0

UCF101

1k

5k

10k 50k 80k

5

10

15

Zero-Shot

U

= -0.0

Food101

1k

5k

10k 50k 80k

10

20

30

U

= 0.2

CIFAR100

1k

5k

10k 50k 80k

5

10

U

= 0.5

ImageNet

1k

5k

10k 50k 80k

5

10

U

= 0.1

Kinetics700

1k

5k

10k 50k 80k

10

20

Zero-Shot

U

= 0.4

SUN397

1k

5k

10k 50k 80k

10

20

U

= 0.0

Resisc45

1k

5k

10k 50k 80k

2.5

5.0

7.5

U

= 1.3

Pets

1k

5k

10k 50k 80k

1

2

U

= 3.7

Flowers

1k

5k

10k 50k 80k

Sample Size

10

20

Retrieval

R@1

U

= 0.5

Flickr30 T2I

1k

5k

10k 50k 80k

Sample Size

20

40

U

= 0.3

Flickr30 I2T

1k

5k

10k 50k 80k

Sample Size

0

2

4

U

= 0.3

COCO T2I

1k

5k

10k 50k 80k

Sample Size

0

5

U

= 0.1

COCO I2T

Last

Similar

H.4 Application to other modalities

Audio-text. We further evaluate the proposed method for audio-language alignment. Here, as a pretrained unimodal encoder, we use BEATs-iter3 [62] for audios, and extract text embeddings using RoBERTa [34]. We train a linear alignment on 3 , 839 audio-caption pairs from the Clotho dataset [63].

To evaluate cross-modal alignment, we perform audio-to-text and text-to-audio retrieval on a held-out set of 1 , 045 pairs from the same dataset. As shown in Table 6, our STRUCTURE regularization significantly improves the retrieval performance compared to the baseline without regularization. This demonstrates the generalizability of our regularization method across modalities beyond vision-language alignment.

Single-cell image-transcriptome. In addition to the audio application, we have conducted additional experiments on the biological domain using only 19,900 paired human single-cell image-transcriptome data. In

this experiment, we linearly align a UCE [64] encoder for the transcriptome data and a masked autoencoder (MAE) trained on single-cell images for the image data. We split the 19,900 paired samples into 100 training samples and 19,800 test samples to simulate the low data setting. Results below show that our regularization also helps in the biological domain, yielding relative improvements of 43.1% on the retrieval task and 30.3% on the cell type classification task, even if the Platonic representation hypothesis is not guaranteed in the biological domain.

H.5 Modality Gap Analysis

We measured the modality gap [65] for three different alignment setups for a ViT-Giant and RoBERTa Large, trained on COCO with a linear alignment function. The modality gap was measured for the two image-text datasets considered in our submission (i.e., COCO and Flickr30) to ensure that the comparison is influenced solely by the data itself and not by any other artifacts introduced by the evaluation, such as a zero-shot template, as would be necessary for the classification datasets. Interestingly, the results given in Table 8 show that the modality gap indeed decreases when first selecting the most similar layers and then further when applying STRUCTURE regularization. This aligns with the findings from the respective paper [65], that the modality gap indeed has a relationship with downstream performance.

H.6 Ablation study

Table 9 examines the influence of different hyperparameters of the proposed method. Specifically the number of hierarchical levels in the STRUCTURE regularizer, regularization strength λ , layer selection strategy, normalization schemes, and distance functions. Varying the number of levels from one to five yields virtually identical accuracy across datasets, indicating that a single-level penalty suffices to preserve pretrained geometry. A strong regularization weight λ = 10 gives the best performance ( e.g. , STL10 at 92.6%, CIFAR10 at 96.3%), whereas λ ∈ { 0 . 1 , 1 } under-regularizes leading to worse results. Without regularization, choosing the last layer yields 75.6% on STL10 and 85.5% on CIFAR10, but using mutual kNN layer selection boosts these to 79.7% and 89.0%, respectively. These results confirm that both strategic layer choice and sufficient regularization are critical for maximally leveraging low-data regimes. Additionally, Table 12 and Figure 7 from the main paper confirm that best-layer alignment and STRUCTURE regularization consistently increase performance across four language-vision model pairs. For normalization schemes and distance functions, STRUCTURE demonstrates robustness to these changes, highlighting that the primary benefit stems from the idea of regularizing hierarchical relationships.

The variable N in the default setting of STRUCTURE represents the size of the batch, typically set to 4 , 096 random samples (here approximately 6% of the entire dataset), which allows for a sufficient estimate of the true data generation distribution while maintaining computational efficiency. In Table 9, we also conducted experiments to demonstrate how the choice of sample size affects performance. Results show that the benefit of the regularization increases with increased sample size, but also show its efficiency, since even with 128 samples, there is a clear improvement. Thus a practical choice is to use bigger batch sizes if possible. Additionally we investigated the choice of randomly selecting the subset of samples used to compute the regularization. Results show that the regularization yields very similar results regardless of whether it uses a fixed (same-sized) or a random subset of the data.

H.7 Comparison to unsupervised methods

Table 10 compares our supervised alignment method (Linear + Similar + R S ) with unsupervised Kernel Local CKA [12] and ASIF [23]. Across all three evaluation tasks (COCO, CIFAR-10, and CIFAR-100), our method yields significant improvement in both top-1 and top-5 accuracy while relying on only 80,000 paired samples. For instance, for COCO retrieval, Kernel Local CKA attains only 13.1% R@1, whereas the proposed method reaches 32.7% (a 19.6 pp. improvement) and improves R@5 from 40.0% to 56.2%. Similarly, on CIFAR10 our approach jumps from 0.06% to 96.8% top-1, and on CIFAR-100 from 1.13% to 46.9%. Compared against ASIF, results demonstrate that our method again substantially outperforms it, achieving relative improvements of 32.9% on the COCO retrieval task, 32.9% on the CIFAR-100 fine-grained classification task, and 5.6% on the CIFAR-10 coarse-grained classification task.

H.8 Extended downstream results

Table 11 expands on the results in Table 2 from the main paper, detailing results on all evaluation tasks, including 22 zero-shot classification and two retrieval datasets with an additional fourth alignment technique (FuseMix [22]). Figure 11 expands on Figure 3 with more zero-shot classification and retrieval tasks. Figure 12 illustrates the performance of model combinations across different vision models, analogous to Figure 7 from the main paper, but with more combinations. Table 12 expands on Figure 7 from the main paper, detailing results on all evaluation tasks, including 22 zero-shot classification and two retrieval datasets.

1k

5k

10k 50k 80k

50

75

Zero-Shot

U

= 10.9

STL10

1k

5k

10k 50k 80k

50

75

U

= 17.3

CIFAR10

1k

5k

10k 50k 80k

20

40

60

U

= undef.

Caltech101

1k

5k

10k 50k 80k

20

40

U

= 33.0

UCF101

1k

5k

10k 50k 80k

10

20

30

Zero-Shot

U

= 9.6

Food101

1k

5k

10k 50k 80k

20

40

U

= 23.1

CIFAR100

1k

5k

10k 50k 80k

0

10

20

U

= 35.8

ImageNet

1k

5k

10k 50k 80k

0

10

20

U

= 25.4

Kinetics700

1k

5k

10k 50k 80k

0

20

40

Zero-Shot

U

= 23.5

SUN397

1k

5k

10k 50k 80k

10

20

30

U

= 8.4

Resisc45

1k

5k

10k 50k 80k

5

10

15

U

= 32.5

Pets

1k

5k

10k 50k 80k

0

2

4

U

= 33.6

Flowers

1k

5k

10k 50k 80k

Sample Size

0

20

40

Retrieval

R@1

U

= 22.4

Flickr30 T2I

1k

5k

10k 50k 80k

Sample Size

0

25

50

U

= 5.7

Flickr30 I2T

1k

5k

10k 50k 80k

Sample Size

0

5

10

U

= 23.5

COCO T2I

1k

5k

10k 50k 80k

Sample Size

0

10

U

= 10.4

COCO I2T

Last

Similar +

S

Last

Similar

Similar +

S

ViT-B

ViT-L

ViT-G

79.0

79.0

90.6

79.1

90.2

97.3

75.6

79.7

92.6

STL10

Last

Similar

Similar +

S

79.4

79.4

93.7

84.8

86.4

97.0

85.5

89.0

96.3

CIFAR10

Last

Similar

Similar +

S

13.4

13.4

27.2

14.2

12.8

23.1

14.8

14.6

30.6

FOOD101

Last

Similar

Similar +

S

30.4

30.4

41.6

32.8

30.8

50.3

34.0

33.1

51.3

CIFAR100

0

20

40

60

80

100

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents STRUCTURE, a novel regularization technique that preserves the neighborhood structure of the latent space in pretrained unimodal encoders. It enables more effective and efficient multimodal alignment, especially in low-data training scenarios. The technique can be added to existing alignment methods and helps maintain the geometric relationships in the input representations. The authors show that their method performs well with significantly less training data, resulting in better alignment and improved performance in downstream tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Requires less training data to achieve strong alignment, making it suitable for low-resource settings.\n- As a regularization method, it can be integrated into many existing alignment frameworks.\n- The paper includes a strong set of experiments across a wide range of benchmarks.\n\nWeaknesses:\n- The unimodal models used (RoBerta and DinoV2 Giant) are very deep and computationally expensive. STRUCTURE shows more benefits when aligning highly similar layers, but these benefits may decrease when using smaller or weaker models (simple Resnets or base ViT).\n- The proposed framework is only tested on two modalities. There is no discussion or evidence on how it would extend to three or more modalities. Moreover within the two modal case, just text and image modalities are tested.\n- It's not clear whether the variable N in Eq. (2) and lines 137–138 refers to the size of the whole dataset (as suggested in line 105) or just the batch size during training. If it's the batch size, the similarity matrix may only capture local (within-batch) relationships, which could limit global alignment quality.\n\nQUESTIONS:\nQuestions:\n- The definition of hierarchical level L in Eq. (4) is not intuitive. Could the authors give more insight or examples to help understand its role?\n- Have the authors tested STRUCTURE with smaller, less powerful networks? This would help understand whether the method still works when the pretrained representations are less aligned.\n- The paper [A] (\"Mind the Gap\", NeurIPS 2022) highlights the modality gap in contrastive models like CLIP. Since STRUCTURE is used with alignment losses (often contrastive), is this gap still present in the shared embedding space $\\mathcal{A}$? Does STRUCTURE help reduce the modality gap, or does it preserve it? How does preserving structure interact with or influence this phenomenon?\n\n[A] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. NeuIPS 2022.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-11"}

comments: SUMMARY:
This paper presents STRUCTURE, a multimodal alignment method that can effectively align two unimodal foundation models into one shared representation space, with only limited labeled (paired) data. The key to STRUCTURE is to enforce consistency between structural relationships (from the original unimodal space to the common space). And the authors further show that selecting the similarity-based layer is better for alignment than simply selecting the last layer.

STRENGTHS AND WEAKNESSES:
Strengths:

-This paper is well-written.

-The idea of preserving hierarchical relationships for alignment is technically sound and interesting.

-The proposed regularization term and layer selection strategy can be seamlessly integrated into other methods to enhance the alignment.

-The results are strong and comprehensive.

Weaknesses:

-The proposed regularization uses $l$-hop reachability to calculate the similarity relation for different hierarchical levels. It seems that introducing such hierarchical relationships is a main contribution of the regularization. However,  as shown in Table 3 and Table 7, integrating the information from multiple hierarchies seems unnecessary. And the hierarchical level $l$ is set to 1 in practice, where only the sample-level relation is considered. The authors should discuss how to effectively use the hierarchical relation. Ideally, the alignment effect would increase as $l$ increases and gradually converge. 

-It is also suggested to discuss $l$-hop when first introduced, which would make it easier for readers to understand.

QUESTIONS:
Please see weaknesses.  I will raise the score if the concerns about hierarchical relations are resolved.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a regularization scheme for multimodal alignment on top of frozen pre-trained unimodal encoders, fit for settings where multimodal data is scarce. The authors also propose a method to select the layer of the unimodal encoders with which to perform multimodal alignment.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The empirical results are strong and convincingly demonstrate the efficacy of the proposed regularization scheme and layer-selection strategy. The authors additionally provide good supporting analyses and ablations to further corroborate their claims.\n- The proposed regularizing loss seems novel and is well-motivated from the lens of maintaining the semantic structure of the latent space of the underlying unimodal encoders. The layer-selection strategy is also interesting and novel, and Figure 2 does a good job of motivating the need for such a strategy.\n- The paper is well written with a clear methodological description and motivation for design choices.\n\n## Weaknesses\n- This paper is very similar in spirit to FuseMix [1] from CVPR 2024 but it was never mentioned in this work: the motivation of using pre-trained and frozen unimodal encoders to combat the scarcity of multimodal data was already explored in FuseMix, even with tens of thousands of multimodal pairs. FuseMix also has a similar setup in terms of proposing a regularization scheme (via a mixup augmentation routine). I acknowledge that there are still differences between this work and FuseMix in terms of the proposed methodology, but some of the key claims and motivations of this paper are not novel so I would ask the authors to include a discussion of their work compared to FuseMix (both in their rebuttal and in the updated manuscript) and include it as a baseline with experimental results. Some of the paper's claims are overstated.\n- The authors do not discuss limitations of their proposed method.\n\n## References\n[1] Vouitsis, Noël, et al. \"Data-efficient multimodal fusion on a single gpu.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nQUESTIONS:\n- How are the k samples per class in Figure 5 incorporated into the training set? Are they oversampled or just become another element in the dataset?\n- Have the authors considered using text embedding models instead of RoBERTa or LLMs, as was done in FuseMix?\n\nAlso see weaknesses above with corresponding questions. Should my concerns and questions be adequately addressed, I will consider increasing my score.\n\nLIMITATIONS:\nThe authors do not acknowledge limitations or potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a structure-based regularizer applicable on any latent space transformation. The core idea is to keep the geometry (i.e., local neighborhoods) of the space consistent before and after the transformation. The main proposed application is in the context of building a multimodal model out of independently pre-trained and frozen unimodal foundation models. In the experiments, the proposed regularizer is applied independently on each modality-specific space when optimizing a variety of already existing alignment methods, showing that : i) it allows alignment methods to use significantly fewer paired samples, thanks to the induced structural prior; ii) it substantially increases their performance on different tasks (zero-shot classification and retrieval) but also with different backbone pairs and modalities (text-image, text-audio) ; iii) it is robust to the number of datapoints used to compute it; iv) it can handle distribution mismatches with few in-domain samples.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths (Quality, Clarity, Significance)\n\n1. The paper is well-rounded. It has a high variety of experiments and ablations that validate the proposed method.\n2. I found it extremely easy to follow. From the start and throughout it, it clearly communicates the high-level problem, the proposed approach, why it is relevant, and, at each step/paragraph, what is being done and why. The reader is guided through it. This is due to a mix of strong writing and well-structured presentation.\n3. It clearly articulates and concretely shows the advantages of imposing a structural prior when trying to bridge different modalities with a simple and effective method. I believe it has the potential to be widely adopted and/or extended in this subfield.\n\n## Minor Weakness\n\nI believe a core citation/comparison missing is [a], a method that is similar with respect to: i) the task, since it creates CLIP-like models starting from pre-trained and frozen unimodal encoders; ii) the procedure, since it relies on their **similar neighborhood structure.** More in general, I think that some room in the related works should be dedicated to works in representation alignment methods for latent spaces, especially the ones focusing on geometric priors [e.g., b, c, d, e]. I would suggest broadening the “Platonic representation hypothesis.”, reframing it to this more general topic.\n\n---\n\n**[a]** ASIF: Coupled data turns unimodal models to multimodal without training; Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodolà, Francesco Locatello; NeurIPS 2023.\n\n**[b]** Relative representations enable zero-shot latent space communication; Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, Emanuele Rodolà; ICLR 2023.\n\n**[c]** Latent space translation via semantic alignment; Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, Emanuele Rodolà; NeurIPS 2023.\n\n**[d]** Offline bilingual word vectors, orthogonal transformations and the inverted softmax; Samuel L. Smith, David H. P. Turban, Steven Hamblin, Nils Y. Hammerla; ICLR 2017.\n\n**[e]** A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings; Mikel Artetxe, Gorka Labaka, Eneko Agirre; ACL 2018.\n\nQUESTIONS:\n## Minor comments/questions\n\n1. It took me a while to understand the difference between trustworthiness and continuity when they are introduced. A different phrasing might help. What clarified it for me was thinking of it as a binary classification task with labels \"neighbor\" and \"not neighbor\" and the ground truth being the labeling in the original space while the prediction being the new one. This made it easier to relate trustworthiness as capturing \"false positives\" (precision), while continuity to \"false negatives\" (recall).\n2. I would argue that 1000 labeled samples are \"extremely resource-constrained\" when compared to the massive scale at which multimodal models like CLIP are trained, not in general. As an example, consider visual alignment of low-resourced languages, or scientific data where labeling requires costly wet-lab experiments.\n3. Reference [9] is from CVPR 2025, not 2024.\n4. I think the “Train-test data distribution shift.” is undersold. The fact that as few as 10 samples per class enhance the performance this much should be better highlighted! At the same time, I’m wondering if adding a mix of 10 samples for each class in each dataset has a consistent improvement across all datasets, or if specific samples for the task are needed while other specific ones might have a noising effect when added.\n5. Why are the scaling up and down different experiments and not unified?\n6. Where are the most suitable layers for the alignment found? What’s the similarity distribution?I would find a heatmap comparing each layer of the first model to each other layer of the other model quite interesting and informative to how this matching is helping. If substantial, this analysis could replace one of the two scaling up/down plots if they were merged.\n7. Table 2 is already pretty comprehensive, but I was wondering if an extra column could be squeezed in to show the row-wise average and variance. It would help when comparing them.\n\n## Major Questions\n\n1. Are the classification/retrieval errors uniform enough across classes? Are there cases where almost entire classes are consistently misclassified? It would makes sense if they cannot be reliably mapped across spaces or their relative positions with respect to other classes significantly differ across modalities. It might be helpful checking macro/weighted F1 instead of the coarser accuracy, or to analyze the error variance/distribution across classes for each dataset.\n2. Is there an intuition behing this specific choice of normalization (i.e., first L2 norm, then centering)? Have you tried different ones like Standard Scaling or first centering and then normalizing? Same for the distance function used.\n3. My understanding is that STRUCTURE is a regularization term that acts independently on each space, but is applied on all the spaces to be aligned to preserve their structure in the process. However, the regularization term doesn’t rely on paired data, so I was wondering if the authors could provide tests in a setting where the **paired** training data for the alignment function is different from the **unpaired** training data for the regularizer. This would help in understanding if the structural prior that STRUCTURE maintains is generalizable to the whole original space, or if it is tied to the specific subspace described by the paired samples.\n4. Is the STRUCTURE loss always computed on all the data used to train the alignment function? Take as an example the Figure 3 setting, where the (paired) training data is increased to show its robustness. Varying the training data while computing the STRUCTURE regularization only on a fixed subset (even **unpaired, a random subset of the original space,** to combine this analysis with the one described in the previous bullet) would be a stronger signal that STRUCTURE itself needs less data to be effective.\n\n---\n\nWhile not a breakthrough, the method is elegant, well-motivated, and practically impactful. Combined with the writing clarity, it makes it a clear accept for me. Among the questions in the list, my priority to consider increasing the rating would be a better disentangling and characterization *regarding the needed amount of in-domain paired data between STRUCTURE and the alignment function*.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}