With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You
Authors: Fabian Gröger
Shuo Wen
Huyen Le
Maria Brbic
Venue: NeurIPS 2025 poster ()
Keywords: multimodal learning
vision language models
representation learning
Primary area: deep_learning
TL;DR: Aligning pretrained unimodal models with the proposed framework using limited paired data yields ~52% gains in cross-modality zero-shot classification and ~92% in retrieval.

Abstract: Multimodal models have demonstrated powerful capabilities in complex tasks requiring multimodal alignment, including zero-shot classification and cross-modal retrieval. However, existing models typically rely on millions of paired multimodal samples, which are prohibitively expensive or infeasible to obtain in many domains. In this work, we explore the feasibility of building multimodal models with limited amount of paired data by aligning pretrained unimodal foundation models. We show that high-quality alignment is possible with as few as tens of thousands of paired samples$\unicode{x2013}$less than 1\% of the data typically used in the field. To achieve this, we introduce STRUCTURE, an effective regularization technique that preserves the neighborhood geometry of the latent space of unimodal encoders. Additionally, we show that aligning last layers is often suboptimal and demonstrate the benefits of aligning the layers with the highest representational similarity across modalities. These two components can be readily incorporated into existing alignment methods, yielding substantial gains across 24 zero-shot image classification and retrieval benchmarks, with average relative improvement of 51.6\% in classification and 91.8\% in retrieval tasks. Our results highlight the effectiveness and broad applicability of our framework for limited-sample multimodal learning and offer a promising path forward for resource-constrained domains.

With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You
Fabian Gröger 22,,, 1,,3, Shuo Wen'.", Huyen Le', Maria Brbić'
EPFL University of Basel 3SSLU
brbiclab.ept wpt.l..chprojjecss//ttttttt.
Abstract
Multimodal models have demonstrated powerful capabilities in complex tasks requiring multimodal aligmment, including zero-shot classification and gross-modal retrieval. However, existing models typically rely millions of paired multimodal samples, which are prohibitively expensive or infeasible to obtaín in many domains. In this work, we explore the feasibility of building multimodal models with limited amount of paired data by aligning pretrained unimodal foundation models, We show that high-quality alignment possible with as few tens of thousands of paired samples less than 1% of the data typically used in the field. To achieve this, introduce STRUCTURE,
effective regularization technique that preserves the neighborhood geometry of the latent space of unimodal encoders. Additionally, show that aligning last layers is often suboptimal and demonstrate the benefits aligning the layers with the highest representational similarity across modali- ties. These components can be readily incorporated into existing alignment methods, yielding substantial gains across 24 zero-shot image classification and retrieval benchmarks, with average relative improvement of 51.6% in classification and 91.8% in retrieval tasks. Our results highlight the effectiveness and broad applicability of our framework for limited-sample multimodal learning and offer a promising path forward for resource-constren domains.
1 Introduction
Unimodal foundation models (FMs) have achieved remarkable progress in recent years, demon- strating high performance on variety of complex tasks across diverse domains. Large language models (LLMs) 2 have shown unparalleled abilities across wide range of natural language un- derstanding and generation tasks, reaching human-level performance on many complex reasoning and comprehension tasks. parallel, vision FMs 3 have delivered similar breakthroughs in visual per- ception, enabling systems to perform tasks such as object recognition and segmentation with minimal supervision. In scientific domains, models such as AlphaFold (5] and ESM2 (6] have demonstrated the potential of aarge-scale models to solve highly specialized and complex problems, including accurate protein structure prediction and sequence-based modeling biomolecular functions.
While these models are highly effective within their respective modalities, many applications involve multimodal data, where is important to map different modalities into a shared representation space to enable meaningful cross-modal comparison, retrieval, and classification. Pioneering multimodal models like CLIP 7 and CLAP g align visual and auditory inputs with language through contrastive learning and achieve strong zero-shot performance across broad spectrum of multimodal tasks. However, these powerful multimodal models rely on vast amounts of paired training data (e.g., CLIP uses 400M pairs), which are often unavailable in many domains like healthcare and biology, where collecting high high-tuality multimodal data is expensive and fabor-intensiv.
Joint first authorship
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Modality
Modality
Aligmmet
Enc
f
Modality 1
Pretrained space
we align
using only limited samples?
TEXT
Einc
f2
Shared space
Modality 2
samples modality
Frozen
Enc: Encoder
comrsponn
aligned
misaligned
samples of modality 2
Trainable
f: Aligmment function
Figure I: Overview of the proposed approach for cross-modal alignment with limited data. The objective is align representations from two modalities (e.g., images and text) into shared em- bedding space. The central challenge is guiding the model toward well-aligned solution, rather than a misaligned one, when only small amount paired data is available. The key idea to freeze pretrained encoders and learn lightweight alignment functions that preserve each modality's pretrained latent structure during alignment.
A promising direction to address this limitation is leverage powerful representations from pretrained unimodal FMs for multimodal alignment. However, existing alignment methods [ still require large quantities of paired examples, often tens of millions, to learn shared embedding space effectively. On the other hand, unsupervised techniques 12 13 do not use labeled data but primarily perform sample-level matching, thus failing construct shared embedding space. This opens a fundamental question: Can we align FMs from different modalities into shared representation space using limited number of multimodal samples?
In this work, we answer this question affirmatively by introducing a modular alignment strategy that requires few tens of thousands of paired examples, less than 1% of the data used by existing modalities aligmment methods [9] Specifically, we introduce STRUCTURE, novel regularization technique designed to preserve the multiscale neighborhood structure of each unimodal latent space during alignment, ensuring that the relationships between samples captured by the unimodal encoders retained. In addition, based the observation that higher similarity correlates with better alignment performance, we propose to align those layers with the highest representational similarity. Incorporating both components into existing alignment methods leads consistent relative performance gains across 22 zero-shot classification and two retrieval benchmarks, averaging 51.6% in classification and 91.8% in retrieval, demonstrating superior performance low-data settings. Finally, we show that incorporating just a few labeled examples per class from the target domain into the training set can effectively bridge the performance gap to large-scale multimodal models trained on hundreds of millions of multimodal data, highlighting the importance of domain coverage over sheer data volume in multimodal alignment.
Related work
Multimodal foundation models. Recent advances in multimodal FMs have produced unified architectures capable of understanding and generating content across vision and language. Farly models such as CLIP 7 and ALIGN [14] use contrastive learning to align image and text embeddings. achieving strong zero-shot performance in classification and retrieval. In contrast, models like BLIP [15] and GIT [66] adopt generative pretraining support tasks such image captioning and visual question answering. More recent efforts, including FLAVA [17,, PaL. (18), Kosmos-1 [19]. and Gemini (20), aim for broad task coverage through unified multimodal training, reflecting shift toward scalable, general-purpos. multimodal systems. Despite their impressive capabilities, existing multimodal models rely heavily on massive paired datasets, often hundreds millions of samples. In contrast, in this work we explore how to align powerful unimodal models that are readily available in many domains using only limited multimodal supervision, reducing reliance on large-scale paired data and enabling multimodal learning in low-data regimes.

---

Parameter-frozen modality alignment. Parameter-frooe modality alignment is an efficient strat- egy for building multimodal systems by aligning pretrained unimodal encoders without updating their internal weights. Unsupervised alignment methody seek to align representations without relying on paired data. Techniques such as Centered Kernel Alignment [12] focus on maximizing representa- tional similarity between modalities, leveraging shared structural patterns. However, a key limitation of these methods [3 [2 [1] is that they primarily perform sample-level matching, thus failing to construct shared embedding space. Furthermore, they are not able to utilize paired data even when available, missing out on rich alignment signals that could significantly enhance performance. In contrast, supervised alignment methods assume access to abundant paired data, such as millions of paired samples, and train lightweight alignment modules (e.g.. linear projections or MLPs) while keeping the encoders frozen. These approaches ( [O] have achieved strong performance in tasks like zero-shot classification and retrieval. However, these methods are limited by their dependence on large-scale multimodal datasets, requiring tens of millions of paired samples to effectively train the models. In contrast, our work explores parameter-froe frozen alignment in a low-data regime, with tens of thousands of paired samples, which is 0.02% of the paired samples used by previous works. To address the challenges of low-data regimes, we propose an effective strategy that can be easily integrated into existing methods [ 10 to make them more effective under low-data conditions.
Other works have also focused on data-efficient alignment. FuseMix [22] shares our motivation of using limited paired data but focuses improving training efficiency via latent space augmentations. Our geometric regularization and layer selection are complementary to their approach, and as shown in our experiments, combining them yields further gains. Learning- free methods like ASIF [ also leverage neighborhood structure, but for direct matching without training an alignment function.
Platonic representation hypothesis. The Platonic representation hypothesis [24] suggests that models from different modalities can converge to similar internal representations. This insight motivates the possibility of aligning separate representation spaces. While earlier multimodal FMs like CLIP 7 depend on end-to-end training with vast amounts of paired data, the Platonic view offers a more efficient alternative: aligning independently trained unimodal models within a shared embedding space. This approach opens the door to scalable and flexible multimodal learning without the need for joint training. and it has recently spurred interest in supervised and unsupervised alignment techniques [9] 12). Building on this direction, which is related to the broader field of geometric priors in representation alignment 25 28 1 our work explores its potential in the challenging low-data setting.
3 Problem setup
We consider the task of aligning representations from independently pretrained encoders across different modalities. As illustrated in Figure ! we keep the encoders frozen and learn lightweight alignment functions that map from each modality's 's latent space into shared space where semantically related samples are close. We next formalize this setup.
Modality alignment with pretrained representations. Let Xi C Rd: and Xr C Reda be the latent spaces of two pretrained unimodal encoders, which correspond to outputs from either the last or intermediate layers of the encoders, and di and d2 be their respective dimensions, which do not need to be equal. Given a set of N paired multimodal samples D = a² E X2. our goal is to learn two alignment functions fi R + A and fo Rd - A, which map the modality-speciinn Alignment in the shared space is achieved when the paired samples (zj. x²)) are closer to each other than to any non- non-paired sample. Specifically,
where o E X2 and
spaces X1 and X2 into a shared embedding space A S I#* of dimension k.
sim sim(f1(zi), f2(xb)) 2 (2)( fa(z))) Vj 3 i,
where sim(.) denotes a similarity function, such as cosine similarity.
Different from previous works, we focus this alignment problem under the challenging condition of limited paired data. Specifically, we focus on the scenario when N is relatively small (i.e., tens of thou- sands of samples) compared to tens of millions of paired samples considered in previous works (9] 10)
We unify different alignment methods under a joint framework that consists of three main components: (i) moodallttyspeccfff latent spaces X1 and X2. (ii) alignment functions fi and fz, and (iii) the objective
3

---

Table 1: Overview of existing methods and our approach within the modality alignment framework. Le represents the standard symmetric contrastive losses from CLIP 77. Rg denotes STRUCTURE, a regularization term proposed in our work.
Method
Modality-specifie spaces
Alignment functions f Objective function LA
Linear mapping 10
Last layers
Lincar
Le Lc
Non-linear mapping 9
Last layers
ML.
Token alignment
9
Last layers
Token level MLP
Lc
CSA (29
Last layers
Linear
Reformulated Lo
Our work
Most similar layers
Any
Any Rs
function LA. which guides the construction of the shared space A. Table summarizes how existing methods instantiate each of these components. In our work, we propose general framework that can work with any alignment function and any objective function by regularizing the objective using the STRUCTURE regularization R.s, and using as modality- modality-specific latent layers those layers that have highest representational similarity. We introduce both components in the following section.
Multimodal alignment with limited data
To align pretrained unimodal encoders using limited number of paired multimodal samples, propose two key components: (i) STRUCTURE, a regularization that preserves the intrinsic geometry of each modality's latent space, and (ii) strategy for selecting modality-specifie latent space, those layer pairs that have highest representational similarity. Both components can be seamlessly integrated into existing alignment methods [9] 10 29
(see Table TT.
Preserving neighborhood via STRUCTURE regularization. With a limited number of paired samples, it is crucial to preserve the latent structure of pretrained unimodal encoders, trained on millions or even billions of examples, as they encode meaningful relationships between samples. To achieve this, we introduce STRUCTURE, a regularization term that preserves the neighborhood relationships of the pretrained space of unimodal encoders within the aligned space.
Given a podallty-sppccfii space X' C Rd and its corresponding shared space A C R where X' can either Xi or X2, the regularization term aims to ensure the hierarchical (i.e., multi-scale) consistency between the relationships expressed by X' and A. Specifically, each sample E X and a, E (i.e., ag = =((x.) (xx;)) is first noooralll,e 3. = s. /////1//122211 122 a. = a4/1|ail2, and then centered to remove any global translation bias:
-
Eソ=
ã ã.
N [ T2 ... FNJ E Nxd and Ā
Σソ= aj.
We denote the normalized and centered matrices [ā1, a., _ xk
X
The (scaled) similarity matrices are computed with temperature 7 as:
Sx X SA AAT
To interpret the similarities probability distributions, we apply softmax function row-wise. Specifically, we define Px softmax(Sx). PA = softmax(SA) as the corresponding matrices with the similarity distributions of the respective spaces.
To capture relationships reachable by exactly hops on the similarity graph, we define for each hierarchical level ( I,...,,, L, where L E N is the total number of levels")
( = (Px)',
PM = (PA)".
Intuitively, Px can be seen as the transition matrix of random walk on the data manifold. The l-hop matrices thus capture multi-step relationships and progressively more global structure.
For square matrix P. the power pd is defined by repeated matrix multiplication, f.e., p = P...PP (with/ factors), and should not be confused with entrywise exponentiation, generally (P'),, # Pby.

---

The key idea of our regularization is to enforce consistency between the structural relationships, i.e. the relative positions and neighborhood structure the embedding space, captured by X and A. Thus, we employ Bensen-Shanne divergence because of its symmetric nature to measure the divergence between similarity distributions. Specifically, at each level we define the level-specific divergence as:
(S : pp()) DKL. ( M
1 DKi. (P() ) +
+
(5)
where
M ( P(1
(6)
and DKL is the Kullback-Leibler divergence. In practice, a small constant added inside the logarithm for numerical stability, ie.,, p + and P\ +c.
The final formulation of the STRUCTURE regularizer is a weighted average of the divergences across levels, where lower levels are weighted more heavily to counteract the more concentrated distributions of the higher levels:
L IS P20)
RS. 11 (X, A) = L
(X,
We denote RS as the regularization that operates on levels, and set it to if not otherwise specified. Together with any objective function La used for representation alignment (e.g., Le in work (9 10 the combined loss defined as
L = LA LA A (1) (X1, (x),)) (X2,
(8)
Reg. for Modality
Rog. for Modality
where À is the regularization weight.
Each JS(.) is non-negative and vanishes iff its inputs coincide, with equality exactly when Px PA for all 1, resulting from perfect multi-scale alignment. Moreover, by virtue Rş is invariant to global scaling, translations, and orthonormal rotations, depending solely on the intrinsic, hierarchical relational structure of the embeddings. Since Rs operates in sample space, and thus is influenced by the size of N, we investigate the generalization gap between the empirical and expected values the regularization.
Lemma (Generalization bound). The generalization gap between the empirical and the expected values of Rs is bounded by
RRN R² 5
R=150
VN
where Rx is the empirical STRUCTURE regularizer and RR is its expectation over the data distribu- tion. This shows that the empirical regularizer faithrff approximates its population expectation as the number of samples increases.
Proof. Given in Appendix
Similarity-baan -bsed layer selection. In parameter-frozen alignment, the quality of alignment is closely linked to the representational similarity between the unimodal representation spaces X; and X2 Given two unimodal FMs, these spaces typically correspond different layers of the models. There- fore, selecting the appropriate layers for alignment is critical. To identify the layers with the greatest potential for effective alignment, a metric is required quantify the similarity between these represen- tation spaces. Examples include Centered Kernel Alignment (CKA) [30], which measures the overall correspondence pairwise relationships by comparing centered Gram matrices, unbiased CKA BI), which further removes the systematic overestimation inherent in standard CKA statistics when sample sizes are small, and mutual d-nearest-neighhor (kNN) 24], which identifies the closest samples in each modality's 's feature space and records the fraction of neighbors they share. Prior work has, however, solely relied on aligning models at their last layers [9) ignoring layer-based similarity. We instead challenge this approach and show in Figure 2 strong correlation between the representational
5

---

RoBERTa + VIT-L
RoBERTa + VIT-G
Llama3-88 + VIT-G
Llama-138 + VIT-G
100
100
100
Performance
80
80
80
80
60
60
60
60
40
40
40
40
20
20
20
20
0.2
0.4
0.05 0.10 0.15
0.05 0.10 0.15
Similarity
Similarity
Similarity
Similarity
Caltech101
ImageNet
Pets
Figure 2: Zero-shot performance of different model combinations when aligning different layers as a function their representational similarity measured mutual kNN (MkNN). Here, the star indicates the performance achieved when aligning the last layers of the models, and p is the average Spearman's 's rank correlation coefficient across different datasets.
similarity measured in terms of mutual kNN of the layers of pretrained unimodal encoders and downstream zero-shot performance once aligned using single linear layer with our regularization.
We thus propose the following procedure for layer selection: (i) compute the representational similarity between pairs of layers in terms of mutual kNN on small set of paired samples (in the order of 5,000 pairs), typically randomly selected from the training set, and (ii) choose the layers with the highest similarity for alignment. Throughout our work, we compute representational similarity according to mutual KNN with & chosen according to Rice's criterion" similar to what has been used in prior work 24 to measure similarities different latent spaces. We show that this selection procedure leads to consistent results with different subset sizes and repeats in Appendix H.2
5 Experiments
Experimental setup. In our experiments, we align frozen pretrained unimodal encoders using a limited number of image- -exge--exxttte pairs. To align models, of 80,000 paired samples [33]. To demonstrate versatility of the STRUCTURE regularization, we
the MS COCO train split consisting
apply different alignment strategies shown in Table I including linear mapping (Linear) [1O), non-linear mapping (MLP) (9]. and CSA (29). In addition, to evaluate the benefits of selecting layers based on the similarity of representations rather than the last layer, trained the models either the final layers ("Last") or the similarity-base with different combinations of models. By default, we consider RoBERTa (34] and DINOv2 ViT Giant 3 as our unimodal encoders.
selected layers ("Similar") Experiments are performed
We evaluate model performance zero-shot classification and cross-modal retrieval tasks. For zero-shot classification, we consider 22 datasets from the CLIP benchmark (7]. For cross-modal retrieval, we consider the Flickr30 and MS COCO test splits to evaluate both text-to-image and image-to-text performance. Detailed dataset descriptions and training configurations are described in Appendices and respectively.
Performance comparison. We conduct a comprehensive evaluation of zero-shot classification and cross-modal retrieval performance across diverse datasets and alignment techniques. Table shows that adding STRUCTURE regularization helps improve generalization performance across all three alignment techniques. On average, STRUCTURE yields substantial relative gains in both zero-shot classification and cross-modal retrieval. Specifically, for zero-shot classification,, the average relative improvement from using STRUCTURE (compared the baseline using the most similar layers without regularization) is 74.0% for MIP, 48.4% for Linear, and 26.8% for CSA. In retrieval tasks, the corresponding relative improvements are 137.0% 122.8%. and 15.9%. respectively, These results highlight that MLP and Linear alignment strategies benefit the most from STRUCTURE, while CSA shows more moderate but still substantial gains, indicating the broad applicability of our STRUCTURE across different alignment methods.
2 $ N1. which comes from the literature of choosing the optimal number of histogram bins 32

---

Table Performance comparison zero-shot classification and cross- moosll retrieval tasks across eight datasets and three alignment methods. Alignment is performed for a RoBERTa and a ViT-Giant. Underline shows the best performance in the respective group (i.e., alignment method), bold indicates the overall best performance for the respective dataset, and italic indicates our framework.
Zero-shot Classification (Accuracy)
Retrieval (R@ I)
Method
CIARRIO
Caltech101
Food101
CIFAR100
ImageNet
Pets
Flick330
Flickr30 121
Linear Last 10
75.6
85.5
37.9
14.8
34,0
9.9
7.0
32.5
22.1
Linear Similar
79.7
89.0
39.5
33.1
10.5
4.9
Linear Similar Rs
92.6
96.1
56.0
30.6
51.3
24.7
13.2
65.8
53.7
MLP+ Last
76.6
79.2
38.2
15.6
35.3
10.6
5.3
31.6
20.3
MLP Similar
84.0
81.5
38.8
17.1
34.5
11.4
6.1
36.4
25.0
MLP Similar + Rs
92.7
96.3
56.0
30.5
52.1
25.1
13.2
53.8
CSA
20
77.9
78.5
31.4
29.3
47.4
23.2
14.4
38.3
CSA Similar
80.0
80.8
33.6
28.0
47.4
23.3
14.9
48.6
39.0
CSA Similar Rs
91.7
97.2
61.5
28.6
56.4
26.8
17.9
56.1
43.1
In addition to regularization, selecting layers based on representational similarity instead of defaulting to the final layer also leads to consistent improvements across all methods. On average, this strategy yields relative gains classification accuracy of 2.5% for Linear, 4.8% for MLP, and 2.0% for CSA. For retrieval tasks, the relative improvements MLP, and 2.7% for CSA. This strategy alone provides a significant boost in performance and, when combined with STRUCTURE, results in the best overall generalization. The complete results across all 24 datasets are shown in Appendix H.8). showing consistent trends on the extended benchmark: average improvements from bascline to our approach are 65.0% for Linear in classification and 122.7% in retrieval, 61.5% and 136.8% for MLP, and 28.3% and 15.9% for CSA, respectively. Interestingly, on the CIFAR10 dataset, our alignment strategy even outperforms CLIP by around 2% while using only 0.02% of the data used to train CLIP.
even more pronounced. 8.6% for Linear, 18.3% for
Zero-Shot Classification (Accuracy)
Retrieval (R@1)
STL10
CIFAR10
ImageNet
Flickr30 T21
20
75
75
40
10
20
50
10.9
17.3
35.0
22
1k
5k
10k
50k
80k
1k
5k
10k
50k
80k
1k
5k
10k
50k
80k
1k
5k
10k
50k
80k
Food101
CIFAR100
Pets
Flickr30 12T
30
15
40
50
20
10
10
20
25
1k
5k
10k
50k
80k
1k
5k
10k
50k
80k
1k
5k
10k 50k
80k
1k
5k
10k
50k
80k
Sample Size
Sample Size
Sample Size
Sample Size
Last
Similar Rs
Figure 3: Comparison of zero-shot and retrieval performance for linear alignment when scaling down the training data, repeated five times for each sample size. Here, U quantifies the proposed method's label efficiency by computing utility compared to using the last layer.
Scaling down the training data. To measure the effect of the size of the training dataset on the performance, we subsampled the COCO training set and evaluated the effectiveness of the proposed method across varying dataset sizes for both zero-shot classification and retrieval tasks. We observe that across different tasks and sample sizes, STRUCTURE regularization combined with layer selection significantly improves performance, even in extreme resource-constrained settings of only 1,000 samples (see Figure Similarly, we find that the layer selection strategy continues to provide improvements with less data (Appendix H. These results demonstrate that combining
7

---

latent-genn geometry preservation with strategic layer selection is highly performant under severe data scarcity, making multimodal alignment practical even in the most constrained low-data regimes.
To quantify label efficiency, we measure the utility [35] U(N) (LV/N) for different-sized sub- sets and report its average value. Here, N is the number of labeled examples used by the regularized model, and N is the minimal number of labeled examples required when naively relying on the last layer for alignment without any regularization to match that same accuracy. This metric tracks the reduction in samples needed to achieve equivalent performance. For example, for zero-shot classifica- tion on the CIFARTO0 dataset and text-to-image retrieval performance on the Flickr30 dataset, the outlined method yields an average utility of 23.1x x and 22.4x, respectively, indicating that roughly 23x fewer samples are needed achieve the same performance as the baseline. These substantial label savings demonstrate that our approach significantly reduces annotation requirements while preserving high performance, thereby enhancing practicality for real-world multimodal applications.
Scaling up the training data. We next explored the effect of increasing the size of the multimodal dataset used for alignment. We gradually incorpo- rate increasingly larger subsets of the LAION-15M dataset (36) into the COCO dataset of 80K samples. Following the
CIFAR100
ImageNet
Zero Shot-Shot Accuracy
40
60
30
50
20
40
10
approach of Gadre et al. [37, we filter out low-quality LAION samples by computing CLIP scores (38] for each image-text pair and discarding those with a score below 0.15. From the remaining 847K samples, we randomly draw subsets and incrementally add them to the 80K COCO training set in steps of 80K. Figure presents the results of incrementally adding different numbers of paired samples for CIFAR100 and ImageNet zero-shot classification, and we compare performance with and without applying STRUCTURE regularization on layers selected based representational similarity. The steepest performance gain occurs with the initial addition of 80K samples, while further increases yield smaller but steady improvements. Notably, the impact of STRUCTURE regularization diminishes as more data becomes available, highlighting its particular effectiveness in low-data regimes, where preserving pretrained structure is most beneficial.
coco
80k
160k
240k
320k
4n0k
4adk
sEor
80k
189k
2401
320k
400k
4auk
senk
Added LAION Samples
Added LAION Samples
Similar
Similar Rs
Figure 4: Zero-shot performance when randomly adding LAION samples to the COCO training set, repeated three times, when aligning the best layers and adding the regu- larization.
Train-test fest data distribution shift. Despite the advantages of our alignment approach low-data regimes, performance remains low on certain datasets. For example, accuracy on the Pets dataset reaches only around 15%. We attribute this to distribution shifts between the COCO-based training set and the target evaluation domains, including differences in image content, label granularity, and overall dataset cover-
In-Domain
100
80
Accuracy
60
CIFAR100
40
age and scale. To understand the impact of this shift, we
Flowers
20
Food101
augmented the training set (SOK COCO pairs) with a small number of k in-domain examples per class drawn from dif- ferent image classification benchmarks. These l samples per class are added directly to the 80,000-sample COCO training set (c.g.. 80, 000 10 x total samples for CIFARIO). We find that mixing a small number of samples from the target domain into the training set can effectively close distribu- tion gaps and achieve strong performance. Figure 5 plots zero-shot accuracy as a function of number of samples per class l across five datasets, including Flowers, F0od101, CI- 100, Pets, and ImageNet. Remarkably, we can achieve or even outperform CLIP trained on hundreds of millions of multimodal samples by adding a few in-distribution labeled samples. Specifically, with just three samples per class, accu-
ImageNet
0
5
10
15
20
Samples per Class
Figure 5: Zero-shot classification per- formance as in-domain samples are added the training set. Performance is evaluated on multiple fine tie-grained in-domain tasks, where the added and evaluation samples come from the same dataset. Here, the star indicates the CLIP performance.
8

---

STL10
CIFAR10
FOOD101
CIFAR100
100
RoBERTa
75.6
79.7
92.6
85.5
89.0
96.3
14.8
14.6
30.6
34.0
33.1
51.3
80
Accuracy
Uamma-88-
68.9
85.3
92.8
74.3
80.8
92.5
22.7
26.5
41.7
27.0
29.4
47.8
40
70.8
79.2
93.8
69.5
69.9
87.2
20.6
18.1
23.1
48.2
20
Last
Smilar
Similar Rs
Last
Smilar
Simiar Rs
Last
Stnilar
Last
Similar
Siniler Ra
Similor
Performance of different language models combined with a DINOv2 VIT-G when choosing the last or most similar layer for alignment, and the influence of adding STRUCTURE regularization when training single linear alignment layer.
Figure 7:
racy on the Flowers dataset rises from around 24% to over 95%, outperforming CLIP's 93% accuracy. Similarly, on the CIFARION 100 dataset, using only four samples per class yields a 3% improvement over CLIP. On the ImageNet dataset, our alignment strategy matches CLIP's performance with only 17 samples per class. On both Pets and Flowers datasets, we achieve comparable results CLIP using approximately 20 samples per class. These results suggest that in a limited data regime, labeling just few iomannin samples can yield comparable on millions of paired samples.
superior performance large models trained
Different pairs of models. We explore the performance across different language models, each combined with a DINOv2 ViT-G encoder. Figure 7 shows that our regularization term and layer selection strategy consistently improve results across all combinations of models. On average, our method yields improvements of 15.2% for RoBERTa, 20.5% for Llama3-8B, and 19.8% for Llama- 13B. Notably, the RoBERTa-based combination achieves the highest overall performance, supporting previous findings
[9] that RoBERTa aligns particularly well with vision FMs.
Neighborhood preservation. To empirically verify that the STRUCTURE regularization pre- serves pretrained neighborhood, we monitored Trustworthiness and Continuity (39 on fixed random subset of 5,000 embeddings from the training and the held-out validation set at the end of each epoch with and without the STRUC- TURE regularization. Trustworthiness measures the fraction of k-nearest neighbors in the aligned space that were already neighbors in the original pretrained embedding, while Continuity mea- sures the fraction original neighbors that re- main in the aligned space. As shown in Figure over 2,000 epochs, both the training Trustwor- thiness and Continuity steadily decline under the standard alignment objective, while the gap between training and validation values steadily widens. In contrast, with STRUCTURE regular- ization, Trustworthiness and Continuity curves on training and validation remain between 0.99 and 1.00 (gap < 0.0 0.002) with no drift. This demonstrates that our regularizer enforces consistent, geometry-respectingg alignments throughout optimization, avoiding the over-warping seen in unregu- larized alignment.
Trustworthiness
1.00
1.00
Continuity
0.98
0.95
0.98
0.93
0.96
1000
2000
1000
2000
Epoch
Epoch
Similar (Train)
Similar R; (Train)
Similar (Val)
Similar + R; (Val)
Figure 6: Evolution of Trustworthiness and Continuity over 2, 000 training epochs on a fixed random subset of 5, 000 embeddings from the training (solid lines) and validation set (dashed lines) when aligning the most similar layers without and with regularization. Trustworthiness penalizes introducing new neighbors not present in the original pretrained space, while continuity penalizes losing true (pretrained) neighbors.
Further experiments. Additional results are provided in the Appendix H including experiments with text-audio alignment and application the biological domain (see H.4) more ablations such
the robustness to the parameter X, number of regularization levels L. layer selection metric,

---

normalization schemes, and distance functions (see H.6 as well as comparisons to unsupervised alignment methods (see H.7p.
6
Conclusion
We present a simple, yet effective, framework for parameter-frozen modality alignment in low-data regimes, leveraging two key components: (i) novel STRUCTURE regularizer that preserves the multi-scale neighborhood geometry of each modality's ' pretrained latent space, and (ii) an automatic layer-selection procedure that identifies and aligns the pair of intermediate layers with the highest representation similarity. Both components integrate seamlessly with existing alignment pipelines, be they linear projections. MLPs, or advanced matrix- deatridddccoposttioon methods, offering a plug-and-play solution for any modality alignment method. We evaluate our strategy by incorporating it into three existing modality alignment methods and observe consistent performance improvements across 24 benchmark datasets covering zero-shot classification and retrieval tasks, with average relative improvements 51.6% classification and 91.8% in retrieval tasks. This stems from a faithful preservation of pretrained geometry throughout training, enforced by the proposed regularization. We further show that injecting only a handful of in in-domiin examples per class achieves comparable performance to multimodal models trained on hundreds of millions of samples.
Limitations
While our method performs competitively in the low-data regime, there remains a performance gap on more challenging tasks compared to models like CLIP that are trained on hundreds of millions of paired samples. That said, we show that this gap can be significantly reduced by incorporating only a few in-domain samples in Figure 5 which highlights the potential of few-shot adaptation. Additionally, this prompted us to consider which data is most effective for aligning the spaces of different modalities. A promising future direction is to systematically investigate how different types of data influence cross-modal alignment quality in a low-data regime.
Currently, we have only investigated aligning two modalities. However, our framework can be straightforwardly extended to three or more modalities. Since the STRUCTURE regularization computed independently for each modality to preserve its own pretrained structure, this extension only requires adding an additional regularization term to the loss function for each new modality. For instance, aligning three modalities (X1, X2, Xa) would involve the following objective:
C = LA + A (L) (X1, (x))) + RS (X2, (x())) + RD (X3: (x))))
Since aligning more than two modalities is still a very active research topic, we leave these investiga- tions to future efforts.
Acknowledgements
We thank Artyom Gadetsky, Ramon Vinas Torné, Myeongho Jeon, Luca Viano, and Simone Li- onetti for their valuable suggestions, which helped improve the clarity of the manuscript. We grate- fully acknowledge the support of the Swiss National Science Foundation (SNSF) starting grant SNSF grant EC001O_231920 and the CIFAR MHU Catalyst
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko A Attnschmi,,, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.0872 2023.
[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302.1339 2023.
[3] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning Robust Visual Features without Supervision. Transactions Machine Learning Research, 2023.
10

---

[4] Alexander Kirillov,, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment Anything. In International Conference on Computer Vixion, 2023.
[5] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figumov, Olaf Ronneherger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with AlphaFold. Nature, 2021.
[6] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel- ZarandZa Tom Sercu. Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRriv, 2022.
[7] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, 2021.
[8] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. CLAP: Learning Audio Concepts from Natural Language Supervision. In International Conference on Acoustics, Speech and Signal Processing, 2023.
[9] Mayug Maniparambil, Raiymbek Akshulakov, Yasser Abdelaziz Dahou Djilali, Sanath Narayan, Ankit Singh, and Noel E O' O'Conno. Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment. Conference on Computer Vision and Pattern Recognition, 2024.
[10] Zaid Khan and Yun Fu. Contrastive Alignment of Vision Language Through Strameter-Efficien Transfer Learning. In International Conference on Learning Representations. 2023.
[11] Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly Mapping from Image to Text Space. In The Eleventh International Conference on Learning Representations, 2023.
[12] Mayug Maniparambil, Raiymbek Akshulakov. Yasser Abdelaziz Dahou Djilali, Mohamed EI Amine Seddik, Sanath Narayan, Karttikeya Mangalam, and Noel E O'Connor. Do Vision and Language Encoders Represent the World Similarly? In Conference on Computer Vision and Pattern Recognition, 2024.
[13] Laure Ciemnik, Lorenz Linhardt, Marco Morik, Jonas Dippel, Simon Korblithh, and Lukas Muttenthaler. Training objective drives the consistency of representational similarity across datasets. CoRR, 2024.
[14] Chao Jia, Yinfei Yang. Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun- Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference Machine Learning, 2021.
[15] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unitied iision-languag understanding and generation. In International Conference Machine Learning, 2022.
[16] Jianfeng Wang. Zhengyuan Yang. Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A Generative Image-to-text Transformer for Vision and Language. Trunsactions on Machine Learning Research, 2022.
[17] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Conference on Computer Vision and Pattern Recognition, 2022.
[18] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni. Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. PaLI: A Jointly-Scalce Multilingual Language-Image Model. In International Conference on Learning Representations, 2022.
[19] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding Multimodal Large Language Models to the World. arXiv e-prints, pages arXiv-2306. 2023.
[20] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Captiste Alayrac. Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A Family of Highly Capable Multimodal Models, arxiv e-prints, pages arXiv 2023.
[21] Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl- Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in Neural Information Processing Systems, 2017.
11

---

[22] Noël Vouitsis, Zhaoyan Liu, Satya Krishna Gorti, Valentin Villecroze, Jesse C Cresswell. Guangwei Yu, Gabriel Ganie, and Maksims Volkovs. Data-efficient multimodal fusion single gpu. In Conference on Computer Vision and Pattern Recognition, 2024.
[23] Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodola, and Francesco Locatello. Asif: Coupled data turns unimodal models to multimodal without training. Advances in Neural Information Processing Systems. 2023.
[24] Minyoung Huh, Brian Cheung, Tongzhou Wang. and Phillip Isola. Position: The Platonic Representation Hypothesis. In International Conference on Muchine Learning, 2024.
[25] Luca Moschella, Valentino Maiorca, Marco Fumera, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. In International Conference on Learning Representations, 2023.
[26] Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and Emanuele Rodolà. Latent space translation via semantic alignment. Advances in Neural Information Processing Systems, 2023.
[27] Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. International Conference on Learning Representations, 2017.
2017..
[28] Mikel Artetxe, Gorka Labaka, and Eneko Agire. A robust self-leaming method for fully unsupervised cross-lingual mappings of word embeddings. In Annual Meeting of the Association for Computational Linguistics, 2018
[29] Po-han Li, Sandcep P. Chinchali, and Ufuk Topeu. CSA: Data-efficient Mapping of Unimodal Features Multimodal Features. In International Conference on Learning Representations, 2025.
[30] Simon Kornblith, Mohammad Norouzi, Honglak Lec, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, 2019.
[31] Le Song, Alex Smola, Arthur Gretton, Justin Bedo, and Karsten Borgwardt. Feature selection via dependence maximization. Journal of Machine Learning Research, 2012.
[32] George R Terrell and David W Scott. Oversmoothed nonparametric density estimates. Journal of the American Statistical Association, 1985.
[33] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014.
[34] Yinhan Liu. Myle Ott, Naman Goyal. Jingfei Du, Mandar Joshi, Dangi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907.1 1692, 2019.
[35] Alejandro Newell and Jia Deng. How Useful Is Self- Supe-Superviee Pretraining for Visual Tasks? Conference on Computer Vision and Pattern Recognition, June 2020.
[36] Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. Advances in Neural Information Processing Systems, 2022.
[37] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang. Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna. Pang Wei W Koh, Olga Saukh, Alexander J Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. DataComp: In search of the next generation of multimodal datasets. In Advances in Neural Information Processing Systems, 2023.
[38] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A Reference- free Evaluation Metrie for Image Captioning. In Empirical Methods in Natural Language Processing, 2021.
[39] Jarkko Venna and Samuel Kaski. Neighborhood preservation in nonlinear projection methods: An experimental study. In International Conference Artificial Neural Networks, 2001.
12

---

[40] Leslie N Smith. Cyclical learning rates for training neural networks. In Winter Conference on Applications of Computer Vision, 2017.
[41] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical Report, University of Toronto, 2009.
[42] Adam Coates, Andrew Ng, and Honglak Lec. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Artificial Intelligence and Statistics, 2011.
[43] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A Large-scale Hierarchical Image Database. Computer Vision and Pattern Recognition, 2009.
[44] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: incremental bayesian approach tested on 101 object categories. In Conference on Computer Vision and Pattern Recognition Workshop, 2004.
[45] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- Food-101-minn mii-innnni discriminative components with random forests. In European Conference on Computer Vision, 2014.
[46] Maria-Elena Nilsback and Andrew Zisserman. Automated Flower Classification over a Large Number of Classes. In Sixth Indian Conference on Computer Vision, Graphics & Image Processing, 2008.
[47] Jonathan Krause, Michacl Stark, Jia Deng, and Li Fei-Fei. 3D Object Representations for Fine-grained Categorization. In International Conference on Computer Vision Workshops, 2013.
[48] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained Visual Classification of Aircraft. arXiv preprint 13Xxv:3306.5155 2013.
[49] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and Dogs. In Computer Vision and Pattern Recognition, 2012.
[50] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based Learning Applied to Document Recognition. In Proceedings of the IEEE, 1998.
[51] Mircea Cimpoi, Subhramsu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing Textures in the Wild. In Computer Vision and Pattern Recognition, 2014.
[52] Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun Database: Exploring Large Collection of Scene Categories. In International Journal of Computer Vision, 2016.
[53] Tan J Goodfellow, Dumitru Erhan, Pierre Lue Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. Challenges in Representation I Learning: A Report on Three Machine Learning Contests. In Neural Networks, 2015.
[54] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A Novel Datasct and Deep Learning Benchmark for Land Use and Land Cover Classification, In IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
[55] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote Sensing Image Scene Classification: Benchmark and State of the Art. In Proceedings of the IEEE, 2017.
[56] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. Man vs. Computer: Benchmarking Machine Learning Algorithms for Traffic Sign Recognition. In Neural Networks, 2012.
[S7] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are We Ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In Computer Vision and Pattern Recognition, 2012.
[58] Bastiaan Veeling, Jasper I Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation Equivariant CNNs for Digital Pathology. In Medical Image Computing and Computer Assisted Intervention, 2018.
[59] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A Dataset of 101 Human Actions Classes from Videos in the Wild. arXiv preprint 1212,002 2012.
[60] Joao Carreira, Erie Noland, Chloe Hillier, and Andrew Zisserman. A Short Note on the Kinetics-700 Human Action Dataset. arXiv preprint arXiv: 19ee: 069877 2019.
[61] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region- to to-phrase correspondences for richer image-to te-tetence models. In International Conference Computer Vision, 2015.
13

---

[62] Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu. Daniel Tompkins, Zhuo Chen, Wanxiang Che. Xi- angzhan Yu, and Furu Wei. BEATs: Audio Pre-Training with Acoustic Tokenizers. In International Conference on Machine Learning. 2023.
[63] Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In International Conference on Acoustics, Speech and Signal Processing, 2020.
[64] Yanay Rosen, Yusuf Roohani, Ayush Agarwal, Leon Samotorčan, Tabula Sapiens Consortium, Stephen R Quake, and Jure Leskovec. Universal cell embeddings: A foundation model for cell biology. bioRxiv, 2023.
[65] Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in Information Processing Systems, 2022.
contrastive representation learning. Advances in Neural
NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Yes, all claims are backed by careful evaluation of related work (see Section 2 and all claims regarding methods' performance backed up by empirical results (see Section 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of work performed by the authors?
Answer: [Yes]
Justification: Yes, the paper discusses limitations in Section of the main paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with datasct size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
14

---

While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Yes, for all theoretical results the paper provides full of assumptions and complete proof (see Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Yes, experimental details are described in the experimental setup (see Section 5 and all hyperparameters given in Appendix D Furthermore, Appendix B provides a PyTorch implementa- tion for the proposed STRUCTURE regularization.
Justification:
Guidelincs:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
the contribution is dataset and/or model, authors should describe steps taken make their results reproducible or verifiable.
Depending on contribution, reproducibility can be accomplished various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation. it may be necessary to either make it possible for others replicate the model with the same dataset, or provide access the model. general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (c.g., in the case of large language model), relcasing of a model checkpoint, or other means that appropriate to the rescarch performed.
While NeurlPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. . a large language model), then there should either be a way to access this model for reproducing the results a way to reproduce the model (e.g..
with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed- source models, it may be that access to the model is limited in some way (e.g.. registered
15

---

users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, code will be made public once the paper is de-anonymized.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. httpp/ippp.cccpuullpppp cc/public/
for more details.
While we encourage the release of code and data, we understand that this might not be possible,
Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g.. for a new open-source benchmark).
"No" acceptable
The instructions should contain the exact command and environment needed
reproduce
the results. See the NeurlPS code and data submission guidelines (httpe: //tpss///ippp.p. cc cc/public/ for more details.
guides/CodeSubbes CodeSubmi otte://oooSuuaiissoooooooooooo weeiiisssoooiiii.oo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, experimental details are deseribed in the experimental setup (see Section 5 and all hyperparameters are given Appendix
Guidelines:
The answer means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error hars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, the paper reports bars whenever training not fully deterministic, such as in Figures 3
However, since our default training and evaluation setup are fully deterministic (backbones frozen, Adentity-initializee projections, fixed seed and data order), repeating the experiment yields identical results. All standard deviations are therefore zero and have been omitted, as can be seen in Figures 3 deviation.
10
10 where the last dot corresponds to repeats of the training, leading to zero standard
Guidelincs:
The answer means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the emror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
16

---

The method for calculating the enror bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e-g., Normally distributed errors). not verified.
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is
For asymmetric distributions, the authors should be careful symmetric error bars would yield results that are out of range (e.g. negative error rates).
show in tables figures
error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we specify the computational resources of the paper in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether full research project required more compute than experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes, we have carefully reviewed and adhered to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Yes, the positive societal impact of the paper is discussed the introduction and conclusion. Negative socictal impacts are not expected as the outlined methodology enables multimodal alignment even for limited data scenarios while being computationally efficient.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA No. they should explain why their work has societal impact why the paper does address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there direct path any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arisc when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms monitor- ing misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible relcase of data or models that have high risk for misuse (e.g.. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks as beyond code are not releasing any new artifacts such as data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, the authors are the original owners of assets and have specified the referenced assets in experimental details (see Section B). dataset descriptions (see Appendix E}. and model descriptions (see Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should he provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperawittcode com/ dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release any new assets beyond code, which is adequately documented in order to reproduce all results from the paper.
18

---

Guidelines:
answer means paper does not release new assets.
Researchers should communicate the details of the dataset//codc/modlll as part of their submis- sions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper descrihe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include information that would break anonymity (if applica- ble), such the institution conducting the review.
16. Declaration LLM usage
Does the paper deseribe the usage of LLMs if it is important, original, non-standard component the core methods this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The paper does not involve the development of any LLMs, nor their use beyond minor editing.
Guidelines:
The answer NA means that the method development in this research does not involve LLMSs any important, original, or non-standard components.
Please refer LLM (https: http::////aacggeeee////o//erree/ for what should or should be described.
19

---

A
Computational Resources
Our experiments used cluster of 8 NVIDIA GeForee RTX 3090 GPUs, but each individual training required only single GPU and less than 4GB VRAM and took most hours. most computationally demanding part was obtaining the embeddings from the pretrained unimodal encoders, which on a single GPU with a batch size of 16 has speed of 3 sec/iter.
B Implementation
import torch
import torch.nn. fuffutitional as F
reg_structure(,, A, L-1, tau=.05, 05, eps=le-8): X_hat F. (x, p=2, dim=1, eps=eps)
A_hat = F. (ormalize p=2, dim-1, eps=eps)
X_tilde X_hat X_hat.mean (dim=0, keepdin-True)
A_tilde A_hat A_hat.mean (dim=O, keepdim-True)
Sx (X_tilde @ X_tilde.T) / tau
(A_tilde 0 A_tilde.T) tau
Px F.sof (Sx, dim=1)
Pa F.softmax (Sa, dim=1)
r_$ 0.0
for range (1, L + 1):
Px_1, Pa_1 Px. Px.matrix_power( (1), Pa.matrix_poverll
M_1 0.5 (Px_1 Pa_1)
d_js 0.5 ((Pa]l (torch. 1og(Pal epa) torch.log(h (pa))) sum
(Px_1
eps) torch.log(M_] eps)) sum ())
r_$ += d_js
return r_$ /
Listing 1: PyTorch reference implementation of the STRUCTURE regularizer Rs (X,
C Proof of the generalization gap of STRUCTURE regularization
Unlike classical weight-space penalties such as the &, norm, which impose sparsity directly on the model parameters and are agnostic to the number of training examples available, the proposed STRUCTURE regularizer is constructed in the sample space, via pairwise similarities among the N input embeddings. Consequently, its numerical value and statistical behavior depend explicitly on N, the size of the dataset batch size, respectively. To ensure that this penalty remains well-behaved the number of samples changes, introduce its population counterpart, defined as expectation of the empirical similarity-base regularizer under true data distribution. the following section, we will show that the empirical estimator is unbiased and concentrates around its population expectation a rate of O(1/v O(1/v/) N). thereby providing computationally efficient yet statistically reliable approximation the ideal regularization term.
fixed number hierarchical levels L, we define the STRUCTURE regularizer
JS(P!) p())
R (L) (X,
A)
(10)
L
where Px softmax (Sx) softmax ) PA defined analogously, PC) (Px)'. p (Pa)*.
We define empirical and expected values as
Rx = Rs (X,
Rd E() [R.).
In order to prove the generalization gap of the regularization, we investigate how much each sample pair can influence the output and use this derive the bound using McDiarnid's inequality.
20

---

Lemma 2 (Per-sample sensitivity). Replacing single puir (a), by arbitrary (#), W22) while keeping the other N- pairs fixed, changes the value of (10) by most
AN 4 1log2 2
(11)
Proof. Replacing single sample (2), alters exactly row and column of each similarity matrix Sx, SA. which after row-wise softmax meams exactly two rows Px (and two of PA) can change. measure the matrix-level deviation by averaging the total variation (TV) distance of corresponding rows,
d(P, P') Σ TV (PC.),,P'(.)) TV(,,F') = Σ Pt - pil 1
changing two rows gives d(Px, Px) v & and similarly d(PA, PA) S z/. Next, we check the influence of the sample perturbation for the Jensen Jensen-Shannon Shannon divergence
IS PA) 1 DKL ( + DRu(PalIM)
where M (Px + PA). Each term bounded by log 2, more generally (V,, p') S and TV(g. ) S 8 then [S(p, g) JS(p's 4')l I 2 p') + TV e')]. Hence, at any fixed level L, the two-matrix, two-row perturbation shifts the JS divergence over 1,..., with weights /l cannot inerease this bound.
4log2 and averaging most log 2 + ( N
Lemma (Generalization bound). The generalization gap between the empirical and the expected values of Rs is bounded by
RN R² S O )7))
(12)
Proof. McDiarmid's boufded-difference inequality states that for any function f of independent variables X1, X XN.
2aa
Xe -x) X 20) s 2exp
ΣN
where o, bounds the change in f when only the 7th input is perturbed. Using a = An from
11
Σ a = NA N (Algg2 2)
N
16 log"
Hence, for every 0,
Pr Pr(1Rxx R² 26) 2exp
e2 N
(13)
8 8log?
Next, we fix confidence level 1 - 5 with < and set the right-hand side of
13
to 5. Solving for gives
log (2/8)
E 2V22og log
N
Thus, with probability at least 6.
log(2/0)
Rx - R2 $2122222 2
(14)
N
showing O convergence.
D Hyperparameters
Table lists all default hyperparameters that used throughout the paper if not otherwise specified.
21

---

Table 3: List default hyperparameters used throughout the paper.
Category
Hyperparameter
Value
Validation size
5,000
Layer selection
Metric
Mutual kNN (k=rice)
Epochs
1,000
Batch size
4,096
Learning rate scheduler
Cosine
Auto learning rate finder
[40]
Alignment training
Gradient clipping
1.0
Early stopping epochs
200
Optimizer
AdamW
Weight decay
0.0001
Temperature
0.05
Rg levels L
Alignment objective
10.0
warmup À warmup steps
Linear
1,000
Alignment layer
Output dimension
512
E
Description of evaluation datasets
We use 22 vision datasets for evaluation of zero-shot classification and retrieval datasets for evaluation on retrieval tasks, similar to those studied in Radford et al. (7]. except for four tasks because of either un- availability of the dataset or lack of diversity in the modality-specific pretrained spaces, leading to random performance. These datasets cover wide range of vision tasks, including general object classification datasets CIFARIO (41), CIFARI00 41) STL10 42), ImageNet
[43. Caltech101 [44], fine- ane-grained object classification
datasets 100 45). Flowers [46]. Cars 47 FGVC Aireraft 48). Pets [ handwritten digits classification
dataset MNIST (50; texture classification dataset DTD [51: scene classification dataset SUN397 (52); the
facial emotion recognition dataset 2FR2013
6 the satellite image classification datasets EuroSAT
64
Resise45 [55] the German Traffic Sign Recognition Benchmark (GTSKB) [56] the KITTI Distance dataset 57
the metastatic tissue classification dataset PatchCamelyon (PCam) [58 : action recognition datasets UCF101
Kinetics700 [60]; the country classification dataset Country211 7 Furthermore, we include two standard
image-text retrieval benchmarks: MS COCO (33] and Flickr30 61 For the two video datasets, UCF101 and Kineties700, we take the middle frame of each video clip as the input of the pre-trained models. We use accuracy for zero-shot classification evaluation and recall@) (R@) 1) for retricval evaluation. For zero-shot classification evaluation, follow the same setup and use the same prompts in Radford et al.
F Description of pretrained unimodal encoders
We compare performance when using three self-supervised vision encoders from the DINOv2 family for alignment. Namely, Vision Transformer Base (VTT-B), Large (VIT-L). and Giant (VTT-G). where each is pretrained massive unlabeled image соrрoга using distilled masked prediction objectives to produce rich, high-dimensional patch-level embeddings [3]. For language, we compare RoBERTa, a transformer-ban optimized on large-scale English text, which outputs contextualized token representations that are aggregated into fixed-length sentence embedding [ In addition, we leverage two sizes of the Llama3 decoder-only transformer family, namely 8B and 13B parameter version, which are pretrained web-scale text data for generative tasks (2). In all experiments, freeze the unimodal encoders and learn only lightweight projection layers atop their latent outputs align them into shared multimodal embedding space.
encoder
G Layers with the highest similarity
Table 5 lists the model combinations in this work along with the layers of each encoder that have the highest representational similarity measured in terms of mutual kNN with k chosen according to Rice's criterion.
22

---

Table 4: List benchmarks we used for both shot classification and retrieval evaluation.
Task
Dataset
Number of Classes
Train size
Test size
Food101 45
101
75,750
25,250
CIFAR 10
10
50,000
10,000
CIFAR100 [1
100
50,000
10,000
SUN397 [52]
397
19,850
19,850
Cars -7
196
8,144
8,041
FGVC Aircraft
48
100
6,667
3,333
DTD 51
47
3,760
1,880
OxfordPets
49
37
3,680
3,669
Caltech101
44
102
3,060
6,084
Flowers 46
102
2,040
6,149
MNIST 50
10
60,000
10,000
Classification
FER2013 53
7
28,709
3,589
STL10 442
10
5,000
8,000
EuroSAT [54]
10
10,000
5,000
Resisc45 [35]
45
25,200
6,300
GTSRB [56]
43
26,640
12,630
KITTI Distance
[57
5,985
1,496
Country211 7
211
42,200
21,100
PatchCamelyon
58
2
294,912
32,768
UCF101 559
101
9,537
3,783
Kinetics700 60
700
536,485
33,966
ImageNet
44
1000
1,281,167
50,000
MS COCO 33
N/A
82,783
40,504
Retrieval
Flickr30 66
N/A
29,783
5,000
Table 5: Layer combination with the highest representational similarity in terms of mutual kNN, where chosen according to Rice's criterion.
Language Model
Vision Model
Language Index
Vision Index
ViT-L
24
22
RoBERTa
ViT-G
24
38
Llama3-8B
ViT-G
25
36
Llamal3B
VIT-G
29
36
H Further results
In this section, provide detailed results of the experiments presented in the main paper. In Section H.7 compare supervised alignment method to unsupervised approach, Section [166HH66psspee details the ablation study that isolates the effects of regularization of our layer selection procedure across different validation set sizes, and Section H.8 presents detailed results of the downstream performance zero-shot classification and retrieval tasks.
layer selection, Section H.2 Jnalyzes the consistency
H.1 Layer-wise performance
In Section 4
we propose selecting layers for alignment based on their representational similarity rather than defaulting to the final layers. To provide more comprehensive view, Figure S shows detailed layer- wise analysis between the last five layers of RoBERTa encoder and DINOv2 VTT-L encoder in terms of the respective downstream performance these combinations achieve after alignment. Results show that the most suitable layers (here 24/22 for the language and vision encoder pair) yield strong performance across all datasets. Additionally, we can see that for CIFAR 10 and UCF101, there is tendency that later vision layers lead to better performance while the choice of the language layers is less influential.
23

---

st|10
cifar10
Vision model layer index
23
115
13.4
13.7
03.4
06.
95.0
Vision model layer index
23
37.8
36.6
06.4
$1.5
92.7
92.5
22
9111
$5.0
61.3
652
22
91.
66.6
05.6
90.00
90.0
87.5
-85.0
20
144
16.0
82.3
68.4
91.7
85.0
20:
92.5
75.0
64.1
19.8
81.1
82.55
-80.0
19
18.2
26.8
873
09.9
19
86.1
$0.3
66.9
79.
20
21
22
23
24
20
21
22
23
24
Language model layer index
Language model layer index
caltech101
ucf101
Vision model layer index
Vision model layer index
32.5
23
55.0
23
30.0
22 344
5655
56.8
52.5
22
24.7
11.0
10.6
32.5
27.5
55..0
21
60.6
17.6
68.4
04.1
9.2
21
15.4
162
23.0
22.5
23.1
-25.0
47.5
20
42.7
20.5
50.0
50.9
-45.0
20
21.9
25.7
23.9
-22.5
-20.0
19
-42.5
19
-17.5
20
23
24
20
23
24
Language model layer index
Language model layer index
Figure 8: Downstream performance of the last five layers for a RoBERTa encoder and DINOv2
ViT-L encoder.
H.2
Layer selection consistency
Figure evaaluates the stability of layer selection by repcating the process 100 times random validation subsets of varying sizes (N 100, 500, 1000, and 5000) for mutual kNN with k=rice. All metries illustrate high robustness against repeated selection on different subsets, justifying our choice of 5000-sample set to ensure reliable layer pairing across runs.
Vurual sNV Scudee)
Vistom mote layer b000
Wiitaon model roce..
120
103
100
27-
Saliction to
Viidaa mosel by
Seleciin 50
Selocioo
-22
-00
12
Widda model
12
24
10 24
10
24
Language
(a) N 100
(b) N 500
(c) N 1, 000
(d) N 000
Figure 9: Consistency of different mutual kNN (k=rice) when repeatedly selecting N samples and
selecting the best layer combination.
H.3 Low-data comparison layer selection
Figure 10 compares the layer selection strategy when scaling down the training data using the same setup as for Figure 3 of the main paper. Results show minor difference performance and utility when using the
similar layers for alignment. However, results show that the proposed layer selection strategy is simple addition to existing alignment methods as the method includes choosing the last layers to align if they indeed are the most similar. This can be clearly seen as the performance across tasks is positive. Thus the selection process can additionally boost performance minimal additional overhead.

---

STL10
CIFAR10
Caltech101
UCF101
80
Zero Shot Accuracy
80
30
20
60
60
20
40
V 0.6
0.4
10
0 0.9
10
-
0.0
1k
5k
10k
50k
80k
1k
5k
10k
50k
80k
1k
5k 10k
50k 80k
1k
5k
10k
50k
80k
Food101
CIFAR100
ImageNet
Kinetics700
Zero Shot
Accuracy
15
30
10
10
10
5
5
5
10
U=
1k
5k
10k
50k 80k
1k
5k
10k 50k 80k
1k
5k 10k
50k
80k
1k
5k
10k 50k 80k
SUN397
Resisc45
Pets
Flowers
20
Zero Shot Accuracy
20
7.5
2
10
5.0
1 =
0.4
0.0
V
37
1k
10k 50k 80k
1k 5k
10k 50k 80k
1k
10k 50k 80k
1k
5k
10k 50k 80k
Flickr30 T21
Flickr30 12T
COCO T21
COCO 12T
40
Retrieval
20
5
R@J
20
2
10
V 0.5
0.3
0 0.3
0.1
0
1k
5k
10k
50k 80k
1k
5k
10k
50k BOK
1k 5k
10k
50k 80k
1k
5k
10k 50k 80k
Sample Size
Sample Size
Sample Size
Sample Size
Similar
Figure 10: Zero-shot and retrieval performance for training linear alignment using the last or most similar layer. Here, U quantifies the proposed method's label efficiency by computing the utility compared using the last layer.
H.4 Application to other modalities
Audio-text. We further evaluate the proposed method for audio- nudio-language alignment. Here, pretrained BEATs-ite33 [62] for audios, and extract text embeddings using RoBERTa (34] We
unimodal encoder, train a linear alignment 3,839 audio-caption pairs from the Clotho dataset [63]
To evaluate cross modsl alignment, we perform audio-to-text and exx-to-audio retrieval on held-out of
1, 1,045 pairs from the same datasct. As shown in Table our STRUCTURE regularization significantly improves the retrieval performance compared to the baseline without regularization. This demonstrates the generalizability of our regularization method across modalities beyond vasion-language alignment.
Table 6: Retrieval performance (R@k) for audio-text alignment.
Audio-to- Texeo-to-Teet
Text-to-Audio
Method
R@1 1
R@5
R@1
R@5
Linear + Last 110
0.2
1.2
0.4
2.9
Linear + Similar
0.6
2.0
0.0
1.4
Linear + Similar + Rs
7.9
23.6
8.0
24.6
Single-cell trange-transcrittmee. In addition to the audio application, we have conducted additional experiments the biological domain using only 19,900 paired human single-cell image-transcriptan data. In
25

---

this experiment, we linearly align UCE [64] encoder for the transcriptome data and a masked autoencoder (MAE) trained on single-cell images for the image data. We split the 19,900 paired samples into 100 training samples and 19,800 lest samples to simulate the low data setting. Results below show that our regularization also helps in the biological domain, yiclding relative improvements of 43.1 1% on the retricval task and 30.3% on the cell type classification task, even if the Platonic representation hypothesis is not guaranteed in the biological domain.
Table 7: Retrieval and classification performance for single-cell image-transcriptome alignment
Method
Retrieval (R@5)
Classification
Linear + Last 10
4.2
40.3
Linear Similar Rs
6.0
52.5
H.S Modality Gap Analysis
We measured the modality gap [65] for three different alignment setups for a ViT-Giant and RoBERTa Large, trained on COCO with a linear alignment function. The modality gap measured for the two image- text datasets considered in our submission (i.e., COCO and Flickr30) to ensure that the comparison is influenced solely by the data itself and not by any other artifacts introduced by the evaluation, such as zero-shot template, as would be necessary for the classification datasets. Interestingly, the results given in Table X show that the modality gap indeed decreases when first selecting the most similar layers and then further when applying STRUCTURE regularization. This aligns with the findings from the respective paper (65] that the modality gap indeed has a relationship with downstream performance.
Table 8: Modality gap for different alignment setups on the COCO and Flickr30 test sets. The gap was measured the Euclidean distance between the mean of the image embeddings and the mean of the text embeddings after L2 normalization.
Method
COCO
Flickr30
Linear Last [10]
6.6
6.8
Linear Similar
6.0
6.3
Linear + Similar + Rs
4.5
3.8
H.6 Ablation study
Table examines the influence of different hyperparameters of the proposed method. Specifically the num- ber of hierarchical levels in the STRUCTURE regularizer, regularization strength X, layer selection strategy, normalization schemes, and distance functions. Varying the number of levels from one to five yields virtually identical accuracy across datasets, indicating that a single-level penalty suffices to preserve pretrained geometry. A strong regularization weight 10 gives the best performance (e.g. STL at 92.6%. CIFARIO at 96.3%), whereas À E (0.1,1} under-regularizes leading to worse results. Without regularization choosing the last layer yields 75.6% on STL.10 and 85.5% on CTFAR 10, but using mutual kNN layer selection boosts these to 79.7% and 89.0%, respectively. These results confirm that both strategic layer choice and sufficient regularization are critical for maximally leveraging low-data regimes. Additionally, Table 12 and Figure 7 frro the main paper confirm that best-layer alignment and STRUCTURE regularization consistently increase performance across four language- vision model pairs. For normalization schemes and distance functions. STRUCTURE demonstrates robustness to these changes, highlighting that the primary benefit stems from the idea of regularizing hierarchical relationships.
The variable N in the default setting STRUCTURE represents the size of the batch, typically set to 4, 096 random samples (here approximately 6% the entire dataset), which allows for a sufficient estimate of the true data generation distribution while maintaining computational efficiency. In Table we also conducted experiments to demonstrate how the choice of sample size affects performance. Results show that the benefit of the regularization increases with increased sample size, but also show its efficiency, since even with 128 samples, there is clear improvement. Thus a practical choice to use bigger batch sizes possible. Additionally we investigated the choice of randomly selecting the subset of samples used to compute the regularization. Results show that the regularization yields very similar results regardless of whether it uses a fixed same-sized) or a random subset of the data.
26

---

Table Ablation components for two coarse- soanse-rriieed (STL10, CIFARIO) and two fine- ane-grained tasks (F1.d101, CIFARI00), when applying STRUCTURE on different layers I, changing the contribution of Rs, choosing the optimal layers according different metrics, changing the size and the type of subset used to compute the regularization, changing normalization, and using different distance functions. All experiments are performed for linear alignment.
Ablation
STLI0 CIFARIO
Food101
CIFARI00
Regularization levels
R
92.6
96.3
30.6
51.3
92.5
96.4
30.6
51.3
R
92.6
96.3
30.6
51.1
Rs
92.5
96.1
30.6
51.0
Regularization strength
X = 0.1
90.8
95.1
30.9
50.2
À 1.0
91.4
95.9
30.8
50.8
À = 10.0
92.6
96.3
30.6
51.3
Layer selection (w/o Rg))
Last
75.6
85.5
14.8
34.0
CKA B30
87.9
74.5
11.5
23.3
Unbiased CKA
31
87.9
74.5
11.5
23.3
Mutual KNN (k=10)
24
79.4
88.1
16.0
32.7
Mutual kNN (k=rice)
79.7
89.0
14.6
33.1
Batch size
N 128
90.3
96.5
29.2
48.6
N 1, 024
92.1
96.8
30.2
50.4
N 4, 4,096
92.6
96.3
30.6
51.3
N 8, 192
92.5
96.5
31.0
52.1
Subset type random fixed
92.6
96.3
30.6
51.3
92.6
96.5
30.7
51.5
Normalization schemes
Normalize centering
92.6
96.3
30.6
51.3
Centering normalize
92.1
96.4
30.8
51.1
Normalize
92.7
96.1
30.5
51.5
Standard scaling
90.8
95.1
31.0
50.3
Distance functions
Cosine
92.6
96.3
30.6
51.3
RBF
90.8
94.9
30.9
50.3
Spearman
90.4
91.6
30.0
50.0
H.7
Comparison to unsupervised methods
Table compares our supervised alignment method (Linear Similar + Rs) with unsupervised Kernel Local CKA 12 and ASIF 23 Across three evaluation tasks (COCO, CIFAR-10 10, and CIFAR-100), our method yields significant improvement in both top-1 and top-5 accuracy while relying on only 80,000 paired samples. For instance, for COCO retrieval, Kernel Local CKA attains only 13.15 % R @ 1, whereas the proposed method reaches 32.7% (a 19.6 pp. improvement) and improves R@5 from 40.0% 56.2%. Similarly, CIFARIO our approach jumps from 0.06% to 96.8% 6.8% top-1, and on CIFAR- from 1.13% 46.9%. Compared against ASIF, results demonstrate that our method again substantially outperforms it, achieving relative improvements of 32.9% on the COCO retrieval task. 32.9% 9% on the CIFAR- 100 fin-ggraine classification task, and 5,6% on the CIFAR- 10 coarse-grainn classification task.
27

---

Table 10: Top-1 and Top-5 accuracy of unsupervised approaches compared against our framework on different datasets.
COCO
CIFAR10
CIFAR100
Method
R@1 I
R@5
Top-1
Top-5
Top-1
Top-5
Kernel Local CKA 12
13.1
40.0
0.06
3.2
1.13
4.0
ASIF
23
24.6
45.4
91.7
94.5
23.8
38.2
Linear Similar Rg
32.7
56.2
96.8
96.8
46.9
46.9
H.8 Extended downstream results
Table 11 expands on the results in Table 2 from the main paper, detailing results on all evaluation tasks, including 22 zero-shot classification and two retrieval datasets with an additional fourth alignment technique (FuseMix (22). Figure [ expands Figure with more sero-shot classification and retrieval tasks. Figure 12 illustrates performance of model combinations across different vision models, analogous Figure from the main paper,
more combinations. exxeedd on Figure from the main paper, detailing results on all evaluation tasks, including 22 zero-shot classification and twa retrieval datasets.
Table 11: Performance for multiple evaluation tasks and alignment methods. Underline shows the best performance the respective group (i.e., alignment method), and bold indicates the overall best performance for the respective dataset. This table expands table 2 from the main paper, detailing results on 22 zero-shot classification and tWO retrieval datasets.
Zenvelec Clasiliction (hcearayy
Retrival:
Caloekl01
Fooditil
KinctesT10
Resioe5
Flicke30 I3T
CLIP (peenAl
59.4
$3.5
53.0
05.0
11.5
14.3
75.0
445
67.6
57.5
55.2
58.1
71.5
21.5
22.1
Invar-Strll Strier
16.7
36.5
14.8
09.0
11.1
10.5
30.1
10.0
19.
33.4
36.5
75.1
30.
Liver Stmflar
02.6
96.11
38.
06.1
11.3
31.7
11.8
20.4
23.9
26.2
12.5
65.
531
38.2
108
20.3
1031
18.1
25.4
31.
213
4 Simtar
8441
343
171
813
34.5
11.4
23.1
100
2011
112
11.3
36.8
25.0
25.1
205
CSA Leat [3]
11.9
314
75.5
47.4
23.2
35.5
13.4
33.4
25.6
28.3
42.0
38.3
Stale
116
17.9
45.6
Tuncltis Las 22
61.4
31-2
16.4
16.
34.
100
25.1
0.9
17.7
1531
23.0
32.
21.9
PooMic
Zemolla Clasiliction (Accaragy
Retrival (8301)
Methed
Ctub
PER2013
Cars
Airoraf
Pets
Flovess
MNIST
COCO 12T
CLIP (Ppnnt))
65.1
15.2
53.5
10/95
91.7
75.2
54.2
31.0
185
144
19.7
1322
5.0
Linar- Striin
441
32.5
1.1
21.9
530
019
1.5
4.9
0.1
8-2
7.1
Lewar Simtar
13
253
23
30.9
41.8
2.5
11.2
1127
19.4
192
133
MLP Lei
18
24.1
1.2
15.9
506
09
5.5
0.5
14,5
1.4
6.0
MLP Sintar
14
25.4
13
17.9
303
1.5
6.1
9.4
741
CSA Last
14
21.1
43.9
1.5
14.4
10.2
13.6
03
80
Stndz
31.
FucMiis Lad
33
33.0
1.1
14
5.9
1027
90
TheMicc Smile
369
11.
1500
28

---

STL10
CIFAR10
Caltech101
UCF101
60
Zero Shot Accuracy
75
40
20
50
20
1k
5k
10k 50k 80k
1k 5k 10k
50k 80k
1k
5k 10k 50k 80k
1k
5k 10k 50k 80k
Food101
CIFAR100
ImageNet
Kinetics700
30
20
Zero Shot Accuracy
20
20
10
10
20
10-
V 9.6
23.1
35.8
25.4
0
1k
5k
10k
50k
80k
1k
5k
10k
50k 80k
1k
5k 10k
50k
80k
1k
5k
10k
50k 80k
SUN397
Resisc45
Pets
Flowers
40
15
Zero Shot Accuracy
30
20
20
10
2
10
23.5
32.5
0
0
1k
5k
10k
50k 80k
1k
5k
10k 50k 80k
1k
5k
10k
50k 80k
1k
5K
10k 50k 80k
Flickr30 T21
Flickr30 12T
COCO T21
COCO 12T
Retrieval
40
10
R@1
10-
20
0
U = 22.4
U= 23.5
0
= 10.4
1k
10k 50k 80k
1k
5k 10k 80k
10k 50k
1k
5k
10k 80k
Sample Size
Sample Size
Sample Size
Sample Size
Last
Similar + Rs
Figure 11: Zero-shot and retrieval performance for training linear alignment using the last or most similar layer. Here, U quantifies the proposed method's label efficiency by computing the utility compared to using the last layer. This figure expands on Figure 3 with more evaluation tasks.
STL10
CIFAR10
FOOD101
CIFAR100
79.0
79.0
90.6
79.4
79.4
93.7
13.4
13.4
27.2
30.4
30.4
41.6
80
Accuracy
60
VIT
79.1
90.2
97.3
84.8
86.4
97.0
14.2
12.8
23.1
32.8
30.8
50.3
40
75.6
79.7
92.6
85.5
89.0
96.3
14.8
14.6
30.6
34.0
33.1
51.3
20
0
Last
Sinllar
Last
Simllar
Last
Similar
Last
Similar
Similar +
Sinilar +
Similar
Sinilar
Figure 12: Performance of different vision model combinations when choosing the last most
similar layer for alignment and the influence of adding STRUCTURE regularization when training single linear alignment layer.
29

---

Table 12: Performance for different model combinations for linear alignment when aligning the last or the most similar layer and when adding Rs. This table expands on Figure from the main paper, detailing results on 22 zero-shot classification and tWO retrieval datasets.
Ritrival (8e1)
Wodd
Vakuo Made
Medbal
CIPARIO
(IRVR19)
Trugrhet DEPII
Kleriech
SUN0
FandAT
Flver/ Flaterl FII
Y.RL
51.2
30.3
12.5
16-4
315
31.
3.4
15.4
16.4
1300
36.5
26.5
REBORT:
VI5
78.
10.3
16.9
18:0
061
21.3
100
21.4
313
003
VIG
853
24.4
11.1
302
31.
61.3
17.6
11.0
267
$11
410
30.5
33.5
206
08.2
36.
86
12.7
20.3
130
13.3
124
Naadar 1gg
555
56.7
365
12.
12.
12.
22.4
262
352
25
759
90.0
16.5
151
310
191
Sendars
$1.1
10.
163
245
300
VIT
11.
15.9
71.1
161
1022
30
33.8
11.
56.
488
06.0
17.3
181
16
3.4
23.3
100
14]
sleepeotaend WA2
153
682
148
Fanrdal Boc.oaaaeeee Aerag
Tanguge Musi
Vidao Moda
Metiel
Gorb
Ris
Gueeey211
FRR3013
ICan
Car
Alnoah
Pat
Flanors
D770
COCO112
COCO 73B
30.2
13
22.4
913
VIL
22.2
1.0
103
RERURTA
28.2
V11
25.1
11.9
VIR
Stade
26.1
00
23.8
177
106
3.
11.
52
Limealt
WIG
36.7
13.4
30.
139
Stal
VIG
Lod
204
10
215
97
1.
8.0
661
16.9
103
Loo
963
28.6
90.3
11.
Sell Ry
03.8
16.6
9m
11.0
336 1.3
132
Ned
140
30


Reviews:
[{"review_id": "9unpdnUjYz", "reviewer": "Reviewer_MNry", "summary": "This paper introduces a structure-based regularizer applicable on any latent space transformation. The core idea is to keep the geometry (i.e., local neighborhoods) of the space consistent before and after the transformation. The main proposed application is in the context of building a multimodal model out of independently pre-trained and frozen unimodal foundation models. In the experiments, the proposed regularizer is applied independently on each modality-specific space when optimizing a variety of already existing alignment methods, showing that : i) it allows alignment methods to use significantly fewer paired samples, thanks to the induced structural prior; ii) it substantially increases their performance on different tasks (zero-shot classification and retrieval) but also with different backbone pairs and modalities (text-image, text-audio) ; iii) it is robust to the number of datapoints used to compute it; iv) it can handle distribution mismatches with few in-domain samples.", "questions": "## Minor comments/questions\n\n1. It took me a while to understand the difference between trustworthiness and continuity when they are introduced. A different phrasing might help. What clarified it for me was thinking of it as a binary classification task with labels \"neighbor\" and \"not neighbor\" and the ground truth being the labeling in the original space while the prediction being the new one. This made it easier to relate trustworthiness as capturing \"false positives\" (precision), while continuity to \"false negatives\" (recall).\n2. I would argue that 1000 labeled samples are \"extremely resource-constrained\" when compared to the massive scale at which multimodal models like CLIP are trained, not in general. As an example, consider visual alignment of low-resourced languages, or scientific data where labeling requires costly wet-lab experiments.\n3. Reference [9] is from CVPR 2025, not 2024.\n4. I think the “Train-test data distribution shift.” is undersold. The fact that as few as 10 samples per class enhance the performance this much should be better highlighted! At the same time, I’m wondering if adding a mix of 10 samples for each class in each dataset has a consistent improvement across all datasets, or if specific samples for the task are needed while other specific ones might have a noising effect when added.\n5. Why are the scaling up and down different experiments and not unified?\n6. Where are the most suitable layers for the alignment found? What’s the similarity distribution?I would find a heatmap comparing each layer of the first model to each other layer of the other model quite interesting and informative to how this matching is helping. If substantial, this analysis could replace one of the two scaling up/down plots if they were merged.\n7. Table 2 is already pretty comprehensive, but I was wondering if an extra column could be squeezed in to show the row-wise average and variance. It would help when comparing them.\n\n## Major Questions\n\n1. Are the classification/retrieval errors uniform enough across classes? Are there cases where almost entire classes are consistently misclassified? It would makes sense if they cannot be reliably mapped across spaces or their relative positions with respect to other classes significantly differ across modalities. It might be helpful checking macro/weighted F1 instead of the coarser accuracy, or to analyze the error variance/distribution across classes for each dataset.\n2. Is there an intuition behing this specific choice of normalization (i.e., first L2 norm, then centering)? Have you tried different ones like Standard Scaling or first centering and then normalizing? Same for the distance function used.\n3. My understanding is that STRUCTURE is a regularization term that acts independently on each space, but is applied on all the spaces to be aligned to preserve their structure in the process. However, the regularization term doesn’t rely on paired data, so I was wondering if the authors could provide tests in a setting where the **paired** training data for the alignment function is different from the **unpaired** training data for the regularizer. This would help in understanding if the structural prior that STRUCTURE maintains is generalizable to the whole original space, or if it is tied to the specific subspace described by the paired samples.\n4. Is the STRUCTURE loss always computed on all the data used to train the alignment function? Take as an example the Figure 3 setting, where the (paired) training data is increased to show its robustness. Varying the training data while computing the STRUCTURE regularization only on a fixed subset (even **unpaired, a random subset of the original space,** to combine this analysis with the one described in the previous bullet) would be a stronger signal that STRUCTURE itself needs less data to be effective.\n\n---\n\nWhile not a breakthrough, the method is elegant, well-motivated, and practically impactful. Combined with the writing clarity, it makes it a clear accept for me. Among the questions in the list, my priority to consider increasing the rating would be a better disentangling and characterization *regarding the needed amount of in-domain paired data between STRUCTURE and the alignment function*.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths (Quality, Clarity, Significance)\n\n1. The paper is well-rounded. It has a high variety of experiments and ablations that validate the proposed method.\n2. I found it extremely easy to follow. From the start and throughout it, it clearly communicates the high-level problem, the proposed approach, why it is relevant, and, at each step/paragraph, what is being done and why. The reader is guided through it. This is due to a mix of strong writing and well-structured presentation.\n3. It clearly articulates and concretely shows the advantages of imposing a structural prior when trying to bridge different modalities with a simple and effective method. I believe it has the potential to be widely adopted and/or extended in this subfield.\n\n## Minor Weakness\n\nI believe a core citation/comparison missing is [a], a method that is similar with respect to: i) the task, since it creates CLIP-like models starting from pre-trained and frozen unimodal encoders; ii) the procedure, since it relies on their **similar neighborhood structure.** More in general, I think that some room in the related works should be dedicated to works in representation alignment methods for latent spaces, especially the ones focusing on geometric priors [e.g., b, c, d, e]. I would suggest broadening the “Platonic representation hypothesis.”, reframing it to this more general topic.\n\n---\n\n**[a]** ASIF: Coupled data turns unimodal models to multimodal without training; Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodolà, Francesco Locatello; NeurIPS 2023.\n\n**[b]** Relative representations enable zero-shot latent space communication; Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, Emanuele Rodolà; ICLR 2023.\n\n**[c]** Latent space translation via semantic alignment; Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, Emanuele Rodolà; NeurIPS 2023.\n\n**[d]** Offline bilingual word vectors, orthogonal transformations and the inverted softmax; Samuel L. Smith, David H. P. Turban, Steven Hamblin, Nils Y. Hammerla; ICLR 2017.\n\n**[e]** A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings; Mikel Artetxe, Gorka Labaka, Eneko Agirre; ACL 2018.", "was_revised": true, "final_justification": "The authors have provided an excellent rebuttal (I also read through the other discussions), strenghtening an already quite strong submission. Although a clear accept, I believe it doesn't fully meet the bar for a 6. Therefore, I'm happy to confirm my original score of 5 while thanking the authors again for their work.", "rebuttal": "We thank the reviewer for the positive assessment, detailed review and the insightful questions.  We appreciate reviewer’s recognition of the clear, well-structured presentation and strong writing, the effectiveness and design of our methodology, and the variety and extensiveness of our experiments. We are encouraged by the reviewer’s belief that our method has the potential for wide adoption, and we are grateful for their willingness to reconsider increasing their score based on our response. We will include all suggestions in the final version of our manuscript.\n\n## Evaluation Setup:\n\nThroughout the response, we linearly align RoBERTa and ViT-G and report accuracy for classification and recall@1 for retrieval, unless otherwise stated.\n\n## Major questions:\n\n> 1\\. Class-wise Errors and Additional Metrics\n\nWe chose the same evaluation setup as for CLIP, including benchmarks and evaluation metrics. However, we fully agree that using a metric that takes into account the class distribution is beneficial.\n\nWe report (1) the macro F1 score and (2) the standard deviation of the per-class F1 score to analyze the variance across classes. Results in terms of macro F1 score show that the performance, using our proposed framework, brings significant performance gains.\n\n| F1 score | STL10 | CIFAR100 | Food101 | ImageNet |\n| :---- | :---: | :---: | :---: | :---: |\n| Last | 74.9 | 32.6 | 12.6 | 9.7 |\n| Similar | 79.5 | 31.4 | 12.7 | 10.1 |\n| Similar \\+ R\\_S | 92.2 | 47.3 | 24.0 | 20.8 |\n\nWhen comparing the standard deviation of the per-class F1 score, we observe that there is indeed a large variation in scores across classes. For CIFAR100, Food101, and ImageNet, the spread increases when using STRUCTURE, which is expected since the classes represented in the alignment dataset (here, COCO) will experience a performance increase, whereas the others will likely remain the same due to the lack of coverage.\n\n| Stdev $\\\\downarrow$ | STL10  | CIFAR100  | Food101  | ImageNet |\n| :---- | :---: | :---: | :---: | :---: |\n| Last | 20.0 | 29.8 | 20.6 | 17.3 |\n| Similar | 17.9 | 29.6 | 20.5 | 17.9 |\n| Similar \\+ R\\_S | 9.4 | 33.2 | 27.8 | 25.5 |\n\nAs the reviewer suggested, we further looked for classes that are entirely missclassified. Without the proposed framework, 520 out of the 1000 ImageNet classes were completely misclassified. When using the proposed framework, 168 classes that were previously entirely unrecognized now achieve non‑zero scores (with an average per-class F1 score of 18.6), while 35 new classes that have previously been partially correctly classified (with an average per-class F1 score of 3.9) are now misclassified.\n\n> 2\\. Additional Ablations\n\nThe intuition behind these specific choices is that we rely on the same distance function used within the standard contrastive objective, and by normalization, we remove any constant offset in the embedding space, which could otherwise dominate raw inner products or cosine similarities. Centering ensures that the subsequent similarity graphs reflect relative geometry.\n\nWe further experimented with different normalization schemes and distance functions. Overall, our method demonstrates robustness to these changes, highlighting that the primary benefit stems from the idea of regularizing hierarchical relationships. \n\n|  | STL10 | CIFAR10 | Caltech101 | Food101 | CIFAR100 | ImageNet | Pets | Flickr I2T | Flickr T2I |\n| :---- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Default setting | 92.6 | 96.3 | 56.0 | 30.6 | 51.3 | 24.7 | 13.2 | 65.8 | 53.7 |\n| Centering First | 92.1 | 96.4 | 55.8 | 30.8 | 51.1 | 24.7 | 13.7 | 65.9 | 53.4 |\n| No Centering | 92.7 | 96.1 | 55.9 | 30.5 | 51.5 | 24.7 | 13.4 | 66.1 | 53.7 |\n| Standard Scaling | 90.8 | 95.1 | 55.7 | 31.0 | 50.3 | 23.9 | 14.6 | 65.1 | 53.1 |\n| RBF Sim. | 90.8 | 94.9 | 55.7 | 30.9 | 50.3 | 23.9 | 14.3 | 65.0 | 53.0 |\n| Spearman Sim. | 90.4 | 91.6 | 55.2 | 30.0 | 50.0 | 23.3 | 13.9 | 63.6 | 51.9 |\n\n> 3\\. Paired vs. Unpaired Samples for STRUCTURE\n\nWe evaluated the influence of using paired or unpaired samples for computing STRUCTURE. Specifically, for all experiments, we keep the alignment dataset the same but we vary the samples on which we compute the regularization. In the default setting (called *paired*) we compute the regularization on 5k samples that are also used to train the alignment function. In the second experiment, suggested by the reviewer, we compute the regularization on 10k samples that are disjoint from the alignment dataset where 50% of samples are used to compute regularization for the vision model, and other 50% for the language model (called *unpaired*). Results show that computing structure on unpaired samples yields comparable benefits as using paired samples. This shows that the structural prior that STRUCTURE maintains is generalizable to the whole original space.\n\n|  | CIFAR10 | CIFAR100 | Caltech101 | Food101 | ImageNet |\n| :---- | :---: | :---: | :---: | :---: | :---: |\n| paired | 96.3 | 51.3 | 56.0 | 30.6 | 24.7 |\n| unpaired | 92.9 | 51.1 | 57.1 | 31.0 | 24.8 |\n\n> 4\\. Fixed vs. Random Subset for STRUCTURE\n\nBy default, for efficiency, STRUCTURE is computed on the current batch of samples, rather than on the entire dataset. We evaluated the influence of using a fixed same-size subset for computing STRUCTURE. Specifically, we compare the use of a fixed random sample from the dataset (fixed) with the standard approach of using the batches to compute the regularization term (random). Results show that the regularization yields very similar results regardless of whether it uses a fixed or a random subset of the data.\n\n|  | CIFAR10 | CIFAR100 | Caltech101 | Food101 | ImageNet |\n| :---- | :---: | :---: | :---: | :---: | :---: |\n| random | 96.3 | 51.3 | 56.0 | 30.6 | 24.7 |\n| fixed | 96.5 | 51.5 | 56.0 | 30.7 | 24.7 |\n\n## Minor Weakness:\n\n> 1\\. Additional Baseline: ASIF\n\nUnlike our method, ASIF \\[1\\] does multimodal matching *without learning*. We now include ASIF as a baseline, and the results demonstrate that our method substantially outperforms it, achieving relative improvements of 32.9% on the COCO retrieval task, 32.9% on the CIFAR-100 fine-grained classification task, and 5.6% on the CIFAR-10 coarse-grained classification task.\n\n|  | COCO | CIFAR 100 | CIFAR 10 |\n| :---- | :---: | :---: | :---: |\n| ASIF | 24.6 | 23.8 | 91.7 |\n| Ours | 32.7 | 46.9 | 96.8 |\n\n> 2\\. Related Work\n\nWe have revised the related work section to provide a broader context for our work within the field of representation alignment \\[b, c, d, e\\]. This updated section now positions the \"Platonic representation hypothesis\" within this more general framework.\n\n## Minor comments:\n\n> 2\\. Definition of Resource Constrained\n\nWe use at least 1000 labeled samples in our experiments because, in the vision-language domain, this scale is often considered “resource-constrained” relative to the large datasets typically used. However, our method does not require 1000 labeled samples to be effective. For example, as shown in our response to Reviewer y6jJ (Q2), the results show that STRUCTURE remains beneficial with only 100 labeled samples in the biological domain.\n\n> 4\\. Additional Experiment on “Train-test data distribution shift”\n\nWe conducted additional experiment of “Train-test data distribution shift” in which we added a mix of 10 samples for each class in each dataset. Results show that this consistently improves performance across all datasets, rather than introducing any noise.\n\n|  | CIFAR100 | Pets | Flowers | Food101 | ImageNet |\n| :---- | :---: | :---: | :---: | :---: | :---: |\n| Target dataset only | 84.64 | 86.53 | 98.92 | 80.61 | 72.70 |\n| Mix all datasets | 84.04 | 86.75 | 99.54 | 80.74 | 72.61 |\n\n> 5\\. Unifying Scale Up and Scale Down Experiment\n\nWe did not unify the scaling up and down experiments because they rely on different data used to train the alignment functions. Specifically, for scaling down, we use COCO, and for scaling up, we add LAION samples to COCO to increase the sample size while maintaining the core as the same as in the other experiments. Furthermore, we wanted to keep them separate as they investigate different behaviours of the proposed method. Specifically, the scaling-down investigation examines how little data is needed to improve the standard approach using standard contrastive methods. In contrast, the scaling-up investigation explores where the influence of regularization diminishes when sufficient samples are available.\n\n> 6\\. Similarity Distribution of Mutual kNN\n\nIn general, for a given model pair, the most suitable layers are the layers with the highest mutual kNN score. However, we did not find a rule where these most suitable layers are found across different model pairs. Since the rebuttal policy forbids us from uploading images, we have added a table below that shows a subset of results. In particular, we aligned each of the last five layers from the vision and language encoder (RoBERTa and ViT-L) and evaluated their respective performance on the STL dataset. Results show that the most suitable layers are 24/22, and they perform significantly better than the others. Additionally, we report the alignment score in terms of mutual kNN in brackets for each combination as the similarity distribution. We will add comprehensive results across each layer as a heatmap in the revised manuscript. \n\n| Language Index vs. Vision Index | 20 | 21 | 22 | 23 | 24 |\n| :---- | :---: | :---: | :---: | :---: | :---: |\n| 19 | 88.2 (0.19) | 87.2 (0.18) | 86.8 (0.19) | 87.3 (0.20) | 88.6 (0.22) |\n| 20 | 84.4 (0.20) | 86.0 (0.19) | 82.3 (0.21) | 88.4 (0.22) | 91.7 (0.25) |\n| 21 | 80.5 (0.22) | 80.2 (0.21) | 93.6 (0.23) | 83.6 (0.24) | 93.4 (0.28) |\n| 22 | 95.1 (0.22) | 95.9 (0.21) | 95.2 (0.22) | 93.3 (0.24) | **97.3 (0.29)** |\n| 23 | 83.5 (0.21) | 83.4 (0.21) | 83.7 (0.22) | 83.4 (0.24) | 86.1 (0.28) |\n\n> 1&3&7\\. Comments\n\nWe have updated the manuscript in response to comments 1, 3, and 7\\."}, {"review_id": "HPVjQ97fr6", "reviewer": "Reviewer_u1eZ", "summary": "The authors propose a regularization scheme for multimodal alignment on top of frozen pre-trained unimodal encoders, fit for settings where multimodal data is scarce. The authors also propose a method to select the layer of the unimodal encoders with which to perform multimodal alignment.", "questions": "- How are the k samples per class in Figure 5 incorporated into the training set? Are they oversampled or just become another element in the dataset?\n- Have the authors considered using text embedding models instead of RoBERTa or LLMs, as was done in FuseMix?\n\nAlso see weaknesses above with corresponding questions. Should my concerns and questions be adequately addressed, I will consider increasing my score.", "limitations": "The authors do not acknowledge limitations or potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The empirical results are strong and convincingly demonstrate the efficacy of the proposed regularization scheme and layer-selection strategy. The authors additionally provide good supporting analyses and ablations to further corroborate their claims.\n- The proposed regularizing loss seems novel and is well-motivated from the lens of maintaining the semantic structure of the latent space of the underlying unimodal encoders. The layer-selection strategy is also interesting and novel, and Figure 2 does a good job of motivating the need for such a strategy.\n- The paper is well written with a clear methodological description and motivation for design choices.\n\n## Weaknesses\n- This paper is very similar in spirit to FuseMix [1] from CVPR 2024 but it was never mentioned in this work: the motivation of using pre-trained and frozen unimodal encoders to combat the scarcity of multimodal data was already explored in FuseMix, even with tens of thousands of multimodal pairs. FuseMix also has a similar setup in terms of proposing a regularization scheme (via a mixup augmentation routine). I acknowledge that there are still differences between this work and FuseMix in terms of the proposed methodology, but some of the key claims and motivations of this paper are not novel so I would ask the authors to include a discussion of their work compared to FuseMix (both in their rebuttal and in the updated manuscript) and include it as a baseline with experimental results. Some of the paper's claims are overstated.\n- The authors do not discuss limitations of their proposed method.\n\n## References\n[1] Vouitsis, Noël, et al. \"Data-efficient multimodal fusion on a single gpu.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "was_revised": true, "final_justification": "The authors have addressed my concerns and remaining questions to my satisfaction; they promised to update their manuscript accordingly. I will thus recommend acceptance.", "rebuttal": "We would like to extend our sincere appreciation to the reviewer for their thorough and encouraging evaluation. We are grateful by the reviewer’s recognition of the well written manuscript and motivation of our work, the novelty of our methodology, and strong empirical results which convincingly demonstrate the efficacy of our methodology.  We appreciate the reviewer’s willingness to increase their score, provided that our response addresses their concerns and questions.\n\n> 1\\. This paper is very similar in spirit to FuseMix \\[1\\] from CVPR 2024 but it was never mentioned in this work \\[...\\]. \n\nAs the reviewer noted, *although FuseMix shared a similar motivation, its underlying methodology differs from our proposed approach*, especially in terms of where the contribution is focused. FuseMix aims to improve the training efficiency by using latent space augmentations. In contrast, our framework suggests selecting the most similar layers for alignment and introduces hierarchical preservation regularization to improve performance in data-limited settings. Thus, **the two methods are complementary to each other** and can be used jointly. \n\nIn response to the reviewer’s feedback, we conducted additional experiments: (1) comparing FuseMix to linear alignment enhanced with our methodology (layer selection \\+ STRUCTURE regularization R\\_S​); and (2) enhancing FuseMix itself by incorporating our methodology into FuseMix to show complementarity. The results show that (1) the proposed framework (Similar \\+ R\\_S) yields significantly better results than the FuseMix approach in the same setting, specifically leading to an average relative improvement of 64.2% on the considered datasets. Furthermore, the results show that (2) when integrating parts of the proposed framework, such as the layer selection and the regularization, with FuseMix, there is a 64.6% relative improvement compared to the original FuseMix approach. This further demonstrates the flexibility and wide applicability of STRUCTURE.  We will add FuseMix results in our manuscript and discuss it in the introduction and related work section. \n\n|  | STL10 | CIFAR10 | Caltech101 | Food101 | CIFAR100 | ImageNet | Pets | Flickr I2T | Flickr T2I |\n| :---- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Last | 75.6  | 85.5 | 37.9  | 14.8  | 34.0  | 9.9  | 7.0  | 32.5  | 22.1 |\n| Similar | 79.7  | 89.0  | 39.5  | 14.6  | 33.1  | 10.5 | 4.9  | 35.3  | 24.0 |\n| Similar \\+ R\\_S | 92.6 | 96.3 | 56.0 | 30.6 | 51.3 | 24.7 | 13.2 | 65.8 | 53.7 |\n| FuseMix \\+ Last | 77.2 | 85.9 | 39.4 | 16.0 | 35.6 | 11.3 | 9.0 | 35.8 | 24.5 |\n| FuseMix \\+ Similar | 81.3 | 89.5 | 40.4 | 15.5 | 35.0 | 11.5 | 6.0 | 37.7 | 26.7 |\n| FuseMix \\+ Similar \\+ R\\_S | 92.1 | 95.9 | 57.1 | 30.4 | 51.5 | 24.9 | 13.5 | 65.3 | 53.5 |\n\n> 2\\. The authors do not acknowledge limitations or potential negative societal impact of their work.\n\nWe would like to point the reviewer to Appendix A of the original submission, which includes a discussion of the limitations. We did not include negative societal impacts, as the outlined methodology enables multimodal alignment even in limited data scenarios, while being computationally efficient. Thus, we are not aware of any negative societal impacts that would directly result from this work, as noted in L823-L825. To highlight these sections, we will include a reference to them in the main paper."}, {"review_id": "abCa9aqs9l", "reviewer": "Reviewer_YaRe", "summary": "This paper presents STRUCTURE, a multimodal alignment method that can effectively align two unimodal foundation models into one shared representation space, with only limited labeled (paired) data. The key to STRUCTURE is to enforce consistency between structural relationships (from the original unimodal space to the common space). And the authors further show that selecting the similarity-based layer is better for alignment than simply selecting the last layer.", "questions": "Please see weaknesses.  I will raise the score if the concerns about hierarchical relations are resolved.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n-This paper is well-written.\n\n-The idea of preserving hierarchical relationships for alignment is technically sound and interesting.\n\n-The proposed regularization term and layer selection strategy can be seamlessly integrated into other methods to enhance the alignment.\n\n-The results are strong and comprehensive.\n\nWeaknesses:\n\n-The proposed regularization uses $l$-hop reachability to calculate the similarity relation for different hierarchical levels. It seems that introducing such hierarchical relationships is a main contribution of the regularization. However,  as shown in Table 3 and Table 7, integrating the information from multiple hierarchies seems unnecessary. And the hierarchical level $l$ is set to 1 in practice, where only the sample-level relation is considered. The authors should discuss how to effectively use the hierarchical relation. Ideally, the alignment effect would increase as $l$ increases and gradually converge. \n\n-It is also suggested to discuss $l$-hop when first introduced, which would make it easier for readers to understand.", "was_revised": true, "final_justification": "The authors have addressed my concerns in the rebuttal. Therefore, I would recommend the acceptance of the paper.", "rebuttal": "We would like to extend our sincere appreciation to the reviewer for their positive assessment and their feedback. We are pleased that the manuscript’s clarity and rigor were noted, and that the reviewer found the idea of preserving hierarchical relationships for alignment technically sound and compelling our results strong and the proposed regularization effective and modular. We thank the reviewer for being open to increasing their score if their concerns are addressed.\n\n> 1\\. The authors should discuss how to effectively use the hierarchical relation. Ideally, the alignment effect would increase as l increases and gradually converge.\n\nRegarding the contribution, we would like to emphasize that the main contribution is broader than hierarchical relationships imposed in the regularization and involves the proposition of a regularization strategy that acts as a geometric prior inferred from the pretrained encoder that can enhance different alignment strategies in low data regime. \n\nIn the original version, we investigated the effect of increasing l on coarse-grained datasets (STL10, CIFAR10, Food101, and CIFAR100). The results are presented in Table 7 of the Appendix. There, we found that for some coarse-grained datasets, such as CIFAR10 or STL, the influence of the number of hops considered is less impactful. In response to the Reviewer’s question, we have now conducted more comprehensive experiments on additional datasets for different levels in the table below. On datasets that can be regarded as more fine-grained, such as Pets or Flickr, we find that enforcing higher-order interactions has a positive impact and slightly improves the performance.\n\nIt is important to note that even if one only considers a single hierarchical level, the modal still enforces consistency between higher-order relationships, as a perfect similarity graph on the lowest level also yields perfect relationships on all higher levels. Thus, increasing the number of hops is a more direct way of enforcing these higher relationships, but the lower levels still enforce them indirectly.\n\n|  | Pets | MNIST | Food101 | UCF101 | Flickr I2T | Flickr T2I | COCO I2T | COCO T2I |\n| :---- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| $R\\_S^1$ | 15.3 | 13.6 | 30.5 | 42.3 | 65.9  | 53.8  | 18.3 | 13.4 |\n| $R\\_S^2$ | 15.4 | 13.5 | 30.4 | 42.3 | 66.0  | 53.7 | 18.4 | 13.4 |\n| $R\\_S^3$ | 15.8 | 13.6 | 30.4 | 42.2 | 66.2 | 54.0 | 18.5  | 13.5 |\n| $R\\_S^5$ | 15.9 | 13.8 | 30.6 | 42.4 | 66.7 | 54.3 | 18.7  | 13.6 |\n\n> 2\\. It is also suggested to discuss l\\-hop when first introduced, which would make it easier for readers to understand.\n\nWe will thus revise paragraphs L143-144 and include a more intuitive description of the l-hops. Specifically, we will add the following sentences after L144: “Intuitively, the similarity graph can be thought of as a Markov chain where the hops refer to the transitions between different states. Thus, the $l$ hops represent the probability of transitions reachable with $l$ hops.” Additionally, we will discuss l-hop in more detail when it is first introduced."}, {"review_id": "KWZlIfUZhP", "reviewer": "Reviewer_y6jJ", "summary": "This paper presents STRUCTURE, a novel regularization technique that preserves the neighborhood structure of the latent space in pretrained unimodal encoders. It enables more effective and efficient multimodal alignment, especially in low-data training scenarios. The technique can be added to existing alignment methods and helps maintain the geometric relationships in the input representations. The authors show that their method performs well with significantly less training data, resulting in better alignment and improved performance in downstream tasks.", "questions": "Questions:\n- The definition of hierarchical level L in Eq. (4) is not intuitive. Could the authors give more insight or examples to help understand its role?\n- Have the authors tested STRUCTURE with smaller, less powerful networks? This would help understand whether the method still works when the pretrained representations are less aligned.\n- The paper [A] (\"Mind the Gap\", NeurIPS 2022) highlights the modality gap in contrastive models like CLIP. Since STRUCTURE is used with alignment losses (often contrastive), is this gap still present in the shared embedding space $\\mathcal{A}$? Does STRUCTURE help reduce the modality gap, or does it preserve it? How does preserving structure interact with or influence this phenomenon?\n\n[A] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. NeuIPS 2022.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Requires less training data to achieve strong alignment, making it suitable for low-resource settings.\n- As a regularization method, it can be integrated into many existing alignment frameworks.\n- The paper includes a strong set of experiments across a wide range of benchmarks.\n\nWeaknesses:\n- The unimodal models used (RoBerta and DinoV2 Giant) are very deep and computationally expensive. STRUCTURE shows more benefits when aligning highly similar layers, but these benefits may decrease when using smaller or weaker models (simple Resnets or base ViT).\n- The proposed framework is only tested on two modalities. There is no discussion or evidence on how it would extend to three or more modalities. Moreover within the two modal case, just text and image modalities are tested.\n- It's not clear whether the variable N in Eq. (2) and lines 137–138 refers to the size of the whole dataset (as suggested in line 105) or just the batch size during training. If it's the batch size, the similarity matrix may only capture local (within-batch) relationships, which could limit global alignment quality.", "was_revised": true, "final_justification": "After carefully reviewing the authors' rebuttal and considering the discussion, I am satisfied that most of my concerns have been adequately addressed. \n\nIn particular:\n\n1) The additional experiments with smaller networks.\n2) The explanation of the hierarchical level \n\n\nThe authors’ responsiveness and thoroughness in addressing each point, including new experiments across modalities and datasets, demonstrate the robustness and generality of their method.\n\nOverall, I find the work technically sound, relevant, and well-executed. I am raising my score accordingly.", "rebuttal": "We would like to extend our sincere appreciation to the reviewer for their insightful comments and positive evaluation of our work. We are encouraged by the recognition that our approach requires less training data to achieve strong alignment, making it particularly well-suited to low-resource settings. We are likewise gratified that the method’s versatility as a regularization technique, enabling seamless integration into existing alignment frameworks, was noted. Finally, we appreciate the acknowledgement of our comprehensive experimental suite, which spans a wide range of benchmarks. \n\n> 1.1 & 1.2. STRUCTURE with Smaller Networks\n\nWe focus on large models because they are readily available and are not being updated within our framework, making them essentially an initial feature extraction. Additionally, we would like to highlight that Figure 11 of Appendix I.6. (referenced in the main paper on L225) includes experiments with a base ViT.\n\nIn response to the Reviewer’s feedback, we conducted additional experiments of two model combinations, which can be considered much smaller. Specifically, in the first table, we align a ViT-Tiny with 5.72 million parameters (even smaller than a ResNet18) and an all-MiniLM-L6-v2 with 22.7 million parameters. In the second table, we align a ViT-Tiny with 5.72 million parameters and an all-mpnet-base-v2 with 109 million parameters. Both combinations are trained using linear alignment functions. Results show that adding STRUCTURE still yields a benefit even for these smaller model combinations. \n\nWe would like to highlight two key aspects of these results. First, the alignment score, in terms of mutual kNN, for these pairs of models is significantly lower than for the other (bigger) combinations. Specifically, 0.09 for  DINO\\_v2 ViT-Tiny+all-MiniLM-L6-v2 and 0.10 for DINO\\_v2 ViT-Tiny+all-mpnet-base-v2, which are much lower than the 0.29 for DINO\\_v2 ViT-L+RoBERTa. Second, as the Platonic Representation Hypothesis suggests, the convergence of the models to the same representation increases with scale and performance. Thus, size itself has a significant influence on the relative performance of the aligned models, as more similar models facilitate their alignment, and enforcing hierarchical relationships becomes more meaningful. However, even when the models are less similar and respectively smaller, the proposed regularization yields consistent improvements.\n\nViT-Tiny+all-MiniLM-L6-v2:\n\n|  | STL10 | CIFAR10 | Caltech101 | Food101 | CIFAR100 | ImageNet | Pets | Flickr I2T | Flickr T2I |\n| :---- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Last | 86.6 | 65.9 | 40.9 | 5.1 | 17.4 | 5.3 | 4.7 | 17.2 | 14.1 |\n| Similar | 86.6 | 65.9 | 40.9 | 5.1 | 17.4 | 5.3 | 4.7 | 17.2 | 14.1 |\n| Similar \\+ R\\_S | 89.3 | 68.8 | 45.6 | 6.8 | 23.7 | 8.9 | 6.2 | 24.1 | 18.0 |\n\nViT-Tiny+all-mpnet-base-v2:\n\n|  | STL10 | CIFAR10 | Caltech101 | Food101 | CIFAR100 | ImageNet | Pets | Flickr I2T | Flickr T2I |\n| :---- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Last | 85.8 | 63.9 | 47.6 | 5.7 | 22.8 | 8.9 | 6.0 | 22.7 | 15.5 |\n| Similar | 85.6 | 65.3 | 42.9 | 5.9 | 22.8 | 8.9 | 5.2 | 22.9 | 14.8 |\n| Similar \\+ R\\_S | 85.1 | 68.0 | 52.1 | 8.8 | 28.1 | 12.2 | 6.0 | 28.7 | 22.1 |\n\n> 2\\. STRUCTURE on Three or More Modalities and Additional Two Modal Cases\n\nFirst, we would like to discuss the extension of the proposed regularization to aligning more than two modalities. As regularization is only calculated on the pretrained representations of each individual modality, thus not relying on any cross-modal data, extending this approach involves computing the regularization in each modality separately and adding an additional term to the loss function. Specifically, Eq. 8 would get a third term for the new modality: \n\n$$\n\\mathcal{L} = \\mathcal{L}_{\\text{A}} + \\lambda\\left(\n\\mathcal{R}_S^{(L)}(X_1, f_1(X_1)) + \n\\mathcal{R}_S^{(L)}(X_2, f_2(X_2)) + \n\\mathcal{R}_S^{(L)}(X_3, f_3(X_3))\n\\right) \n$$\n\nThus, regardless of the alignment objective function used, the extension of regularization is indeed straightforward and we will add a discussion in our manuscript.\n\nIn addition to text and vision modalities, we would like to point out the additional experiments on the audio and text domains in the original submission in Appendix I.3. In addition, in response to the reviewer’s comment, we have now conducted additional experiments on the *biological domain* using only 19,900 paired human single-cell image-transcriptome data. In this experiment, we linearly align a UCE \\[1\\] encoder for the transcriptome data and a masked autoencoder (MAE) trained on single-cell images for the image data. We split the 19,900 paired samples into 100 training samples and 19,800 test samples to simulate the low data setting. Results below show that our regularization also helps in the biological domain, yielding relative improvements of 43.1% on the retrieval task and 30.3% on the classification task, even if the Platonic representation hypothesis is not guaranteed in the biological domain.\n\n|  | Retrieval | Classification |\n| :---- | :---: | :---: |\n| Last | 4.2 | 40.3 |\n| Last \\+ R\\_S | 6.0 | 52.5 |\n\n\\[1\\] Universal Cell Embeddings: A Foundation Model for Cell Biology, bioRxiv 2023\n\n> 3\\. N in Eq. (2)\n\nThe variable N in the default setting of STRUCTURE itself represents the size of the batch, typically set to around 4,096 random samples (approximately 6% of the entire dataset), which allows for a sufficient estimate of the true data generation distribution while maintaining computational efficiency. In response to the reviewer’s feedback, we conducted experiments to demonstrate how the choice of sample size affects performance. Results are given when aligning RoBERTa with a ViT-G using linear alignment. Results show that the benefit of the regularization increases with increased sample size, but also show that it’s efficiency, since with even 128 samples, there is a clear improvement.\n\n| N | STL10 | CIFAR10 | Caltech101 | Food101 | CIFAR100 | ImageNet | Pets | Flickr I2T |\n| :---- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| 128 | 90.3 | 96.5 | 55.2 | 29.2 | 48.6 | 22.7 | 12.9 | 60.5 |\n| 1,024 | 92.1 | 96.8 | 54.9 | 30.2 | 50.4 | 24.0 | 13.5 | 64.8 |\n| 4,096 | 92.6 | 96.3 | 56.0 | 30.6 | 51.3 | 24.7 | 13.2 | 65.8 |\n| 8,192 | 92.5 | 96.5 | 56.6 | 31.0 | 52.1  | 25.0 | 13.6 | 66.8 |\n\n> 4\\. Understanding of Hierarchical Level L\n\nThe hierarchical level $L$ in Equation 4 can be understood through the lens of a random walk on the data manifold. Specifically, the row-normalized similarity matrix, $P\\_X$, can be interpreted as the transition matrix for a random walk on a graph where data points are nodes and similarity defines edge weights. In this context, the value $P\\_X(i, j)$ represents the probability of transitioning from sample $i$ to sample $j$ in a single step. The hierarchical levels then correspond to the length of this random walk: \n\n(1) $l \\= 1$ (1-hop): represents the probability of a direct transition between two samples. Preserving this level enforces consistency in the immediate local neighborhood structure. It ensures that if sample $j$ is a close neighbor of $i$ in the original space, it remains so in the aligned space.\n\n(2) $l \\> 1$ (l-hop): the matrix power $(P\\_X)^l$ computes the probability of reaching one sample from another in exactly $l$ steps. This captures progressively larger structural patterns. For $l=2$, it considers paths through one intermediary. This enforces consistency at the level of local clusters or \"neighborhoods of neighborhoods.\" Two samples that are not direct neighbors but belong to the same tight cluster will have high 2-hop probabilities. As $l$ increases, we capture more global, long-range connectivity across the data manifold.\n\nBy enforcing consistency between the multi-step transition probabilities of the original space $(P\\_X)^l$ and the aligned space $(P\\_A)^l$, the STRUCTURE regularizer ensures the alignment function preserves the intrinsic geometry of the data manifold at multiple scales. This prevents the \"over-warping\" that can occur in unregularized alignment, where local neighborhoods might be preserved but the global arrangement of clusters is distorted.\n\nWhile our experiments show that setting $L=1$ is often sufficient, this is because perfect 1-hop consistency mathematically implies perfect $l$-hop consistency for all $l$. Therefore, strongly regularizing the most local structure provides a powerful, indirect constraint on the entire manifold geometry. Explicitly including higher levels ($L\\>1$) simply makes this constraint more direct, which can provide additional benefits on fine-grained tasks where subtle, multi-hop relationships are more critical to preserve.\n\nWe will expand the discussion about it in the revised manuscript.\n\n> 5\\. About Modality Gap\n\nWe measured the modality gap for three different alignment setups for a ViT-Giant and RoBERTa Large, trained on COCO with a linear alignment function. The modality gap was measured for the two image-text datasets considered in our submission (i.e., COCO and Flickr) to ensure that the comparison is influenced solely by the data itself and not by any other artifacts introduced by the evaluation, such as a zero-shot template, as would be necessary for the classification datasets. Interestingly, the results show that the modality gap indeed decreases when first selecting the most similar layers and then further when applying the proposed regularization. This aligns with the findings from the respective paper, that the modality gap indeed has a relationship with downstream performance. We will include these findings in the revised manuscript. \n\n|  | COCO | Flickr |\n| :---- | :---: | :---: |\n| Last | 6.6 | 6.8 |\n| Similar | 6.0 | 6.3 |\n| Similar \\+ R\\_S | 4.5 | 3.8 |"}]

Author rebuttal:

Meta-review: To address the multimodal alignment problem between unimodal foundational models, the authors proposed STRUCTURE, which enforces consistency between structural relationships (i.e., neighborhood structure) from the unimodal representations to the joint representation -- thus requiring *far* fewer aligned data pairs. Operationally, STRUCTURE can be applied to many existing alignment methods by adding a unimodal neighborhood preserving regularizer to the joint embeddings alignment procedure. Secondly, the authors empirically demonstrate (and theoretically argue) that selecting layers with a similarity-based criteria is better than jut selecting the last layer. Experiments are conducted for seven zero-shot classification benchmarks and two cross-modal retrieval benchmarks, demonstrating notably improved performance relative to the underlying approaches without STRUCTURE regularization (with additional baselines provided during rebuttal). Finally, secondary experiments are performed to quantify training data size requirements, distribution shift, validation of neighborhood preservation, sensitivity analyses, and a text-audio alignment task.

Strengths of this work identified by reviewers include:
- The paper is well-motivated, well-structured, and well-written -- I concur with the reviewers in that I was able to understand just about everything in one reading.
- Requiring less aligned training data to project unimodal representations into a joint embedding space is a potentially impactful finding and the proposed method does this in a simple way that applies to most any deep learning method.
- The set of experiments is over a large number of task/datasets with sufficient diversity and multi-modal alignment methods, showing consistently good performance. The reviewers all thought that the (primary and secondary) empirical results were strong.

Weaknesses of this work identified by reviewers include:
- There were several 'nit' clarification questions raised by reviewers (e.g., macro vs. micro averaging, additional ablations, paired vs. unpaired examples, more than two modalities, additional baselines, addition of another domain (biology), modality gap -- some requiring additional experiments.  However, they were well-handled during rebuttal such that the reviewers universally raised/maintained their scores. I am not sure the additional experiments were needed for the paper to be sufficiently convincing, but they definitely help. Obviously, these additional findings should be integrated into the next manuscript revision.

There was consensus from the reviewers that this is a well-motivated approach to an important (and widely studied) problem for which strong evidence of positive performance improvements is provided. Additionally, the paper is well-written and the rebuttal provided further convincing results to support the STRUCTURE approach to multi-modal alignment. I concur with the reviewer consensus and believe that many researchers in this space will adopt this method as it is widely applicable, conceptually appealing, and shown to perform well.
Decision: Accept (poster)