CSA: Data-efficient Mapping of Unimodal Features to Multimodal Features
Authors: Po-han Li
Sandeep P. Chinchali
ufuk topcu
Venue: ICLR 2025 Poster (2025)
Keywords: multimodal
representation learning
relative representations
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Canonical similarity analysis (CSA) achieves CLIP-level performance in multimodal tasks using much less data. It maps unimodal features into a multimodal space without extensive GPU training and supports all modalities.

Abstract: Multimodal encoders like CLIP excel in tasks such as zero-shot image classification and cross-modal retrieval. However, they require excessive training data.
We propose canonical similarity analysis (CSA), which uses two unimodal encoders to replicate multimodal encoders using limited data.
CSA maps unimodal features into a multimodal space, using a new similarity score to retain only the multimodal information.
CSA only involves the inference of unimodal encoders and a cubic-complexity matrix decomposition, eliminating the need for extensive GPU-based model training.
Experiments show that CSA outperforms CLIP while requiring $50$,$000\times$ fewer multimodal data pairs to bridge the modalities given pre-trained unimodal encoders on ImageNet classification and misinformative news caption detection.
CSA surpasses the state-of-the-art method to map unimodal features to multimodal features.
We also demonstrate the ability of  CSA with modalities beyond image and text, paving the way for future modality pairs with limited paired multimodal data but abundant unpaired unimodal data, such as LiDAR and text.

Published as a conference paper at ICLR 2025
CSA: DATA-EFFICIENT MAPPING OF UNIMODAL FEATURES TO MULTIMODAL FEATURES
Po-han Li, Sandeep Chinchali & Ufuk Topeu The University of Texas at Austin (pohan i, sandeepc, utopeu Jutexas. edu
ABSTRACT
Multimodal encoders like CLIP excel in tasks such as zero-shot image classifica- tion and cross-modal retrieval. However, they require excessive training data. We propose canonical similarity analysis (CSA), which uses two unimodal encoders to replicate multimodal encoders using limited data. CSA maps unimodal features into multimodal space, using new similarity score to retain only the multimodal information. CSA only involves the inference of unimodal encoders and a cubic- complexity matrix decomposition, eliminating the need for extensive GPU-based model training. Experiments show that CSA outperforms CLIP while requiring 50,000 fewer multimodal data pairs bridge the modalities given pre-trained unimodal encoders on ImageNet classification and misinformative news caption detection. CSA surpasses the state-of-the-as method to map unimodal features to multimodal features. We also demonstrate the ability of CSA with modalities be- yond image and text, paving the way for future modality pairs with limited paired multimodal data but abundant unpaired unimodal data, such as LiDAR and text.
INTRODUCTION
Multimodal encoders like CLIP (Radford et al. 2021) excel in various zero-shot multimodal tasks, such as image classification and cross-modal retrieval. Despite the huge success, they demand exces- sive training data. For example, OpenAl trained the original CLIP model on 400 million image-text pairs using 592 V100 GPUs, and the training size of the new CLIP models (marrco et al. [021 has increased to 12 billion image-text pairs since then. In addition, more data do not guarantee better performance. CLIP relies on Internet data, which often of poor quality (Sharma et al.) 2018 Incorrectly captioned data may lead to specific failure modes in similar instances (Northcutt et al. 2021 Vasudevan et al. 2022). In this work, we focus on learning a multimodal encoder with limited data that is robust to noisy data.
Our conjecture is that unimodal encoders such as DINO (Oquab et al., 2023) and GTR (Ni et al. 2022] can serve as the building blocks of multimodal encoders. Unimodal encoders only need uni- modal data, which are easier obtain than paired multimodal data. They also require significantly less data than multimodal models, and advanced unimodal encoders are already well-developed. We distill the unimodal knowledge in these models to accelerate the learning of multimodal encoders. We map their unimodal features into multimodal feature space, similar to how CL.IP encodes images and text jointly to a shared feature space. In addition, this process should be data-efficient.
We propose canonical similarity analysis (CSA) to verify our conjecture. It replicates a multimodal encoder such as CLIP with two unimodal encoders, as shown Figure [ CSA maps features from unimodal encoders to multimodal space. removes redundant information from unimodal features and employs a novel similarity function to replicate CLIP similarity score for multimodal data, egg. images and captions. CSA operates by inference of unimodal encoders and solving optimization problem, essentially a matrix decomposition without training of neural networks. CSA supports all zero-shot tasks that CLIP is capable of while requiring significantly fewer training data.
Project Page: https://d311033.ithhhihh
Wehsite/

---

Published as a conference paper at ICLR 2025
Multimodal Data
Unimodal Features
Multimodal Features
Downstream Tasks
Image Encoder
Classification
Weighted Coaine Similarity
Retrieval
CSA (matrix projection)
(LLLIP similartyyy
Mistabelldd Data Detection
*Allln Costro annoummment in San Antonio, Tex, on Salurday:
Text E ncoder
Misinformation Detection
AA mon brushes the moull of sarcsphagus.
Frozen Weights
Figure I: Canonical similarity analysis (CSA) replicates the CLIP multimodal similarity with unimodal encoders. CSA uses two unimodal encoders to encode into unimodal features. Then, projects the unimodal features to a joint multimodal feature space. The weighted cosine similarity in this feature space replicates the CLIP similarity. enabling various downstream tasks, e.g., cross- retrieval. CSA is robust and data-efficient method that learns even when data misaligned.
Contributions. Our contributions are threefold: (1) We propose CSA which replicates the CLIP
model using two unimodal encoders while demanding less computation and data (Section 4 (2)
Our theoretical analysis characterizes the trade-off of obtaining informative embeddings and distin- guishing multimodal data pairs, considering various hyperparameters of CSA (Section [5.. (3) We
tested CSA tasks such image classification, modsl retrieval, and misinformative captions detection (Section 6) The experimental results show that CSA outperforms or matches CLIP while requiring 50,000 fewer paired multimodal data that used to train the multimodal mapping function given pre-trained unimodal encoders (see Table [T. CSA outperforms the state-of-the-at method that employs unimodal models with the same amount of training data 18% ImageNet classification and 23% in image-to-text retrieval.
CSA works across all modalities beyond images and text. We additionally demonstrate its capability with audio and text, paving the way for new modality pairs, such as LiDAR and text Yang et al. 2023). In the future, we envision that new modality pairs will have limited paired multimodal data but sufficient unimodal data, where CSA can efficiently map unimodal features to multimodal features.
RELATED WORKS
Unimodal Encoders. Unimodal encoders encode unimodal data to fixed-dimensional features. In computer vision, encoders, e.g., CosPlace (Berton et al.) [022], VIT (Dosovitskiy et al. 2021). and Dino (Caron et al.]
et al., al.||2023) (022)), are trained with augmented data with random cropping and blurring with contrastive loss (van den Oord al. 2019). In natural language processing, common data augmentation method is to randomly mask words or tokens, used in the universal sentence encoder (Cer et 2018), Bert (Devlin et al. 2019). and GTR (Ni al. (022)).
Multimodal Encoders. Multimodal encoders project multimodal data points into feature space shared between modalities. Thus, multimodal features enable training for downstream models (Shridhar et al.) 202T Li et al., [0244b] Goel & Narasimhan, 2024). Another common usage is directly take the inner product of features of data from different modalities, e.g., an image and a sentence, as similarity metric. This metric enables downstream tasks such as zero-shot image classification and cross-modal data retrieval. The CLIP model (Radford al.) (02T) the mile- stone of multimodal encoders, which bridges the gap between language and vision. CLIP inspires encoders for other modalities, such as audio and inertial measurement units. Such encoders include ImageBind (Girdhar et al. 2023), AudioCLIP (Gzzhov et al.) 2022) and CLAP o et al. 2023) Previous work also studied cross-modal retrieval on multi-labe images (Ranjan et al. 2015). while ours focuses
generic multimodal setting with pre- rre-trained unimodal encoders. Although ImageBind encodes 6 modalities, it lacks training data for all possible modality pairs, resulting in suboptimal performance for pairs such audio and inertial measurement units.
2

---

Published as a conference paper at ICLR 2025
Neural Networks with CCA. Previous works have shown that training losses inspired by canonical correlation analysis (CCA) (Hardoon et al. 2004) are useful in multimodal or multi nuuli-distribution tasks (Sun et al. 2019; Lyu & Ful 2020; 2023). Although taking crucial step of introducing CCA to deep learning, these works focus on training deep neural networks from scratch instead of using existing foundation encoder models, as in this work. Other works use techniques similar to CCA, such as principal component analysis (PCA), to process the output features of encoders (Li et al., 2024a Omama al. 2024). In this work, we focus on using CCA map unimodal features to multimodal feature space without any model training, which significantly reduces the size of the required training dataset.
Combining multiple unimodal Encoders. The closest work to ours is ASIF (Norelli et al., 2023 which also uses multiple unimodal encoders to bridge the multimodal gap. ASIF calculates the similarity of a pair of multimodal data based on the unimodal similarity of the given data pair and pre-selected, multimodal anchor dataset, similar to Moschella al. (2023). creates a linear kernel space by the anchor set. However, ASIF demands a huge anchor set to achieve similar performance to CLIP, while our proposed method, CSA, constantly outperforms ASIF all tasks.
PROBLEM FORMULATION
Given two datasets of size N, X {.t 222 INJ. x²= {2,, m², m² (such as vision and language), we want to find two feature extractors (encoders) E E² that map the data to the sume-dimensiona feature space. The encoders generated feature pairs E (a)) E2 (x²)) are close, and the feature pairs from different pairs of data E' (x!) Ex²3)) are far.
from two modalities
Note that the two sets of data are paired, which means that they are sophisticatedly related (such as an image and its corresponding caption), although of different modalities. Previous works trained multimodal encoder pairs to achieve this goal by minimizing the following loss:
((X), 22:
(Multimodal Contrastive Loss)
(E+())) E (z (TT)))((E(())) (=
2N Σ
2 (ot)) E
(xf)))
2N -
where the first term encourages similar features for the same pair of data, and the second term encourages the opposite. Variants of this loss commonly used in contrastive learning (van den Oord et al. 2019). such as CLIP (Radford et al. 2021).
From Unimodal to Multimodal. Training a multimodal encoder to learn multimodal feature space is time-consuming and data-inefficient. The original CLIP model learns from Internet-scale datasets (400 million images) and requires 500 A100 Nvidia GPUs. Instead of training end-to-end multimodal encoders, we leverage existing unimodal encoders to accelerate the multimodal learning process. That is, given unimodal encoders that encode data into unimodal feature spaces, we directly find two mapping functions that map each unimodal feature space to a multimodal space.
Mathematically, we use unimodal encoders E E² that capture only unimodal latent informa- tion and individually encode data w x? into features 2/, &? E RRo', TR² Because these features #1, :8 come from different encoders, their dimensions need not be the same (g' * q²), which com- plicates the discovery of the mapping functions. The dimensions of the features are significantly smaller than those of the raw data x, Hence, utilize unimodal encoders to significantly reduce the dimension of the data and the complexity of the problem. Our ultimate goal is to efficiently find two mapping functions A, B that map the feature spaces to multimodal one while preserving the contrastiveness property characterized in Equation]] I
4 CANONICAL. SIMILARITY ANALYSIS (CSA)
The problem in Section is easier to state than to solve. First, unimodal encoders have completely different architectures (e.g., transformers vs. U-net), losses, and output dimensions. Second, map- ping data from various-on diniouu-dimessionll feature spaces to a multimodal one means that we need to dis- card latent information from the higheradimensional space. The challenge lies in determining what
3

---

Published as a conference paper at ICLR 2025
information to discard to maintain the contrastiveness property. Also, do not want to preserve all information as there might be some moddlittysppeciin information that we wish not to preserve through the mapping, such as words that cannot be expressed in images punctuation.
We thus propose canonical similarity analysis (CSA), We observe that all downstream tasks use cosine similarity to obtain the similarity of multimodal data points. It inspires to correlation coefficients, the cosine similarity between centered vectors, to approximate this similarity and build the mapping functions from unimodality multimodality. Precisely, we find the pairs of bases in each unimodal feature space that maximize the correlation coefficients. We use CCA to find such bases. Then, instead of directly using an inner product, we use a weighted cosine similarity score to evaluate the similarity of the multimodal data. We discard information from low-correlated bases, as they might not be relevant to the multimodal feature space, and we weight the contributions of the bases to the similarity score based on their correlation coefficients.
4.1 CANONICAL CORRELATION ANALYSIS
Recall the previous notation of two sets of data X = [x] T2. xxJ X² = {2 22
We use unimodal encoders E1, E² to encode and normalize the data into zero-meaned latent features {+], 2 fi=1-
f- To find the bases that maximize the correlation coefficients, we use CCA to map the two feature spaces a multimodal feature space of dimension min(g', .²):
Tr A2' (B2²))
B
arg max
(Formulation of CCA)
AER B
S.L. (A2)) = (B22) =I,,
where 21 E RR''NN, 22 E R are the matrices consisting of column feature vectors I, I, E grxr is the identity matrix, and Tr is the trace function. The objective of Eqquation@can can be viewed as the sum of the correlation coefficients of the bases, and the identity constraints normalize the vectors so that covariance becomes correlation. In summary, CCA finds the bases of the two given matrices such that the projected vectors have the highest correlation coefficients P; for = l...,,
Solving CCA is similar to principal component analysis (PCA), which is essentially a singular value decomposition (SVD) of time complexity 92)) Therefore, it possesses the beneficial property of removing noise from the data, which later detailed in the theoretical analysis. [Wennnn (2003) gives the analytical solution:
A B" UE vV (Analytical Solution of CCA)
(3)
where UT PV are the resulting SVD matrices of the rank-r matrix --
Ray xg" The correlation coefficients p of Equation are the diagonal entries of P in descending
order pr 2 P2 2 ... 2 Pr like the descending principal components of PCA.
4.2
CANONICAL SIMILARITY ANALYSIS
We now define CSA. For any pair multimodal data xx),, with slight abuse of the notation of original data and features, we define the similarity as:
( 87), P.(A (B =)) 23:) [I Ie 22
(Canonical Similarity)
(4)
where (+):ss denotes the 1st to s-th row vector, (-); denotes the i-th entry, and s is a pre-defined
hyperparameter. Note that Equation 4 is the weighted cosine similarity of the first s dimensions of the transformed data matrix. The weights determine how much each dimension influences the overall similarity score, depending on their level of correlation. We chose the hyperparameter s so that CSA preserves the information of dimensions with correlations greater than Pa. As stated earlier, the feature space might contain modal-specific information that cannot be mapped to the other modality. Thus, only take into account meaningful bases. We choose CCA with linear correlation instead of complicated kernel CCA because most contrastive learning loss functions only contain the linear inner product of the feature vectors, which is also the case in Equation 2
4

---

Published as a conference paper at ICLR 2025
4.3 TRAINING AND INFERENCE
Similar to standard machine learning models, CSA consists of both training and inference phases. We split the multimodal data into training and test sets, just like in standard machine learning. CSA performs all optimizations on the training set, and the test set is not known a priori and is used only for evaluation. During the training phase, or also called the fitting phase, CSA obtains the training set of encoded features from the unimodal encoders and then solves Equation 2 Note that CSA does not train any encoders, so it does not require GPUs. The resulting correlation coefficients p determines the value of s in Equation 4
arg min P: s.t. P: 2 const.
(5)
Note that const here is an empirical constant threshold to optimize performance in our experience. Finally, for any multimodal downstream task, CSA uses Equation A to evaluate the similarity be- tween any pair of multimodal data. We later show the experimental results of CSA and compare the results with the state-of saate-of-the-arr the--aterrttt methods.
THEORETICAL ANALYSIS OF CSA
In this section, characterize the trade-off of obtaining informative embeddings and distinguishing multimodal data pairs caused by selecting the appropriate s in Equation [ We start with the linear setting, but the conclusion holds for a more general setting.
Assumptions. We follow the assumption of data generation as per Joshi et al. (2024) and analyze the effect of CSA based the theoretical results of contrastive losses (Ji et al. 2023 Tian 2022
Nakada et al. 2023). Suppose that the N pairs of data {a), 221 from two modalities m m are generated as follows:
(6) Here, IR9 is the shared unobservable latent feature of the two modalities, and T RT' xq
= T'= + -
xg,72 T RB*'x xq the linear mapping functions of the latent vectors to the observable data. Without loss of generality, we assume that both the latent feature vectors and noise for each modality independent and identically distributed (i.i.d.) samples from any distribution.
Optimal Linear Encoders of Contrastive Loss. Previous work (Ji et al. 2023) gives the analytical solution of unimodal linear encoder E' trained linear contrastive loss with regularization:
-1
(E
E
(E*
EE
C(X*, x
E*
2(E
(E*
2N
2NV
E
1
for t: = 1,2,
(Unimodal Contrastive Loss)
2
(7)
where X x'* are two sets of original and augmented (e.g., random masked, Gaussian blurred, etc.) data from the same modality, and À is hyperparameter weighting the regularization term. Note that this contrastive loss function is similar to Equation [ but the second modality is the augmented data now. For simplicity, assume that all unimodal data augmented with complementary random masking with probability 0.5, so the optimal linear encoder is: for k 1,2.
Esk
arg min C FeRexpe
Σ moooe7
Fliner
In Equation basis, and di the i-th largest cigenvalue of matrix:
g C is a positive constant related to À in Equation 7 e, is the i-th standard vector
OffDiag
N N-1 X -
(9)
where OffDiag denotes the function that makes all diagonal entries and is square matrix of ones. From Equation 3 and 8, the solution of CSA in linear setting is:
2 = E A'El 1inear (T +t), B = B 2
B'
(T²2, + c7) (10)
5

---

Published as a conference paper at ICLR 2025
10
SNR (t higher is better)
0.6698
original
Amin (f higher is better)
1.0
0.0035
shuffled
Similarity score
0.0010
0.0003
0-
0.5
0.0
-10
-0.
10
20
30
1
11
21
31
Selected dimension for similarity score s
Selected dimension for similarity score
(a) SNR and Amin vs. selected dimension.
(b) p-value (lower better) similarity scores.
Figure 2: Trade-off of canonical similarity analysis: (a) When is low, the signal-to-nonn ratio (SNR) of the data is high, and the minimum singular value Amin is large. It preserves the distance between contrastive data. (b) When 8 is low, the p-value large, and original and shuffled distrihutions are alike, so the similarity score is meaningless. (a) shows the desirable properties when s is low and (b) shows the opposite.
Sensitivity to Noise. The noise in Equation 6 affects the unimodal linear encoders Ei Flineer E2 liuear from the second term of Equation 9 It hinders the unimodal encoders in finding the inverse of T1.7² but in encoding the noise. The noise then propagates and affects the covariance matrices of 2 and the decomposed orthogonal matrices U and V in Equation 3] Hence, we rely the hyperparameter s to prevent noise from dominating. When s increases, we include the less correlated dimensions caused by noise in the canonical similarity metric in Equation
Distinguish Contrastive Data Pairs. In Equation| 7 similar sample pairs, i.e., the augmented and original data, stay close each other, while dissimilar ones, i.e., augmented data from different data points, are far apart. However, in the objective of CCA Equation 2 we focus only on maximizing the correlation coefficients of similar samples while ignoring dissimilar ones.
We now analyze how CSA distinguishes contrastive latent feature pairs from the two unimodal encoders. Given data pairs (1 wā) and (x],, xz)), we assume that the predicted latent feature pairs (2) 2)), (22 2f?) are distant because the unimodal encoders are trained for it. The distance between latent features preserved in the matrix transformations bounded as follows:
Amin (A*) (Am) (Eineea 1| 1bll2 S Amnin (A E ||
21 S ||A (2) =
(11)
Amin (B*) Amin ( 2 21112 S Amin (B'E Eiinear 11x 2 2 - B (E) =b)
where Amin (.) denotes the minimum singular value, which is the opposite of the Lipschitz constant of matrix. From Equation IT we can see that the distance between the CSA features (right-hand side) is lower bounded by the distance of the original data with a constant of minimum singular values (left-hand side). Thus, know how the hyperparameter selecting the dimension of CSA affects the distance between dissimilar data pairs. When the hyperparameter decreases, CSA takes into account fewer dimensions per Equation 4 That is, Amin i Ps is effectively larger, which ultimately results in more distant features in the CSA space. and vice versa. In summary, a smaller preserves distance in the CSA space, which distinguishes contrastive data pairs.
Trade-off of Canonical Similarity Analysis. Previous results show that a smaller s leads to less noisy but more contrastive similarity scores, which desirable. However, s is not desirable because it loses much information about the true latent features, and most vectors will have identical similarity scores. In practice, we do not know the dimension of the latent feature q and can only use CSA to project the encoded features of two modalities to a dimension that is less noisy, but also contains enough information about the true shared latent feature z.
Numerical Experiments. We validate the trade-off through numerical experiments following Equa- tion Figure 2a shows that if we only account for a few dimensions (s is low), the signal-to-noise ratio (SNR) of the data is then high, and the minimum singular value Xmin is large, which preserves the distance between contrastive data. Note that Figure 2a shows the values in decibels (dB). Fig- ure 2b compares paired data (a), x?) and randomly shuffled data (x), # j. High Wilcoxon p-values mean that the two distributions of similarity scores from Equation 4 are alike. In this case, the canonical similarity score is meaningless, and larger is more desirable. The intrinsic trade-off

---

Published as a conference paper at ICLR 2025
Method
Train images
Train text
Parameter
Feature dimension g,
CLIP
2B
2B
1.3B
1280
GTR
X
2B
335M
768
DINOv2
142M
1.IB
1536
CSA (ours on ImageNet)
35k
700
CSA (ours on Leafy Spurge)
800
X
5 250
CSA (ours Flickr30k)
5k
25k
X
200
CSA (ours COSMOS)
41k
41k
X
750
Table 1: Comparison of methods: CSA utilizes two unimodal encoders to match the performance of CLIP while requiring significantly less data. We show the CLIP parameters of only one encoder for fair comparison other unimodal encoders.
Classification accuracy
Classification sccuracy
0.11
0.60
0.5%
0.4
0.50
CSA (aurs)
CSA (ours)
ASIF
ASIF
5000 0.0
10000
15000 20000 25000 30000 35000
400
500
600
700
800
Amount of training data
Amount of training data
(a) ImageNet.
(b) Leafy Spurge.
Figure 3: Image classification: CSA (blue) is highly data-efficient requiring only 35, 000 training samples match the performance of CLIP in ImageNet and 360 samples in Leafy Spurge.
Method
mAP@5
Precision @
Precision@5
Method
Precision
CSA (ours)
36.6%
59.3%
43.4%
CSA (ours)
44.7%
CLIP
73.8%
92.9%
77.2%
CL.IP 79.5%
ASIF
14.6%
25.6%
20.0%
ASIF
0.1%
(a) Image-to-text retrieval
(b) Text-to-image retrieval
Table Cross-modal retrieval on Flickr30k: CSA exhibits lower performance compared to the multimodal baseline, CLIP. However, outperforms the unimodal baseline, ASIF, in both retrieval scenarios.
in selecting s is now clear. When s is too small, do not have meaningful similarity scores that distinguish unpaired data, but when s is too large, we have noisy and closer clusters of features.
EXPERIMENTS
Baselines. We compared CSA with another baseline, ASIF (Norelli et al. (023). has the same setting CSA, which uses two unimodal encoders to calculate the similarity of multimodal data pair. Unlike CSA, which solves an optimization problem, ASIF infers multimodal similarity by the unimodal distance between data points. ASIF is the only fair baseline for CSA. We trained CSA and ASIF with the same training set and unimodal encoders and evaluated all the methods in the test We also compared CSA with the state-of-the-art multimodal encoder model from OpenCLIP (Ihaarco et al. 2021: Cherti et al. 2022). acknowledging their incomparable training set and model parameters, shown Table [ In addition to encoder models, we compared the performance of LLa 13B (Touvron et al. 2023). multimodal large language model, and prompted it to solve the specified tasks from the provided image and text. The unimodal encoders of CSA and ASIF are GTR (Ni et al. 2022) and DINOv2 (Oquab et al. 2023) here. Section shows the detailed settings.
Image Classification. We first examined CSA's ability image classification on ImageNet and Leafy Spurge. All methods input an image and all possible captions "This is image of label" and select the most similar caption as the predicted label. In Figure 3al CSA requires only 35,000 training samples to match the performance of CLIP and consistently outperforms ASIF, which needs millions of data points to achieve a similar performance to CLIP. Since CLIP is zero-shot, its performance is independent of the amount of data. Leafy Spurge challenges the capability of methods with

---

Published as a conference paper at ICLR 2025
1.0
0.8
Tue prsiiivee rele
0.8
Tuue positve
0.6
0.6
0.4
(ours)
EI AUC--AA
0.2
ASII AUC-0,50
0.2
ASII AUC-0:50
ULeVA
X ULNNA
0.0
0.0
0.2
0.4
0.6
0.5
1.5
0.0
0.2
0.4
0.6
0.3
1.5
False positive rate
False positive rate
(a) Results for a training size of 5000.
(b) Results for training size of 35000.
Figure 4: Detecting mislabeled ImageNet images: CSA (blue) outperforms CLIP, ASIF, and LLaVA with a higher AUC. (a) and (b) illustrate the results for CSA and ASIF across 2 training set sizes, showing the superior performance of CSA with limited and noisy training data.
extremely limited data. This dataset includes only 800 training images and 100 test images of plants with binary classes: leafy spurge and others. The images are out-of-distribution for any encoders, as this dataset was newly published in 2024 and captured using drones. Figure Sbishows that while the performance of CSA fluctuates, it consistently outperforms CLIP and ASIF in this challenging scenario, demonstrating CSA's ''s effectiveness with limited -iitt-ddiitiitttion data.
Table shows the size of the training set and the number of parameters of the neural network encoders. Notably, CSA surpasses CLIP by utilizing two unimodal model encoders trained on 2 billion text samples and 142 million images, along with only 35,000 multimodal data points. This is 50,000x less unimodal data compared to the CLIP training set. We also visualize the image features of CSA and CLIP in Section
Cross-modal Retrieval. We examined CSA and the other baselines on a cross-modal retrieval dataset, Flickr30k (Young al. (2014), which contains 15,000 images, each paired with 5 textual descriptions. This dataset is common for evaluating the performance of image-to-text and text-to- image retrieval. For image-to-text retrieval, the task to find the corresponding caption from a set of possible candidates (the so-called reference set) given a query image and vice versa. We retrieved the most similar images or text from the reference set according to the similarity scores of all meth- ods and showed the results in Table 2 For image-to-text retrieval, we show mean average precision (mAP), precision @1 I, and precision @5. 5. For text-to-image retrieval, we only evaluated performance using precision@ @1 1 since there is only one correct image. From Table CSA exhibits lower perfor- mance compared to the multimodal baseline, CLIP. However, it outperforms the unimodal baseline, ASIF, in both retrieval scenarios. Notably, ASIF completely fails in the text-to-image retrieval task.
Detecting Mislabeled ImageNet Images. We again show CSA's superiority with limited and noisy data. Previous works show that ImageNet is imperfect and contains incorrectly labeled images, leading to several failure modes of downstream models (Vasudevan et al.) 2022). Trained with the original incorrect ImageNet dataset, all methods detect if the input image and the label align. We use
labels from (Nrrthcutt et al.) 202I) to define incorrectly labeled images.
All methods input an image and caption "This is image of label' and output if the image and caption align or not (true or false). LLaVA answers questions about whether the given image and caption align. CSA, CLIP, and ASIF output true if the calculated similarity score of the image and caption pair is greater than a threshold. We enumerated the thresholds to obtain several true and false positive rates and show the receiver operating characteristic (ROC) curves in Figure 4 In the legend, we listed the area under the curve (AUC), where higher value indicates better performance. In Figure 4 CSA outperforms all other baselines, even LA,,A, given 5,000 image-label pairs. We also see that when the size of the training data increases from 5,000 to 35,000, the performance of CSA and ASIF increases as well. We emphasize that the comparison of AUC does not directly compare accuracy. As shown in Figure 4b. the true positive rate of CSA outperforms LLaVA by nearly 10% under the same false positive rate.
Note that the training sets of CSA and ASIF contain mislabeled ImageNet images, which are 11% of the training data in both dataset sizes. It shows CSA robustness to noisy data and efficiency in learning from just 5,000 pairs. Similarly, the CLIP encoders' training set also contains mislabeled
8

---

Google-retrievel Published as a conference paper at ICLR 2025
BLADK LIEE
True postt rato
"Jiiin Cadtro nix atrouroemert Artivato Ter, Saturday'
Tormeto Raptorr amboe at Dicney Campur Blace Matter Rluiss
2.4
CSA jours). AUC-0070
CUP AUC+0.68
San A narrio, Tex, Saurrby, Mr. Cectro the larmer secretary bruwing and urson development would be ane prstesstisssteee
pholograph abraws purchwoud Black lon, Metter to batsport membiars nate
3.-
AJCHO.64
0.0
0,4
0.5
08
10
Accurate Caption
Misinformative Caption
False positive rate
(a) Illustrative examples.
(b) Results.
Figure 5: Detecting misinformative news captions: (a) We consider the retrieved captions misinformative if they do not align with the images and the corresponding captions. (h) CSA (blue) outperforms CLIP, ASIF, and LLaVA with higher AUC. The orange cross is the supervised learning method from the original COSMOS paper trained with labels of object locations, which the only method that outperforms CSA.
ImageNet images, making the comparison fair. We later demonstrate CSA robustness with noisier training data.
Detecting Misinformative Captions. Detecting mislabeled images is easy since text captions differ only in labels. We considered a much harder task to detect misinformation news captions from the COSMOS dataset (Aneja al.) 2023). The task involves determining whether caption aligns with news images and its original captions, as shown in Figure [a The benchmark considers the retrieved captions misinformative if they do align with the images and the corre- sponding captions simultaneously. The ultimate goal of the task is to identify and prevent the spread of misinformative news captions on the Internet.
We evaluated the similarities between the image and its retrieved caption, as well as between the two captions, and identified misinformation when both similarity scores fell below the respective thresholds. Figure [5b] shows all the ROC curves and their AUC. Again, CSA outperforms CLIP with two unimodal encoders. The supervised learning method from the original COSMOS paper is the only method that performs better than CSA, although trained with supervised labels of object locations. Note that the LLaVA agent completely fails this task with a false positive rate of 100%, as it always thinks that the retrieved captions align with the given images and the original captions.
Robustness to noisy data. To test CSA's robustness, we randomly shuffled a percentage of the training labels of ImageNet and evaluated the resulting test accuracy. Fig- ure shows that CSA constantly outperforms ASIF and achieves 70% accuracy even if 50% of the training labels incorrect. When all the data shuffled,, both CSA and ASIF degrade to 0% accuracy.
Classification accuracy
0.8
0.4
0.2
CSA (ours)
Towards more modalities and ablation study, We show additional results in the Appendix to provide compre- hensive study CSA. To show that CSA generalizes other modalities, we tested it on audio and text data in Section text-to-LiDAR retrieval in Section and text- to-timeseries in Section G See Section C for the corre- lation coefficients of multimodal data and the results of CSA with various in Equation 4 in Section D and uni- We visualized and examined the CSA embeddings in Section and Section Lastly, we present the ablation study of CLIP encoder architectures in Section
ASIF
0.0
5000 10000 15000 20000 25000 30000 35000 Amount of shuffled training data
Figure 6: Robustness to noisy ImageNet data: CSA remains performant even when the training labels, comprising total of 35,000 samples, are partially shuffled.
modal encoders in Section E
Limitations. CSA is suitable for bimodal data, similar to CLIP, unlike Imagebind (Girdhar et al. 2023) is capable of 6 modalities. Like ASIF, CSA is based unimodal encoders. If the unimodal encoders are not foundation models but trained on limited datasets like MNIST, the performance of CSA degrades. Once the encoded features obtained, CSA is just SVD. In our case, the underlying

---

Published as a conference paper at ICLR 2025
implementation of CSA is NumPy, which is efficient in handling large matrix operations parallelly on multiple CPUs. Also, solving SVD with NumPy takes more time and computationally efficient than training models GPUs for datasets of the same size. There are tricks speed up SVD approximately. For instance, GPU libraries like CuPy Okuta et al. (2117). and randomized SVD algorithms compute approximate results faster than full SVD. Lastly, for given hyperparameter 8, we only need to obtain the first & singular values, which can also speed up the computation.
Summary. CSA always outperforms the unimodal baseline, ASIF, and matches or exceeds CLIP's performance, except in cross-modal retrieval. This exception likely arises because cross-modal re- trieval involves comparing thousands of similarity score pairs between the query data and the refer- ence set, unlike the other tasks that compare against certain captions. It highlights the trade-off of CSA discussed in Section CSA encounters trade-off between distinguishability and informative- ness, Le., to maintain the distant features. Retrieval tasks require a more curated balance between these aspects, and other tasks benefit from greater distinguishability of similarity scores.
We conclude that CSA is robust and data-efficient method to replicate the CLIP similarity score using two unimodal encoders. In [able the amount of data required compared to CLIP underscores that unimodal encoders are significantly more efficient to train. A small amount of multimodal data suffices to learn the mapping of unimodal features to a multimodal space, even the training data unprocessed by humans (COSMOS) or incorrectly labeled (ImageNet).
CONCLUSIONS AND FUTURE WORKS
In this work, we proposed CSA, which maps two unimodal feature spaces from pre-trained encoders to a multimodal feature space. CSA is extremely data-efficient. We characterize the intrinsic trade- off of CSA, and CSA shows competitive performance compared to CLIP models in image classifi- cation, mislabeled data detection, text-to-LiDAR retrieval, and misinformation detection tasks with limited data. CSA also outperforms the state-of-the-art method in the same setting.
Future work includes extending CSA to more than 2 modality pairs, just as generalized CCA (Horst, [961] extends CCA more sources. In addition, understanding the relationship between the size of the training set and its performance is crucial. If we can fine-tune the unimodal encoders, which loss function will result the most suitable feature space for CSA remains an open problem. Lastly, CSA essentially finds a mapping function between two feature spaces, which need not be modality features. We aim to test CSA on mapping intramodal data, such multi-view images of an object.
REPRODUCIBILITY STATEMENT
For the theoretical analysis of this work, we state all assumptions made in the assumptions para- graph. For all the hyperparameters and detailed settings of the experiments, please refer to Appendix Section A Lastly, we put the core code of CSA in the supplementary details. The code includes dat- aloaders, execution code, and links to download all the datasets and models used. We will provide an example configuration file after publication, as it contains non-anonymous directory paths.
ACKNOWLEDGEMENT
This work was supported in part by the National Science Foundation grants No. 2133481 and No. 2148186, NASA 80NSSC2IM0071, the Office of Naval Research (ONR) under Grant No. 2-0014-22-1-2254, the Defense Advanced Research Projects Agency (DARPA) contract FA8750- 20-8-1018, and DARPA ANSR: RTXCW2231110, Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. We also thank Mohammad Omama and Sai Shankar Narasimhan for their valuable feedback and discussions on the paper.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank. Musiclm: Generating music from text, 2023. URL https: /ttpss//arxiv.
org/abs/ 2301 11325
Shivangi Aneja, Chris Bregler, and Matthias Nießner. Cosmos: catching out-of-context im- age misuse with self-supervised learning. In AAAI, 2023. ISBN 978-1-57735-880-00. 10.1609/aaai.v37// v37.:aaii.771122266666 URL htttps://doot
doi:
Gabriele Berton, Carlo Masone, and Barbara Caputo. Rethinking visual geo-localization for large- scale applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Con- stant, Mario Guajardo-Cespeddss Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Universal sentence encoder, 2018. Utthtttss///xxit.tcoom http://awww....rr///////a///////
11175
James Chapman and Hao-Ting Wang. Cca-zoo: A collection of regularized, deep learning based, kernel, and probabilistic cca methods in scikit-learn style framework. Journal of Open Source Software, 6(68):3823, 2021.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. arXiv preprint arXiv: 12.07/2 2022.
Maximilian Christ, Nils Braun, Julius Neuffer, and Andreas W Kempa-Liehr. Time series feature extraction on basis of scalable hypothesis tests (tsfresh-aa python package). Neurocomputing, 307: 72-77, 2018.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: //openzeview. net www.eeeetteottttooeoooooo d=w.mmmiiiiiiio d=Y dfbbbFdNTTT
Samir Yitzhak Gadre. Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smymiss, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jicyu Zhang, Eyal Orgad, Rahim En- tezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexan- der Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets, 2023. URL httph:t////// http:s://axxvvvvo bbbs/2304 14108
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all, 2023.
Harsh Goel and Sai Shankar Narasimhan. Improving end-to-end autonomous driving with synthetic data from latent diffusion models. In First Vision and Language for Autonomous Driving and Robotics Workshop, 2024.
Andrey Guzhov, Federico Raue, Jöm Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), pp. 976-980, 2022.
11

---

Published as a conference paper at ICLR 2025
David R. Hardoon, Sandor Szedmak, and John Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural Computation, (6(12):2639-2666) 2004.
Paul Horst. Generalized canonical correlations and their application to experimental data, Journal of clinical psychology, 14, 1961.
Gabriel Ilbarco. Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021.
Wenlong Ji, Zhun Deng, Ryumei Nakada, James Zou, and Linjun Zhang. The power of contrast for feature learning: A theoretical analysis. Journal of Machine Learning Research, 24(330):1-78, 2023.
Siddharth Joshi, Arnv Jain, Ali Payani, and Baharan Mirzasoleiman. Data-efficient contrastive language-image pretraining: Prioritizing data quality over quantity. In International Conference on Artificial Intelligence and Statistics, 2024.
Po-han Li, Sravan Kumar Ankireddy, Ruihan Philip Zhao, Hossein Nourkhiz Mahjoub, Ehsan Moradi Pari, Ufuk Topeu, Sandeep Chinchali, and Hyeji Kim. Task-aware distributed source cod- ing under dynamic bandwidth. Advances in Neural Information Processing Systems, 36, 2024a.
Po-han Li, Oyku Selin Toprak, Aditya Narayanan, Ufuk Topcu, and Sandeep Chinchali. Online foundation model selection in robotics. arXiv preprint arXiv:2402.055: 085500 2024b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023.
Qi Lyu and Xiao Fu. Nonlinear multiview analysis: Identifiability and neural network-assisted implementation. JEEE Transactions on Signal Processing, 68:2697-2712, 2020. URL https:
//ieeexplore Leee. wwwpor/////o///////////////.ee/e 90048884
Qi Lyu and Xiao Fu. simite-sample analysis of deep CCA-based unsupervised post-nonlinear mul- timodal learning. IEEE Transactions on Newral Networks and Learning Systems, 34(1 1) 1):9568- 9574, 2023. URLpttps: httttss://eeexpploo/
ieee. wttp://oooeeo///////////
Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. In International Conference on Learning Representations, 2023. httphhtttt///ppeeeeeeeeee ftte//oormiidd=:11
eGJ
Ryumei Nakada, Halil Ibrahim Gulluk, Zhun Deng, Wenlong Ji, James Zou, and Linjun Zhang. Understanding multimodal contrastive learning and incorporating unpaired data. In International Conference on Artificial Intelligence and Statisties, pp. 4348 4380. PMLR, 2023.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. Large dual encoders are gener- alizable retrievers. In Proceedings of the 2022 Conference Empirical Methods in Natu- ral Language Processing, December 2022. doi: 10. 112116621/1/2222mmmm.pp. mainlp--min.669.. URL mmnlp-main... 669
https: / ht/aaaaaannnoooooooooooo www.oogg/22222
Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodolà, and Francesco Locatello. ASIF: Coupled data turns unimodal models to multimodal without training. In Advances in Neural Information Processing Systems, 2023. htttps://oppereeeet
net/ piet/Eorum?ico id= 30 1mWE1
Curtis G. Northcutr, Anish Athalye, and Jonas Mueller. Pervasive label errors in test destabilize machine learning benchmarks. In Proceedings of the 35th Conference on Neural Information Processing Systems Track on Datasets and Benchmarks, December 2021.
Ryosuke Okuta, Yuya Unno, Daisuke Nishino, Shohei Hido, and Crissman Loomis. Cupy: A cumpy-compatible library for nvidia gpu calculations. In Proceedings of Workshop on Machine Learning Systems (Learningyss) in The Thirty-frrst Annual Conference Neural Information Processing Systems (NIPS), 2017. URL _6.p_____1pd__
wrw.oorgggng.. wwtp:/sstppggp/popttpt.toptptctt /
papers/paper pepers/paper_16.pppp
12

---

Published as a conference paper at ICLR 2025
Mohammad Omama, Po-han Li, and Sandeep P Chinchali. Exploiting distribution constraints for scalable and efficient image retrieval. arXiv preprint arXiv:2410.00022, 2024.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nico- las Ballas, Wojciech Galuba, Russell Howes, Po-Ya Yao Huang, Shang-Was Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
Alec Radford, Jong Wook Kim. Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
pp.
Viresh Ranjan, Nikhil Rasiwasia, and CV Jawahar. Multi-label cross-modal retrieval. Proceedings of the IEEE international conference computer vision, 2015.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned. hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018. doi: 10.18653/v1/P18-- 1238. /tLhtttss://accaattt hoc.laantoooooy.oor//ll---222
Mohammad Shokoohi- Yekta, Bing Hu, Hongxia Jin, Jun Wang, and Eamonn Keogh. Generalizing dtw to the multi-dimennional case requires an adaptive approach. Data mining and knowledge discovery, 31:1-31, 2017.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
Sai Shubodh, Mohammad Omama, Husain Zaidi, Udit Singh Parihar, and Madhava Krishna. Lip- loc: Lidar image pretraining for cross modal localization. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 948-957, 2024.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. Proceed- ings of the 44th international ACM SIGIR conference on research and development in information retrieval, pp. 2443-2449, 2021.
Zhongkai Sun, Prathusha Sarma, William Sethares, and Yingyu Liang. Learning relationships be- tween text, audio, and video via deep canonical correlation for multimodal language analysis, 2019. URL
org/ 1111199111111 11111.55540
Yuandong Tian. Understanding deep contrastive learning via coordinate- wise optimization. In Ad- vances in Neural Information Processing Systems, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, 2019.
13

---

Published as a conference paper at ICLR 2025
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma- chine Learning Research, 9(86):257--2055, 2008. URL http:// wwwp.org/papers///// wandermaatemDaa.hrmm
Vijay Vasudevan, Benjamin Caine, Raphael Gontijo Lopes, Sara Fridovich-Keil. and Rebecca Roelofs. When does dough become a bagel? analyzing the remaining mistakes on imagenet.
Advances in Neural Information Processing Systems, 35:6720- 6734, 2022.
David Weenink. Canonical correlation analysis. In Proceedings of the Institute of Phonetic Sciences of the University of Amsterdam, volume 25, pp. 81-99, University of Amsterdam, 2003.
Yusong Wu*, Ke Chen*, Tianyu Zhang*, Yuchen Hui*, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audin pretraining with feature fusion and keyword- to-caption augmentation. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), 2023.
Senqiao Yang, Jiaming Liu, Ray Zhang, Mingjie Pan, Zoey Guo, Xiaoqi Li, Zehui Chen, Peng Gao, Yandong Guo, and Shanghang Zhang. Lidar-lim: Exploring the potential of large language models for 3d lidar understanding. arXiv preprint arXiv: 2023.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
14

---

Published as a conference paper at ICLR 2025
Appendices
A ADDITIONAL DETAILS ON THE EXPERIMENTS
We run all inferences of LLAVA and encoder models on NVIDIA RTX A5000 GPU, and solving Equation 2 with 35,000 multimodal feature vectors on a 64 core Xeon Gold 6226R CPU machine takes less than 10 minutes. The implementation of CCA is from CCA Zoo Chapman & Wang 2021).
Multimodal Encoders. The multimodal image-text encoder used throughout the experiments is laion/CIIPP- -iw.-II-VVV--------- VVT-bbigG-14-laionnbk 14-laiion2n--99 390-bb60kk from Huggingface. The multimodal audio-text en- coder used is laion/larger clap general from Huggingface (Wu* et al., 2023).
Unimodal Encoders. We use several unimodal encoders and show the difference in performance in Section E To encode images, we tested ANNOv--Gian (Oquab et al., (2023) and the unimodal part of the multimodal encoders previously mentioned. To encode text, we tested ----agee (Ni al.) 2022) and the unimodal part of the multimodal encoders mentioned above. To show CSA 's's ability to combine unimodal models, we never tried using the paired unimodal encoders of a multimodal encoder in our experiments, i.e., using CLIP to encode both images and text.
Flickr30k. We trained ASIF and CSA on the Flickr validation set, which includes 145,000 images and 5 captions for each image. We then validated the models test set of 5,000 images and 25,000 captions.
COSMOS. We trained ASIF and CSA the COSMOS validation set, which includes 41,006 image-caption pairs. We then validated the models using a test set of 1,700 image-caption pairs, with half of the captions labeled as misinformation by human annotators.
B TOWARDS MORE MODALITIES NODALITIES-AUDIO AND TEXT
1.0
C.8
positive rate
0.6
0.4
CSA (ours). AUC=0.94
C.2
CLAP, AUC=0.94 94
ASIF, AUC-0.79
0.0
0.0
0.2
0.4
0.6
0.8
10
False positive rate
Figure 7: Classification YouTube audio and genre tags: CSA (blue) performs as well CLAP, the CLIP- inspired multimodal audio and text encoder, and outperforms ASIF classifying genre tags of YouTube audio.
We now show CSA's generalization ability to more modalities with MusicCaps (Agostinelli et al. 2023]. We use GTR and CLAP to encode YouTube audio along with the tagged genre descriptions of the audio. We conducted a classification task in which the models input the audio and tag and output if the audio aligns with the caption. Similar to the mislabeled ImageNet experiment, we show the ROC curves and compare the AUC in Figure 7 We trained ASIF and CSA for 3, 777 data points and tested all methods on 1, 625 data points. We randomly sampled a tag for each data point during both training and inference. In Figure 7 CLIP-inspired multimodal audio and text encoder, and outperforms ASIE Thus, we conclude that CSA extends its capabilities beyond image and text, effectively handling audio and text well.
that CSA performs as well CLAP, the
15

---

Published as a conference paper at ICLR 2025
Correlation coefficient
1.0
0.8
0.4
0.2
0.0
200
400
600
800
Projected dimension
Figure 8: Correlation coefficients of COSMOS image and caption features under CSA: The data inherently noisy, indicated the correlation cnefficients the unimodal feature spaces, which concentrate on 0.2 0.4. The unimodal encoders here are GTR DINOv2.
C CORRELATION OF FEATURE SPACES
To take deeper look into unimodal feature spaces, show the correlation coefficients of COSMOS image and caption features under CSA in Figure g The data are inherently noisy, as indicated by the correlation coefficients of the unimodal feature spaces, which concentrate 0.2 to 0.4. This distribution of correlation coefficients highlights that, despite the fact that the original multimodal data noisy and show complex correlations, CSA effectively map them multimodal space where the similarity score remains meaningful for the zero-shot downstream tasks.
D SENSITIVITY TO HYPERPARAMETER
Method
mAP
Precision
Precision@55
Method
Precision
CSA
10
9.5%
18.1%
13.4%
CSA
10
15.2%
CSA
50
32.7%
53.9%
40.0%
CSA
41.2%
CSA
100
36.0%
58.3%
42.9%
CSA
100
43.8%
CSA
200
36.6%
59.3%
43.4%
CSA
200
44.7%
CSA
500
31.8%
56.3%
38.3%
CSA
500
41.7%
CSA
750
27.3%
50.2%
33.6%
CSA
750
40.1%
CLIP
73.8%
92.9%
77.2%
CLIP
79.5%
ASIF
X
14.6%
25.6%
20.0%
ASIF
X 0.1%
(a) Image-to-text retrieval.
(b) Text-to-image retrieval.
Table 3: Cross-modal retrieval Flickr30k under different CSA achieves optimal performance 200, its performance degrades increases and decreases s, illustrating the trade-off characterized in Section
Retrieval tasks: We show CSA's sensitivity to the hyperparameter in terms of the end perfor- mance. In Table 3 CSA achieves optimal performance in 200, and its performance degrades with increases and decreases in illustrating the trade-off characterized in Section []] However, for tasks other than retrieval, we find that a larger s improves performance in image classification, mislabeling detection, and misinformation caption detection. This phenomenon is likely due to the trade-off between distinguishability and informative embedding features, namely the distance be- curated balance between these aspects, other
tween features. Although retrieval tasks require tasks benefit from greater distinguishability similarity scores.
Detection tasks: Table shows the sensitivity to hyperparameter of CSA on ImageNet mislabeled data and COSMOS misinformative news captions detection. The relation between hyperparameter and the AUC performance is different from retrieval tasks (Flickr30k and KITTI), as shown in Table 3 In detection tasks, the AUC increases as hyperparameter s increases. We explained this phenomenon in the theoretical analysis with the perspective of feature distinguishability above.
16

---

Published as a conference paper at ICLR 2025
0.0
0.0
0.0
0.0
0.4
0.4
+
CSA lours). AUC-0.09
CSA lours). AUC-0.73
CLIP AUC=0.60
CLIP AUC-0.60
ASIF. AUC-0.50
ASIF: AUC-0.5C
X 1iw
X
0.0
0.0
positive
(a) Results for training size of 5000.
(h) Results training size of 35000.
Figure 9: Detecting mistabeled ImageNet images (cont'd): CSA (blue) outperforms CLIP, ASIF, and LLaVA with a higher AUC. (a) and (b) illustrate the results for CSA and ASIF across various training set sizes, showing the superior performance of CSA with limited noisy training data. The unimodal encoders are GTR and CLIP (image) here.
True positive rate
CSA (ours). AUC-C,73
0.2
AUC-
LLaVA
0.0
salse positive rate
Figure 10: Detecting misinformative COSMOS captions (cont'd); CSA (bluc) outperforms CLIP, ASIF, and LLaVA with higher AUC. The supervised-lcarinng method from the original COSMOS paper is the orange It is the only method that outperforms CSA, though trained with supervised labels of object locations. The unimodal encoders are GTR and CLIP (image) here.
E SENSITIVITY TO UNIMODAL ENCODERS
We change the unimodal encoders of ASIF and CSA to showcase their generalization ability to different unimodal encoders. Figure shows the results on the detection of mislabeled ImageNet data with other encoders, and Figure shows the results on the detection of misinformative captions with other encoders. CSA again outperforms ASIF and CLIP while outperforming the results of the combination of GTR and DINOv2 in Section
F TEXT-TO-LIDAR RETRIEVAL
Settings: We additionally CSA's capability on new, unexplored modality pairs. The KITTI dataset, designed for place recognition in autonomous vehicles, contains RGB and LiDAR images captured Karlsruhe, Germany. To add text modality, we use LLaVA Liu et (2023) to generate descriptive captions for the images with a prompt "Describe the static objects and the numbers of objects the image within 20 words." In place recognition, autonomous vehicles utilize local RGB, LiDAR, and text captions to locate themselves within city by retrieving the nearest landmark from a reference set. The landmarks act as the reference data, while the vehicles' local observations serve query data, resulting in trimodal data for this task. We used Lip-loc Shubodh al. 22024), and the image-LiDAr encoder, to encode the LiDAR data and GTR Ni et al. (2022) to encode the captions. The train set contains 5,000 cross-modal instances, and the test contains 6,000 instances. The
17

---

Published as a conference paper at ICLR 2025
retrieval task here is to locate the vehicle at any landmark within 20 meters. We set this distance threshold per Shubodh et al. 20224).
We conducted retrieval on KITTI, augmented with textual captions. Figure I1 shows that CSA achieves the same performance as -iDAR-to-LDDR retrieval, showcasing CSA's capability of bridging multimodal data. Notably, are the first to perform text-to-LiDAR retrieval, only made possible by CSA, which maps LiDAR and text embeddings into a shared feature space.
Results:
Reference modality
LiDAR (Lip-loc)
LIDAR (Lip-loc)
32.4
Method
AUC (ImageNet)
AUC (COSMOS)
Query modality
CLIP
0.66
0.68
ASIF
0.50
0.64
0.65
25
0.59
0.66
Text (GTR)
50
0.62
0.68
CSA
32.8
100
0,65
0.69
500
0.72
0.70
700
0.73
0.70
Table 4: Sensitivity to s on detection tasks: analyzed in the appendix the main paper, the AUC increases as increases. This trend holds for all detection tasks, which is different from the trend of retrieval.
Figure 11: Text-to-LiDAR retrieval on KITTI: The figure shows the Recall@5 5 re- trieval rate KITTI. The cross-modal re- trieval result from the LiDAR query is di- rectly from Lip-loc, while other with text from CSA. CSA matches the performance of Lip-loc, which is trained on KITTI.
G TEXT-TO-TIMESSERIS CLASSIFICATION
Setting: We now demonstrate the effect of CSA on more modality pairs. We conducted classi- fication handwritten alphabets. One modality is the 3-dimensional time series of movement of pens on a pad Shokoohi- Yekta et al. (217)) The other modality is either the images of alphabets or the text of Allphbeet X." We leveraged tsfresh Christ et al. (2018) extract statistical features from time series. We re-used the same image and text encoder as the main experiments in the paper for the other modalities. Note that tsfresh is not contrastive-learning encoder but statistical feature extractor, hence also demonstrated CSA's ability to adapt any form of encoder.
Results: The classification task is similar to that one of ImageNet. We calculated the AUC of multiple classification tasks (an alphabet each) and showed the average AUC under the "ovr (one- vs-rest)" setting. From Table 5 we see that CSA constantly outperforms ASIF by 0.1x of the performance. Notably,
the first in the community to conduct multimodal classification with multivariate time series, so there are no comparable basclines.
Method
Modality
Modality image
AUC (one-vs-rest)
time series
0.50
ASIF
time series
text
0.50
time series
image
0.58
CSA
time series
text
0.54
Table 5: Multimodal time series classification: We further demonstrated CSA's performance on multimodal time series classification, which is first in the community.
18

---

Published as a conference paper at ICLR 2025
30
::NN dimension 2
20
:SN dimensian
-20
-20
-30
-40
-20
20
300
-20
-10
10
20
t-SNE dimension
t-SNE dimension
(a) I-SNE embeddings of CLIP.
(b) I-SNE embeddings of CSA.
Figure 12: t-SNE visualization embeddings: (a) shows the t-SNE visualization CLIP image embed- dings, and (b) shows the t-SNE visualization of CSA's embeddings. Both exhibit strong clustering of embed- dings, with colors representing their respective classes.
H VISUALIZATION OF CSA EMBEDDINGS
In Figure 12] showed the t-SNE van der Maaten & Hinton (0088) visualization of ImageNet embeddings. We selected 20 classes out of 1000 classes. Both CLIP and CSA have strong clustering of embeddings, thus verifying that they have similar performance on image classification.
LINEAR CLASSIFICATION OF EMBEDDINGS
Classification accuracy
1.U
0.8
0.6
CSA (ours)
0.4
CLIP
0.2
CSA Linear SVM
CLIP + Linear SVM
0.0
5000
10000 15000 20000 25000 30000 35000
Amount of training data
Figure 13: Classification SVM with ImageNet embeddings: Trained linear SVMs enhance classification performance, providing a 10% improvement over zero-shot cross-modal classification. Additionally, CLIP with SVM outperforms CSA with SVM by 5%.
We used linear support vector machines (SVMs) classify CLIP's and CSA's image embeddings. Note that linear classification on image embeddings is fundamentally different from cross-modal classification with text embeddings. This experiment has nothing to do with cross-modal feature space but provides insight into the clustering of the unimodal embeddings of CLIP and CSA. The results are shown in Figure [3] While CSA outperforms CLIP with SVM when limited to 5k training samples, CLIP with SVM achieves 5% higher classification performance than CSA with SVM when trained on 35k samples, and 10% higher than CLIP without SVM
J ABLATION STUDY ON ENCODER AWETERAAACCCEEEE ARCHITECTURES
To further ablate the effect of CSA from the disparity of the unimodal encoder's architecture the multimodal one's. We ran CSA with two different CLIP encoders with the same architecture but different train sets, hence having different feature spaces.
19

---

Published as a conference paper at ICLR 2025
Classification accuracy
0.8
X
0.0
8.1
0.4
CSA (ours)
+
CSA lours). AUC-0.70
CLIP AUC-0.00
ASIF
0.0
15000
20000
25000
30000
35000
X 1Lwin
Amount of training data
(a) ImageNet classification.
(b) Detecting mislaheled ImageNet images.
Figure 14: CSA enhances performance across different encoder architectures: (a) and (b) demonstrate that CSA's success in image classification mislabeled data detection due method itself, not architecture of the unimodal encoders.
Setting: All encoders are from the OpenCLIP library, and we choose 2 CLIP encoders with the same backbone (ViT-L/14) and trained on DataComp-IB Gadre et al. (2023) and WIT (Srinivasan et al. (2021). respectively. The latter is the original model from OpenAl and the text encoder in the ablation study. Note that the models used here are smaller than the used in the main paper (ViT- G/14) as there are no multiple ViT-G/14 CLIP models, and the results are not directly comparable. The size of the train set here is 35,000 for detecting mislabeled images.
We used the two encoders to re-run the classification and mislabel detection of ImageNet. We show the results in Figure [4 We observe that CSA constantly outperforms CLIP even with the same encoder architecture, which justifies that the success of CSA is due to its method rather than the encoder architectures of the unimodal encoders.
Results:
20


Reviews:
[{"review_id": "ue2ulqHQ1z", "reviewer": "Reviewer_Wv4t", "summary": "The paper proposes a new approach called Canonical Similarity Analysis (CSA) that addresses the challenge of training multimodal encoders, like CLIP, which typically require large amounts of paired multimodal data. CSA achieves efficient multimodal mapping by using two unimodal encoders, thereby reducing data requirements significantly. CSA utilizes canonical correlation analysis to map unimodal features into a multimodal space and introduces a weighted cosine similarity function to replicate multimodal similarity scores. Experiments across tasks such as image classification, misinformation detection, and cross-modal retrieval demonstrate CSA’s data efficiency and robustness, especially in scenarios with limited or noisy data.", "questions": "The reviewer will consider increasing the score if the authors can address the weaknesses mentioned above, most likely empirically.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1: (**Data Efficiency and Good Performance**) CSA requires significantly fewer multimodal data pairs by relying on unimodal encoders, which could benefit researchers constrained by data or computational resources. It needs as little as 35,000 image-text pairs to match the performance of CLIP on ImageNet, which is especially notable.\n\nS2: (**Computational Simplicity**) One of the most accessible aspects of CSA is its ability to function effectively without requiring GPU-intensive training, which is a significant advantage for researchers with limited computational resources. \n\nS3: (**Theoretical Analysis**)The authors include a theoretical analysis for CSA using canonical correlation analysis.", "weaknesses": "W1: (**Hyperparameter Sensitivity and Limited Justification for $s$ Selection**)\nThe choice of the hyperparameter $s$ in the canonical similarity metric (Section 4.2) lacks comprehensive justification. While the authors discuss a trade-off in feature distinguishability based on $s$, a more detailed sensitivity analysis showing how varying $s$ affects downstream performance across tasks would add clarity, as the framework is quite sensitive to $s$ indicated by content in Table 3. Table 3 provides some insights, but a broader, task-specific evaluation could make the effects of this parameter choice clearer and improve reproducibility for other researchers.\n\n\nW2: (**Weakness on Number of Modalities Supported**) While CSA shows promising results with image-text pairs and briefly explores audio-text applications in the appendix, the experimental evaluation of additional modalities remains limited and lacks depth. The audio experiments, while mentioned, do not provide a strong enough demonstration of CSA’s effectiveness beyond the main image-text modality. This limited exploration weakens CSA’s claim of generalizability across modalities and would benefit from more robust testing on diverse modality pairs to strengthen its applicability in multimodal contexts.\n\n\nW3: (**Scalability Concerns**) While CSA requires fewer training samples to match performance on certain benchmarks (e.g., ImageNet classification with 35,000 image-text pairs), some datasets or applications may still demand larger sample sizes to achieve comparable performance, especially those with more complex or diverse data structures. For instance, datasets like OpenImages (available at https://storage.googleapis.com/openimages/web/index.html), which contain extensive variety in both images and captions and a wide array of object categories, may require larger data samples for CSA to perform effectively. In such cases, the scalability limitations of CSA's matrix decomposition approach become more evident, as the cubic complexity of these operations could lead to significant computational strain on larger datasets. More experimental validation on these varied datasets would clarify the scalability and generalizability of CSA across a broader range of applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NMBjDsN66e", "reviewer": "Reviewer_Zosg", "summary": "The paper proposes CSA: Canonical Similarity Analysis method to train a multimodal encoder from two independent unimodal encoders with limited data. Unlike traditional methods that train both encoders on multimodal data, CSA only requires the unimodal encoders to generate embeddings, avoiding the need to train them on multimodal pairs.\n\nThe authors show that their CSA method matches or beats CLIP and the baseline ASIF on image classification on ImageNet and LeafySpurge datasets. However, the authors’ method falls short of CLIP’s performance on the cross-modal retrieval task. Their experiments also show that CSA outperforms the baselines when detecting mislabeled ImageNet images, and detecting misinformative captions.\n\nThe paper concludes that CSA is a robust method to replicate CLIP similarity scores using two pre-trained unimodal encoders and limited data.", "questions": "1. In Figure 4, does the smaller subset of ImageNet (5k images) include all of the mislabeled images from ImageNet? A statistic showing the percentage of mislabeled images in the training set for Figures 4(a) and 4(b) would offer more insight into these results.\n2. Why did the authors choose not to fine-tune CLIP on ImageNet? Was there some intuition behind this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well motivated: Finding data-efficient ways to train multimodal models using existing pretrained unimodal encoders is an important research direction.\n- The method used in the paper appears novel, with only one other related work (ASIF) that uses independent unimodal encoders to project embeddings onto a multimodal representational space.\n- The proposed approach (CSA) beats or matches CLIP’s performance in image classification tasks using significantly lesser data.\n- The paper’s experiments also show that CSA is more robust to mislabeled images in the image classification task. CSA is also slightly more capable in detecting misinformative captions compared to CLIP and other baselines. Finally, the authors show that CSA is more robust to noisy data compared to the previous baseline ASIF.\n- Overall, this study could pave the way for more advancements in this space of adapting independent unimodal encoders to multimodal models.", "weaknesses": "- In Figure 3, CSA was only trained on in-distribution image-caption pairs. This may lead to an unfair comparison to CLIP, as the CSA training has seen the ImageNet/Leafy Spurge distribution that its being tested on during the training process. CLIP’s rise to fame is due to its general zero-shot capabilities. The zero-shot capabilities of CSA are not fully evaluated in this paper.\n    - A fairer comparison might be to fine-tune CLIP on ImageNet/Leafy Spurge. This could be done by training the last projection layer of CLIP, analogous to keeping the unimodal encoders frozen in CSA training.\n    - Additionally, including out-of-distribution datasets would give a clearer view of CSAs true zero-shot capabilities: An example could be to train CSA on MS COCO and then evaluate the performance on ImageNet / Leafy Spurge.\n\n- Similarly, in Figure 4, comparing CLIP to CSA may not be entirely fair. Although CLIP likely encountered ImageNet images in training, they represent a very small fraction of its large dataset. A fairer comparison could be to fine-tune CLIP (again, perhaps just the  projection layer) on ImageNet images, including those with mislabeled data, before proceeding with this analysis (See Question 2).\n\n- The paper's conclusion suggests that CSA outperforms CLIP in cross-modal retrieval (\"CSA shows competitive performance compared to CLIP models in image classification, mislabeled data detection, **cross-modality retrieval**, and misinformation detection tasks with\nlimited data.\"). However, this is misleading, as Table 2 shows that CSA underperforms CLIP in this task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6oDjD4AbnO", "reviewer": "Reviewer_fPRX", "summary": "This paper introduces a canonical similarity analysis (CSA) framework, a novel approach that projects two distinct unimodal feature spaces from pre-trained encoders into a unified multimodal feature space. The CSA has great data efficiency and discovers the inherent trade-offs of informative embeddings and distinguishing data. The extensive experiments show that the CSA achieves competitive performances against CLIP models across a range of tasks, including image classification, mislabeled data detection, cross-modality retrieval, and misinformation detection, even with a limited dataset.", "questions": "1. The visualization experiments can show the interpretability of multi-modal learning, I wonder to know the visualization results of embedding spaces.\n2. Although the CSA does not require GPUs to optimize the fitting matrix, I wonder to know the run-time cost of the optimization process on different datasets and feature dimensions. Is it faster than the training a traditional network?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes the canonical similarity analysis (CSA) framework, which can replicate the CLIP multi-modal model. It just uses two unimodal encoders but demands much less computation cost and related data.\n2. This paper provides the theoretical analysis on the trade-off of obtaining informative embeddings and distinguishing multimodal data, considering various hyperparameters of CSA.\n3. The extensive experiments on various downstream tasks (such as image classification, cross-modal retrieval, and misinformative caption detection) show that CSA outperforms the traditional CLIP model, while requiring much fewer paired multimodal data and fewer unimodal data.", "weaknesses": "1. This paper proposes a post-tuning mapping framework on the unimodal features, which would compute the matrix optimization without training any encoders. Hence the performances heavily rely on the choices of visual and textual encoders. The experiments part only shows the one encoder situation (gtr + dino). More model encoders analysis for CSA are needed.\n2. In the performance comparisons, the ASIF is the only fair baseline method, which lacks of persuasion. It is important to add more comparative experiments for more related methods, e.g., some prompt tuning and adapter tuning work (PEFT methods). What’s more, when compare CSA with CLIP (e.g., Flickr in Tab. 2, ImageNet in Fig. 3), the CSA is the fine-tuned model on the specific dataset, but the CLIP is the zero-shot model without fine-tuning. Such experimental comparison is unreasonable. The CSA should fine-tuning on the ImageNet and then test on the Flickr.\n3. The feature dimensions of $q$ and $r$ are the important hyper-parameters for CSA, and the Tab.1 also shows the different dimension choices on different datasets, hence the related ablation study is missing in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pTE0H5B2ti", "reviewer": "Reviewer_JqjS", "summary": "A new similarity calculation strategy, CSA, based on Canonical Correlation Analysis, is proposed to replace the correlation matrix in CLIP for aligning images and text modalities. CSA allows better alignment of features encoded by unimodal encoders for the same samples without requiring as many training samples as CLIP. A theoretical explanation of CSA is provided. Experiments are conducted on a large number of datasets, all of which outperform the baseline model ASIF.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It only requires using a pre-trained unimodal encoder along with CSA, training on a small number of sample pairs (such as a single image and its corresponding caption) to achieve alignment between two modalities, greatly saving computational resources.\n2. In image classification tasks, it achieves results comparable to CLIP using fewer resources.\n3. Experimental results significantly outperform the baseline model ASIF.", "weaknesses": "1. The theoretical discussion of CSA lacks detailed explanation for Equation (9), and the connection to contrastive learning is not clearly established, indicating a lack of novelity in CSA.\n2. In comparisons with CLIP for image classification tasks, only a limited amount of training data was used, with no comparisons made on larger-scale training datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hGshXrS9d0", "reviewer": "Reviewer_5r8m", "summary": "The paper proposes a novel method called Canonical Similarity Analysis (CSA) to improve data efficiency when mapping unimodal features (e.g., from image or text) into multimodal feature spaces, effectively replicating models like CLIP but requiring significantly fewer data using pretrained unimodal models. The method leverages two pretrained unimodal encoders and applies Canonical Correlation Analysis (CCA) to align the features using a small portion of the training set. CSA does not involve training neural networks, only solving a matrix decomposition for feature mapping. The authors demonstrate its effectiveness outperforming a pretrained CLIP in some cases with far fewer multimodal data.", "questions": "- Q1. Have you considered evaluating CSA using the same backbone as CLIP (e.g. using the same image and text encoders but pre-trained on two different datasets from OpenCLIP)? This would empirically strengthen your hypothesis and demonstrate the effectiveness of your method.\n- Q2. Just out of curiosity, how scalable is CSA when handling larger training datasets? How much could it benefit from optimizing the matrix decomposition on a large-scale multimodal dataset?\n- Q3. Is it possible to possible to solve CSA on a different dataset from the downstream one? It would be interesting to see whether this improves performance in tasks like cross-modal retrieval, where CSA currently underperforms compared to CLIP.\n- Q4. Could the authors better explain why the Leafy Spurge dataset was chosen as an evaluation benchmark, and how its characteristics make it particularly suitable for demonstrating CSA's capabilities?\n\nI would be happy to improve my rating if the authors address my concerns, particularly regarding the fairness of the current comparison.\n\n**Minor observations**:\n- Error in Table I. In the supplementary material, it is stated that the CLIP model used is the ViT-bigG-14 trained on LAION-2B. However, in Table I, CLIP is attributed to 12B pairs of training data.\n- Error in Table I. Moreover CLIP ViT-bigG-14 is 2.5B of params in total.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper is well-organized, making it easy to follow the proposed approach and experimental setup. \n- A key strength of CSA is its remarkable data efficiency, achieving effective mapping and strong results with significantly less training data compared to the baselines. \n- Additionally, the method demonstrates robustness in handling noisy and limited multimodal data, which is essential in real-world applications where large correctly labeled datasets are often scarce. \n- Finally, the theoretical analysis offers valuable insights into the trade-offs between reducing noise and preserving meaningful similarity scores, further enhancing the paper's contribution.", "weaknesses": "- One major issue is that CSA uses different backbones, such as DINOv2-Giant and a GTR-t5-large as unimodal encoders (lines 715-716), compared to the multimodal baseline using a CLIP ViT-bigG/14 pretrained on LAION-2B (line 712). This creates an unfair comparison and makes hard to objectively state the efficacy of the approach. \n- Additionally, the paper's performance on cross-modal retrieval tasks, particularly in text-to-image retrieval, is noticeably weaker than CLIP, which limits its impact in this commonly used task. \n- Moreover, the greatest weakness is that CSA requires a small portion of training data (i.e. 35k pairs on ImageNet) to solve the matrix decomposition for the features mapping, but all the evaluations are compared to zero-shot CLIP (not fine-tuned on the same portion of training data), which leads to an unfair advantage. Evaluation (e.g Figure 3(a)/3(b) and Table 2(a)/2(b)) should include few-shots adaptation baselines (e.g CoOp [1], CLIP-Adapter [2], etc..) or at least the standard CLIP linear evaluation protocol [3] for a fair comparison with the multimodal baseline. Note that using CSA on 800 pairs of Leafy Spurge could be the main reason for its improved performance against zero-shot CLIP. This is because this data is so different from the training distribution that merely adapting to it leads to the observed improvements.\n\n[1] Learning to Prompt for Vision-Language Models (https://arxiv.org/abs/2109.01134)\n\n[2] CLIP-Adapter: Better Vision-Language Models with Feature Adapters (https://arxiv.org/abs/2110.04544)\n\n[3] Learning Transferable Visual Models From Natural Language Supervision (https://arxiv.org/abs/2103.00020)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)