CSA: Data-efficient Mapping of Unimodal Features to Multimodal Features

## Abstract

Abstract Multimodal encoders like CLIP excel in tasks such as zero-shot image classification and cross-modal retrieval. However, they require excessive training data.
We propose canonical similarity analysis (CSA), which uses two unimodal encoders to replicate multimodal encoders using limited data. \name maps unimodal features into a multimodal space, using a new similarity score to retain only the multimodal information. \name only involves the inference of unimodal encoders and a cubic-complexity matrix decomposition, eliminating the need for extensive GPU-based model training.
Experiments show that \name outperforms CLIP while requiring 300 , 000 × 300,000\times 300 , 000 × fewer multimodal data pairs and 6 × 6\times 6 × fewer unimodal data for ImageNet classification and misinformative news captions detection. \name surpasses the state-of-the-art method to map unimodal features to multimodal features.
We also demonstrate the ability of \name with modalities beyond image and text, paving the way for future modality pairs with limited paired multimodal data but abundant unpaired unimodal data, such as lidar and text.

## 1 Introduction

Multimodal encoders like CLIP (Radford et al., ) excel in various zero-shot multimodal tasks, such as image classification and cross-modal retrieval.
Despite the huge success, they demand excessive training data.
For example, OpenAI trained the original CLIP model on $400$ million image-text pairs using $592$ V100 GPUs, and the training size of the new CLIP models (Ilharco et al., ) has increased to $12$ billion image-text pairs since then.
In addition, more data do not guarantee better performance.
CLIP relies on Internet data, which are often of poor quality (Sharma et al., ).
Incorrectly captioned data may lead to specific failure modes in similar instances (Northcutt et al., ; Vasudevan et al., ).
In this work, we focus on learning a multimodal encoder with limited data that is robust to noisy data.

Our conjecture is that unimodal encoders such as DINO (Oquab et al., ) and GTR (Ni et al., ) can serve as the building blocks of multimodal encoders.
Unimodal encoders only need unimodal data, which are easier to obtain than paired multimodal data. They also require significantly less data than multimodal models, and advanced unimodal encoders are already well-developed.
We distill the unimodal knowledge in these models to accelerate the learning of multimodal encoders.
We map their unimodal features into a multimodal feature space, similar to how CLIP encodes images and text jointly to a shared feature space.
In addition, this process should be data-efficient.

We propose canonical similarity analysis (\name) to verify our conjecture.
It replicates a multimodal encoder such as CLIP with two unimodal encoders, as shown in Figure 1.
\namemaps features from unimodal encoders to a multimodal space.
It removes redundant information from unimodal features and employs a novel similarity function to replicate CLIP similarity score for multimodal data, e.g. images and captions.
\nameoperates by inference of unimodal encoders and solving an optimization problem, essentially a matrix decomposition without training of neural networks.
\namesupports all zero-shot tasks that CLIP is capable of while requiring significantly fewer training data.

Figure: Figure 1: Canonical similarity analysis (\name) replicates the CLIP multimodal similarity scores with two unimodal encoders. \nameuses two unimodal encoders to encode data to unimodal features. Then, it projects the unimodal features to a joint multimodal feature space. The weighted cosine similarity in this feature space replicates the CLIP similarity, enabling various downstream tasks, e.g., cross-modal retrieval. \nameis a robust and data-efficient method that learns even when images and text are misaligned.
Refer to caption: x1.png

Contributions.
Our contributions are threefold:
(1) We propose \namewhich replicates the CLIP model using two unimodal encoders while demanding less computation and data (Section 4).
(2) Our theoretical analysis characterizes the trade-off of obtaining informative embeddings and distinguishing
multimodal data pairs, considering various hyperparameters of \name(Section 5).
(3) We tested \nameon tasks such as image classification, cross-modal retrieval, and misinformative captions detection (Section 6).
The experimental results show that \nameoutperforms or matches CLIP while requiring $300,000\times$ fewer paired multimodal data and $6\times$ fewer unimodal data.
\nameoutperforms the state-of-the-art method that employs unimodal models with the same amount of training data as $18\%$ in ImageNet classification and $23\%$ in image-to-text retrieval.

works across all modalities beyond images and text.
We additionally demonstrate its capability with audio and text, paving the way for new modality pairs, such as lidar and text Yang et al. ().
In the future, we envision that new modality pairs will have limited paired multimodal data but sufficient unimodal data, where \namecan efficiently map unimodal features to multimodal features.

## 2 Related Works

Unimodal Encoders.
Unimodal encoders encode unimodal data to fixed-dimensional features.
In computer vision, encoders, e.g., CosPlace (Berton et al., ), ViT (Dosovitskiy et al., ), and Dino (Caron et al., ; Oquab et al., ), are trained with augmented data with random cropping and blurring with contrastive loss (van den Oord et al., ).
In natural language processing, one common data augmentation method is to randomly mask words or tokens, used in the universal sentence encoder (Cer et al., ), Bert (Devlin et al., ), and GTR (Ni et al., ).

Multimodal Encoders.
Multimodal encoders project multimodal data points into a feature space shared between modalities. Thus, multimodal features enable training for downstream models (Shridhar et al., ; Li et al., ; Goel & Narasimhan, ).
Another common usage is to directly take the inner product of features of data from different modalities, e.g., an image and a sentence, as a similarity metric.
This metric enables downstream tasks such as zero-shot image classification and cross-modal data retrieval.
The CLIP model (Radford et al., ) is the milestone of multimodal encoders, which bridges the gap between language and vision.
CLIP inspires encoders for other modalities, such as audio and inertial measurement units. Such encoders include ImageBind (Girdhar et al., ), AudioCLIP (Guzhov et al., ) and CLAP (Wu* et al., ).
Previous work also studied cross-modal retrieval on multi-label images (Ranjan et al., ), while ours focuses on a more generic multimodal setting with pre-trained unimodal encoders.
Although ImageBind encodes $6$ modalities, it lacks training data for all possible modality pairs, resulting in suboptimal performance for pairs such as audio and inertial measurement units.

Neural Networks with CCA.
Previous works have shown that training losses inspired by canonical correlation analysis (CCA) (Hardoon et al., ) are useful in multimodal or multi-distribution tasks (Sun et al., ; Lyu & Fu, , ).
Although taking a crucial step of introducing CCA to deep learning, these works focus on training deep neural networks from scratch instead of using existing foundation encoder models, as in this work.
Other works use techniques similar to CCA, such as principal component analysis (PCA), to process the output features of encoders (Li et al., ; Omama et al., ).
In this work, we focus on using CCA to map unimodal features to a multimodal feature space without any model training, which significantly reduces the size of the required training dataset.

Combining multiple unimodal Encoders.
The closest work to ours is ASIF (Norelli et al., ), which also uses multiple unimodal encoders to bridge the multimodal gap.
ASIF calculates the similarity of a pair of multimodal data based on the unimodal similarity of the given data pair and a pre-selected, multimodal anchor dataset, similar to Moschella et al. ().
It creates a linear kernel space by the anchor set.
However, ASIF demands a huge anchor set to achieve similar performance to CLIP, while our proposed method, \name, constantly outperforms ASIF in all tasks.

## 3 Problem Formulation

Given two sets of data $X^{1}=\{x_{1}^{1},x_{2}^{1},...,x_{N}^{1}\}$, $X^{2}=\{x_{1}^{2},x_{2}^{2},...,x_{N}^{2}\}$
from two modalities $m^{1},m^{2}$ (such as vision and language),
we want to find two feature extractors (encoders) $\mathbf{E}^{1},\mathbf{E}^{2}$ that map the data to the same-dimensional feature space.
The encoders generated feature pairs $\mathbf{E}^{1}(x_{i}^{1}),\mathbf{E}^{2}(x_{i}^{2})$ are close, and the feature pairs from different pairs of data $\mathbf{E}^{1}(x_{i}^{1}),\mathbf{E}^{2}(x_{j}^{2})$ are far.

Note that the two sets of data are paired, which means that they are sophisticatedly related (such as an image and its corresponding caption), although of different modalities.
Previous works trained multimodal encoder pairs to achieve this goal by minimizing the following loss:

$$ $\displaystyle\mathcal{L}(X^{1},X^{2};$ $\displaystyle\mathbf{E}^{1},\mathbf{E}^{2}):=\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\text{(Multimodal Contrastive Loss)}$ (1) $\displaystyle\frac{-1}{2N}\sum_{i=1}^{N}\left[2(\mathbf{E}^{1}(x_{i}^{1}))^{ \top}(\mathbf{E}^{2}(x_{i}^{2})))-\sum_{j\neq i}\frac{(\mathbf{E}^{1}(x_{i}^{1 })+\mathbf{E}^{2}(x_{i}^{2}))^{\top}(\mathbf{E}^{1}(x_{j}^{1})+\mathbf{E}^{2}( x_{j}^{2}))}{2N-2}\right],$ $$

where the first term encourages similar features for the same pair of data, and the second term encourages the opposite.
Variants of this loss are commonly used in contrastive learning (van den Oord et al., ), such as CLIP (Radford et al., ).

From Unimodal to Multimodal.
Training a multimodal encoder to learn a multimodal feature space is time-consuming and data-inefficient.
The original CLIP model learns from Internet-scale datasets ($400$ million images) and requires $500$ A100 Nvidia GPUs.
Instead of training end-to-end multimodal encoders, we leverage existing unimodal encoders to accelerate the multimodal learning process.
That is, given unimodal encoders that encode data into unimodal feature spaces, we directly find two mapping functions that map each unimodal feature space to a multimodal space.

Mathematically, we use two unimodal encoders $\mathbf{E}^{1},\mathbf{E}^{2}$ that capture only unimodal latent information and individually encode data $x_{i}^{1},x_{i}^{2}$ into features $\hat{z}_{i}^{1},\hat{z}_{i}^{2}\in\mathbb{R}^{q^{1}},\mathbb{R}^{q^{2}}$.
Because these features $\hat{z}_{1}^{1},\hat{z}_{1}^{2}$ come from different encoders, their dimensions need not be the same ($q^{1}\neq q^{2}$), which complicates the discovery of the mapping functions.
The dimensions of the features are significantly smaller than those of the raw data $x_{i}$.
Hence, we utilize unimodal encoders to significantly reduce the dimension of the data and the complexity of the problem.
Our ultimate goal is to efficiently find two mapping functions $\mathbf{A},\mathbf{B}$ that map the feature spaces to a multimodal one while preserving the contrastiveness property characterized in Equation 1.

## 4 Canonical Similarity Analysis ( \name )

The problem in Section 3 is easier to state than to solve.
First, unimodal encoders have completely different architectures (e.g., transformers vs. U-net), losses, and output dimensions.
Second, mapping data from various-dimensional feature spaces to a multimodal one means that we need to discard latent information from the higher-dimensional space.
The challenge lies in determining what information to discard to maintain the contrastiveness property.
Also, we do not want to preserve all information as there might be some modality-specific information that we wish not to preserve through the mapping, such as words that cannot be expressed in images or punctuation.

We thus propose canonical similarity analysis (\name).
We observe that all downstream tasks use cosine similarity to obtain the similarity of multimodal data points.
It inspires us to use correlation coefficients, the cosine similarity between centered vectors, to approximate this similarity and build the mapping functions from unimodality to multimodality.
Precisely, we find the pairs of bases in each unimodal feature space that maximize the correlation coefficients. We use CCA to find such bases.
Then, instead of directly using an inner product, we use a weighted cosine similarity score to evaluate the similarity of the multimodal data. We discard information from low-correlated bases, as they might not be relevant to the multimodal feature space, and we weight the contributions of the bases to the similarity score based on their correlation coefficients.

### 4.1 Canonical Correlation Analysis

Recall the previous notation of two sets of data $X^{1}=\{x_{1}^{1},x_{2}^{1},...,x_{N}^{1}\}$, $X^{2}=\{x_{1}^{2},x_{2}^{2},...,x_{N}^{2}\}$.
We use two unimodal encoders $\mathbf{E}^{1},\mathbf{E}^{2}$ to encode and normalize the data into zero-meaned latent features $\{\hat{z}_{i}^{1},\hat{z}_{i}^{2}\}_{i=1}^{N}$.
To find the bases that maximize the correlation coefficients, we use CCA to map the two feature spaces to a multimodal feature space of dimension $r=\min(q^{1},q^{2})$:

$$ $\displaystyle\mathbf{A}^{*},\mathbf{B}^{*}=\operatorname*{arg\,max}_{\mathbf{A }\in\mathbb{R}^{r\times q^{1}},~{}\mathbf{B}\in\mathbb{R}^{r\times q^{2}}}$ $\displaystyle\mathrm{Tr}\left(\mathbf{A}\hat{Z}^{1}\left(\mathbf{B}\hat{Z}^{2} \right)^{\top}\right)\quad\quad\quad\quad\quad\text{(Formulation of CCA)}$ (2) s.t. $\displaystyle(\mathbf{A}\hat{Z}^{1})(\mathbf{A}\hat{Z}^{1})^{\top}=(\mathbf{B} \hat{Z}^{2})(\mathbf{B}\hat{Z}^{2})^{\top}=\mathbf{I}_{r},$ $$

where $\hat{Z}^{1}\in\mathbb{R}^{q^{1}\times N},\hat{Z}^{2}\in\mathbb{R}^{q^{2}\times
N}$ are the matrices consisting of column feature vectors $\hat{z}$, $\mathbf{I}_{r}\in\mathbb{R}^{r\times r}$ is the identity matrix, and $\mathrm{Tr}$ is the trace function.
The objective of Equation 2 can be viewed as the sum of the correlation coefficients of the bases, and the identity constraints normalize the vectors so that covariance becomes correlation.
In summary, CCA finds the bases of the two given matrices such that the projected vectors have the highest correlation coefficients $\rho_{i}$ for $i=1,...,r$.

Solving CCA is similar to principal component analysis (PCA), which is essentially a singular value decomposition (SVD) of time complexity $O(q^{1}q^{2}r)$. Therefore, it possesses the beneficial property of removing noise from the data, which we later detailed in the theoretical analysis.
Weenink () gives the analytical solution:

$$ $\mathbf{A}^{*},\mathbf{B}^{*}=U\Sigma_{\hat{z}^{1}}^{-1/2},V\Sigma_{\hat{z}^{2 }}^{-1/2},~{}~{}~{}~{}\text{(Analytical Solution of CCA)}$ (3) $$

where $U^{\top}PV=\Sigma_{\hat{z}^{1}}^{-1/2}\hat{Z}^{1}\hat{Z}^{2\top}\Sigma_{\hat{z
}^{2}}^{-1/2}\in\mathbb{R}^{q^{1}\times q^{2}}$ is the SVD matrices of the rank-$r$ matrix.
The correlation coefficients $\rho$ of Equation 2 are the diagonal entries of $P$ in descending order $\rho_{1}\geq\rho_{2}\geq...\geq\rho_{r}$ like the descending principal components of PCA.

### 4.2 Canonical Similarity Analysis

We now define \name.
For any pair of multimodal data $\{x_{j}^{1},x_{j}^{2}\}$, with slight abuse of the notation of original data and features, we define the similarity as:

$$ $\mathcal{S}(x_{j}^{1},x_{j}^{2};s)=\mathcal{S}(\hat{z}_{j}^{1},\hat{z}_{j}^{2} ;s)=\frac{\sum_{i=1}^{s}\rho_{i}(\mathbf{A}^{*}\hat{z}_{j}^{1})_{i}(\mathbf{B} ^{*}\hat{z}_{j}^{2})_{i}}{\|(\mathbf{A}^{*}\hat{z}_{j}^{1})_{1:s}\|_{2}\|( \mathbf{B}^{*}\hat{z}_{j}^{2})_{1:s}\|_{2}},\quad\quad\text{(Canonical Similarity)}$ (4) $$

where $(\cdot)_{1:s}$ denotes the $1$st to $s$-th row vector, $(\cdot)_{i}$ denotes the $i$-th entry, and $s$ is a pre-defined hyperparameter.
Note that Equation 4 is the weighted cosine similarity of the first $s$ dimensions of the transformed data matrix.
The weights determine how much each dimension influences the overall similarity score, depending on their level of correlation.
We chose the hyperparameter $s$ so that \namepreserves the information of dimensions with correlations greater than $\rho_{s}$.
As stated earlier, the feature space might contain modal-specific information that cannot be mapped to the other modality.
Thus, we only take into account meaningful bases.
We choose CCA with linear correlation instead of complicated kernel CCA because most contrastive learning loss functions only contain the linear inner product of the feature vectors, which is also the case in Equation 2.

### 4.3 Training and Inference

Similar to standard machine learning models, \nameconsists of both training and inference phases.
We split the multimodal data into training and test sets, just like in standard machine learning.
\nameperforms all optimizations on the training set, and the test set is not known a priori and is used only for evaluation.
During the training phase, or rather called the fitting phase, \nameobtains the training set of encoded features from the unimodal encoders and then solves Equation 2.
Note that \namedoes not train any encoders, so it does not require GPUs.
The resulting correlation coefficients $\rho$ determines the value of $s$ in Equation 4:

$$ $s=\operatorname*{arg\,min}_{i}\rho_{i}\quad\text{s.t.}\;\rho_{i}\geq\mathrm{ const}.$ (5) $$

Note that $\mathrm{const}$ here is an empirical constant threshold to optimize performance in our experience.
Finally, for any multimodal downstream task, \nameuses Equation 4 to evaluate the similarity between any pair of multimodal data.
We later show the experimental results of \nameand compare the results with the state-of-the-art methods.

## 5 Theoretical Analysis of CSA

In this section, we characterize the trade-off of obtaining informative embeddings and distinguishing multimodal data pairs caused by selecting the appropriate $s$ in Equation 4.
We start with the linear setting, but the conclusion holds for a more general setting.

Assumptions.
We follow the assumption of data generation per Joshi et al. () and analyze the effect of \namebased on the theoretical results of contrative losses (Ji et al., ; Tian, ; Nakada et al., ).
Suppose that the $N$ pairs of data $\{x_{i}^{1},x_{i}^{2}\}_{i=1}^{N}$ from two modalities $m^{1},m^{2}$ are generated as follows:

$$ $x_{i}^{1}=\mathcal{T}^{{1}}z_{i}+\epsilon_{i}^{1},\quad x_{i}^{2}=\mathcal{T}^ {{2}}z_{i}+\epsilon_{i}^{2}.$ (6) $$

Here, $z_{i}\in\mathbb{R}^{q}$ is the shared unobservable latent feature of the two modalities, and $\mathcal{T}^{1}\in\mathbb{R}^{p^{1}\times q},\mathcal{T}^{2}\in\mathbb{R}^{p^{
2}\times q}$ are the linear mapping functions of the latent vectors to the observable data.
Without loss of generality, we assume that both the latent feature vectors and noise for each modality are independent and identically distributed (i.i.d.) samples from any distribution.

Optimal Linear Encoders of Contrastive Loss.
Previous work (Ji et al., ) gives the analytical solution of a unimodal linear encoder $\mathbf{E}^{i}$ trained on linear contrastive loss with norm regularization:

$$ $\displaystyle\mathcal{L}(X^{k},X^{{}^{\prime}k};\mathbf{E}^{k}):=\frac{-1}{2N} \sum_{i=1}^{N}\left[2(\mathbf{E}^{k}x_{i}^{k})^{\top}(\mathbf{E}^{k}x_{i}^{{}^ {\prime}k})-\sum_{j\neq i}\frac{(\mathbf{E}^{k}x_{i}^{k}+\mathbf{E}^{k}x_{i}^{ {}^{\prime}k})^{\top}(\mathbf{E}^{k}x_{j}^{k}+\mathbf{E}^{k}x_{j}^{{}^{\prime} k})}{2N-2}\right]$ (7) $\displaystyle\quad\quad\quad\quad\quad\quad\quad+\frac{\lambda}{2}\|\mathbf{E} ^{k}\mathbf{E}^{k\top}\|_{F}^{2},\quad\quad\text{for }k=1,2,\quad\quad\quad \text{(Unimodal Contrastive Loss)}$ $$

where $X^{k},X^{{}^{\prime}k}$ are two sets of original and augmented (e.g., random masked, Gaussian blurred, etc.) data from the same modality, and $\lambda$ is a hyperparameter weighting the regularization term.
Note that this contrastive loss function is similar to Equation 1, but the second modality is the augmented data now.
For simplicity, we assume that all unimodal data are augmented with complementary random masking with probability $0.5$, so the optimal linear encoder is:

$$ $\mathbf{E}^{i}_{\mathrm{linear}}=\operatorname*{arg\,min}_{E\in\mathbb{R}^{q \times p^{i}}}\mathcal{L}=C\left(\sum_{i=1}^{q}u_{i}\sigma_{i}e_{i}^{\top} \right)^{\top},~{}~{}\text{for }i=1,2.$ (8) $$

In Equation 8, $C>0$ is a positive constant related to $\lambda$ in Equation 7.
$e_{i}$ is the $i$-th standard vector basis, and $\sigma_{i}$ is the $i$-th largest eigenvalue of matrix:

$$ $\textrm{OffDiag}(X^{i}X^{i\top})-\frac{1}{N-1}X^{i}(\mathbf{1}-\mathbf{I}_{N}) X^{i\top},$ (9) $$

where OffDiag denotes the function that makes all diagonal entries 0 0, and $\mathbf{1}$ is a square matrix of ones.
From Equation 3, 6, and 8, the solution of \namein linear setting is:

$$ $\mathbf{A}^{*}\hat{z}_{i}^{1}=\mathbf{A}^{*}\mathbf{E}^{1}_{\mathrm{linear}}x_ {i}^{1}=\mathbf{A}^{*}\mathbf{E}^{1}_{\mathrm{linear}}(\mathcal{T}^{{1}}z_{i}+ \epsilon_{i}^{1}),~{}~{}\mathbf{B}^{*}\hat{z}_{i}^{2}=\mathbf{B}^{*}\mathbf{E} ^{2}_{\mathrm{linear}}x_{i}^{2}=\mathbf{B}^{*}\mathbf{E}^{2}_{\mathrm{linear}} (\mathcal{T}^{{2}}z_{i}+\epsilon_{i}^{2}).$ (10) $$

Sensitivity to Noise.
The noise in Equation 6 affects the unimodal linear encoders $\mathbf{E}^{1}_{\mathrm{linear}},\mathbf{E}^{2}_{\mathrm{linear}}$ from the second term of Equation 9.
It hinders the unimodal encoders in finding the inverse of $\mathcal{T}^{1},\mathcal{T}^{2}$ but in encoding the noise.
The noise then propagates and affects the covariance matrices of $\hat{z}$ and the decomposed orthogonal matrices $U$ and $V$ in Equation 3,.
Hence, we rely on the hyperparameter $s$ to prevent noise from dominating. When $s$ increases, we include the less correlated dimensions caused by noise in the canonical similarity metric in Equation 4.

Distinguish Contrastive Data Pairs.
In Equation 7, similar sample pairs, i.e., the augmented and original data, stay close to each other, while dissimilar ones, i.e., augmented data from different data points, are far apart.
However, in the objective of CCA Equation 2, we focus only on maximizing the correlation coefficients of similar samples while ignoring dissimilar ones.

We now analyze how \namedistinguishes contrastive latent feature pairs from the two unimodal encoders.
Given two data pairs $(x_{a}^{1},x_{a}^{2})$ and $(x_{b}^{1},x_{b}^{2})$,
we assume that the predicted latent feature pairs $(\hat{z}_{a}^{1},\hat{z}_{b}^{1}),(\hat{z}_{a}^{2},\hat{z}_{b}^{2})$ are distant because the unimodal encoders are trained for it.
The distance between latent features preserved in the matrix transformations is bounded as follows:

$$ $\displaystyle\lambda_{\mathrm{min}}(\mathbf{A}^{*})\lambda_{\mathrm{min}}( \mathbf{E}^{1}_{\mathrm{linear}})\|x_{a}^{1}-x_{b}^{1}\|_{2}\leq\lambda_{ \mathrm{min}}(\mathbf{A}^{*}\mathbf{E}^{1}_{\mathrm{linear}})\|x_{a}^{1}-x_{b} ^{1}\|_{2}\leq\|\mathbf{A}^{*}(\hat{z}_{a}^{1}-\hat{z}_{b}^{1})\|_{2},$ (11) $\displaystyle\lambda_{\mathrm{min}}(\mathbf{B}^{*})\lambda_{\mathrm{min}}( \mathbf{E}^{2}_{\mathrm{linear}})\|x_{a}^{2}-x_{b}^{2}\|_{2}\leq\lambda_{ \mathrm{min}}(\mathbf{B}^{*}\mathbf{E}^{2}_{\mathrm{linear}})\|x_{a}^{2}-x_{b} ^{2}\|_{2}\leq\|\mathbf{B}^{*}(\hat{z}_{a}^{1}-\hat{z}_{b}^{1})\|_{2},$ $$

where $\lambda_{\mathrm{min}}(\cdot)$ denotes the minimum singular value, which is the opposite of the Lipschitz constant of a matrix.
From Equation 11, we can see that the distance between the \namefeatures (right-hand side) is lower bounded by the distance of the original data with a constant of minimum singular values (left-hand side).
Thus, we know how the hyperparameter $s$ selecting the dimension of \nameaffects the distance between dissimilar data pairs.
When the hyperparameter $s$ decreases, \nametakes into account fewer dimensions per Equation 4.
That is, $\lambda_{\mathrm{min}}(\mathbf{A}^{*}_{1:s})=\rho_{s}$ is effectively larger, which ultimately results in more distant features in the \namespace, and vice versa.
In summary, a smaller $s$ preserves distance in the \namespace, which distinguishes contrastive data pairs.

Trade-off of Canonical Similarity Analysis.
Previous results show that a smaller $s$ leads to less noisy but more contrastive similarity scores, which is desirable. However, $s=1$ is not desirable because it loses much information about the true latent features, and most vectors will have identical similarity scores.
In practice, we do not know the dimension of the latent feature $q$ and can only use \nameto project the encoded features of two modalities to a dimension that is less noisy, but also contains enough information about the true shared latent feature $z$.

Figure: (a) SNR and $\lambda_{\mathrm{min}}$ vs. selected dimension.
Refer to caption: x2.png

Numerical Experiments.
We validate the trade-off through numerical experiments following Equation 6.
Figure 2(a) shows that if we only account for a few dimensions ($s$ is low), the signal-to-noise ratio (SNR) of the data is then high, and the minimum singular value $\lambda_{min}$ is large, which preserves the distance between contrastive data.
Note that Figure 2(a) shows the values in decibels (dB).
Figure 2(b) compares paired data $(x_{i}^{1},x_{i}^{2})$ and randomly shuffled data $(x_{i}^{1},x_{j}^{2}),i\neq j$.
High Wilcoxon p-values mean that the two distributions of similarity scores from Equation 4 are alike.
In this case, the canonical similarity score is meaningless, and larger $s$ is more desirable.
The intrinsic trade-off in selecting $s$ is now clear.
When $s$ is too small, we do not have meaningful similarity scores that distinguish unpaired data, but when $s$ is too large, we have noisy and closer clusters of features.

## 6 Experiments

Baselines.
We compared \namewith another baseline, ASIF (Norelli et al., ).
It has the same setting as \name, which uses two unimodal encoders to calculate the similarity of a multimodal data pair. Unlike \namesolves an optimization problem, ASIF infers multimodal similarity by the unimodal distance between data points.
ASIF is the only fair baseline for \name. We trained \nameand ASIF with the same training set and unimodal encoders and evaluated all the methods in the test set.
We also compared \namewith the state-of-the-art multimodal encoder model from OpenCLIP (Ilharco et al., ; Cherti et al., ), acknowledging their incomparable training set and model parameters, shown in Table 1.
In addition to encoder models, we compared the performance of LLaVA2-13B (Touvron et al., ), a multimodal large language model, and prompted it to solve the specified tasks from the provided image and text.
The unimodal encoders of \nameand ASIF are GTR (Ni et al., ) and DINOv2 (Oquab et al., ) here.
Section A shows the detailed settings.

**Table 1: Comparison of methods: \nameutilizes two unimodal encoders to match the performance of CLIP while requiring significantly less data.**
| Method | Train images | Train text | Parameter | Feature dimension $q,r$ |
| --- | --- | --- | --- | --- |
| CLIP | $12$B | $12$B | $1.3$B | $q=1280$ |
| GTR | ✗ | $2$B | $335$M | $q=768$ |
| DINOv2 | $142$M | ✗ | $1.1$B | $q=1536$ |
| \name(ours on ImageNet) | $35$k | $1$k | ✗ | $r=700$ |
| \name(ours on Leafy Spurge) | $800$ | $2$ | ✗ | $r=250$ |
| \name(ours on Flickr30k) | $5$k | $25$k | ✗ | $r=200$ |
| \name(ours on COSMOS) | $41$k | $41$k | ✗ | $r=750$ |

Figure: (a) ImageNet.
Refer to caption: extracted/5915368/figure/imagenet/trainsize_vs_accuracy.png

**(a)**
| Method | mAP@5 | Precision@1 | Precision@5 |
| --- | --- | --- | --- |
| \name(ours) | $36.6\%$ | $59.3\%$ | $43.4\%$ |
| CLIP | $73.8\%$ | $92.9\%$ | $77.2\%$ |
| ASIF | $14.6\%$ | $25.6\%$ | $20.0\%$ |

Image Classification.
We first examined \name’s ability of image classification on ImageNet and Leafy Spurge.
All methods input an image and all possible captions “This is an image of label” and select the most similar caption as the predicted label.
In Figure 3(a), \namerequires only $35,000$ training samples to match the performance of CLIP and consistently outperforms ASIF, which needs millions of data points to achieve a similar performance to CLIP.
Since CLIP is zero-shot, its performance is regardless of the amount of data.
Leafy Spurge challenges the capability of methods with extremely limited data.
This dataset includes only $800$ training images and $100$ test images of plants with binary classes: leafy spurge and others.
The images are out-of-distribution for any encoders, as this dataset was newly published in 2024 and captured using drones.
Figure 3(b) shows that while the performance of \namefluctuates, it consistently outperforms CLIP and ASIF in this challenging scenario, demonstrating \name’s effectiveness with limited out-of-distribution data.

Table 1 shows the size of the training set and the number of parameters of the neural network encoders.
Notably, \namesurpasses CLIP by utilizing two unimodal model encoders trained on $2$ billion text samples and $142$ million images, along with only $35,000$ multimodal data points. This is $6\times$ less multimodal data and $300,000\times$ less unimodal data compared to the CLIP training set.

Cross-modal Retrieval.
We examined \nameand the other baselines on a cross-modal retrieval dataset, Flickr30k (Young et al., ), which contains $15,000$ images, each paired with $5$ textual descriptions. This dataset is common for evaluating the performance of image-to-text and text-to-image retrieval.
For image-to-text retrieval, the task is to find the corresponding caption from a set of possible candidates (the so-called reference set) given a query image and vice versa.
We retrieved the most similar images or text from the reference set according to the similarity scores of all methods and showed the results in Table 2.
For image-to-text retrieval, we show mean average precision (mAP), precision@1, and precision@5.
For text-to-image retrieval, we only evaluated performance using precision@1 since there is only one correct image.
From Table 2, \nameexhibits lower performance compared to the multimodal baseline, CLIP. However, it outperforms the unimodal baseline, ASIF, in both retrieval scenarios.
Notably, ASIF completely failed in the text-to-image retrieval task.

Figure: (a) Results for a training size of $5000$.
Refer to caption: extracted/5915368/figure/imagenet/ROC_mislabeled_curves_size5000_dim700.png

Detecting Mislabeled ImageNet Images.
We again show \name’s superiority with limited and noisy data.
Previous works show that ImageNet is imperfect and contains incorrectly labeled images, leading to several failure modes of downstream models (Vasudevan et al., ).
Trained with the original incorrect ImageNet dataset, all methods detect if the input image and the label align.
We use human-evaluated labels from (Northcutt et al., ) to define incorrectly labeled images.

All methods input an image and caption “This is an image of label” and output if the image and caption align or not (true or false).
LLaVA answers questions about whether the given image and caption align.
\name, CLIP, and ASIF output true if the calculated similarity score of the image and caption pair is greater than a threshold.
We enumerated the thresholds to obtain several true and false positive rates and show the receiver operating characteristic (ROC) curves in Figure 5.
In the legend, we listed the area under the curve (AUC), where a higher value indicates better performance.
In Figure 5, \nameoutperforms all other baselines, even LLaVA, given $5,000$ image-label pairs. We also see that when the size of the training data increases from $5,000$ to $35,000$, the performance of \nameand ASIF increases as well.
We emphasize that the comparison of AUC does not directly compare accuracy.
As shown in Figure 5(b), the true positive rate of \nameoutperforms LLaVA by nearly $10\%$ under the same false positive rate.

Note that the training sets of \nameand ASIF contain mislabeled ImageNet images, demonstrating \name’s robustness to noisy data and efficiency in learning from just $5,000$ pairs.
Similarly, the CLIP encoders’ training set also contains mislabeled ImageNet images, making the comparison fair.
We later demonstrate \name’s robustness with even noisier training data.

Figure: (a) Illustrative examples.
Refer to caption: x4.png

Detecting Misinformative Captions.
Detecting mislabeled images is easy since text captions differ only in labels.
We considered a much harder task to detect misinformation news captions from the COSMOS dataset (Aneja et al., ).
The task involves determining whether a Google-retrieved caption aligns with news images and its original captions, as shown in Figure 6(a).
The benchmark considers the retrieved captions misinformative if they do not align with the images and the corresponding captions simultaneously.
The ultimate goal of the task is to identify and prevent the spread of misinformative news captions on the Internet.

We evaluated the similarities between the image and its retrieved caption, as well as between the two captions, and identified misinformation when both similarity scores fell below the respective thresholds.
Figure 6(b) shows all the ROC curves and their AUC. Again, \nameoutperforms CLIP with two unimodal encoders.
The supervised learning method from the original COSMOS paper is the only method that performs better than \name, although trained with supervised labels of object locations.
Note that the LLaVA agent completely fails this task with a false positive rate of $100\%$, as it always thinks that the retrieved captions align with the given images and the original captions.

Figure: Figure 7: Robustness to noisy ImageNet data: \nameremains performant even when the training labels, comprising a total of 35,000 samples, are partially shuffled.
Refer to caption: extracted/5915368/figure/imagenet/trainsize_vs_accuracy_700shuffled.png

Robustness to noisy data.
To test \name’s robustness, we randomly shuffled a percentage of the training labels of ImageNet and evaluated the resulting test accuracy.
Figure 7 shows that \nameconstantly outperforms ASIF and achieves $70\%$ accuracy even if $50\%$ of the training labels are incorrect.
When all the data are shuffled, both \nameand ASIF degrade to $0\%$ accuracy.

Towards more modalities and ablation study.
To provide a comprehensive study to \name, we show additional results in the Appendix.
To show that \namegeneralizes to other modalities, we tested it on audio and text data from MusicCaps (Agostinelli et al., ) in Section B.
See Section C for the correlation coefficients of multimodal data and the results of \namewith various $s$ in Equation 4 in Section D and unimodal encoders in Section E.

Limitations.
\nameis suitable for bimodal data, similar to CLIP, unlike Imagebind (Girdhar et al., ) is capable of $6$ modalities.
Once the encoded features are obtained, \nameis an optimization problem suitable for CPUs but not GPUs, so one cannot accelerate the computation of \namewith GPUs.
Like ASIF, \nameis based on unimodal encoders. If the unimodal encoders are not foundation models but trained on limited datasets like MNIST, the performance of \namedegrades.

Summary.
\namealways outperforms the unimodal baseline, ASIF, and matches or exceeds CLIP’s performance, except in cross-modal retrieval.
This exception likely arises because cross-modal retrieval involves comparing thousands of similarity score pairs between the query data and the reference set, unlike the other tasks that compare against certain captions.
It highlights the trade-off of \namediscussed in Section 5.
\nameencounters a trade-off between distinguishability and informativeness, i.e., to maintain the distant features.
Retrieval tasks require a more curated balance between these aspects, and other tasks benefit from greater distinguishability of similarity scores.

We conclude that \nameis a robust and data-efficient method to replicate the CLIP similarity score using two unimodal encoders.
In Table 1, the amount of data required compared to CLIP underscores that unimodal encoders are significantly more efficient to train. A small amount of multimodal data suffices to learn the mapping of unimodal features to a multimodal space, even if the training data are unprocessed by humans (COSMOS) or incorrectly labeled (ImageNet).

## 7 Conclusions and Future Works

In this work, we proposed \name, which maps two unimodal feature spaces from pre-trained encoders to a multimodal feature space.
\nameis extremely data-efficient.
We characterize the intrinsic trade-off of \name, and \nameshows competitive performance compared to CLIP models in image classification, mislabeled data detection, cross-modality retrieval, and misinformation detection tasks with limited data.
\namealso outperforms the state-of-the-art method in the same setting.

Future work includes extending \nameto more than $2$ modality pairs, just as generalized CCA (Horst, ) extends CCA to more sources.
In addition, understanding the relationship between the size of the training set and its performance is crucial.
If we can fine-tune the unimodal encoders, which loss function will result in the most suitable feature space for \nameremains an open problem.
Lastly, \nameessentially finds a mapping function between two feature spaces, which need not be modality features.
We aim to test \nameon mapping intramodal data, such as multi-view images of an object.

## Appendix A Additional details on the experiments

We run all inferences of LLAVA and encoder models on an NVIDIA RTX A5000 GPU, and solving Equation 2 with $35,000$ multimodal feature vectors on a $64$ core Xeon Gold 6226R CPU machine takes less than $10$ minutes.
The implementation of CCA is from CCA Zoo Chapman & Wang ().

Multimodal Encoders.
The multimodal image-text encoder used throughout the experiments is laion/CLIP-ViT-bigG-14-laion2B-39B-b160k from Huggingface.
The multimodal audio-text encoder used is laion/larger_clap_general from Huggingface (Wu* et al., ).

Unimodal Encoders.
We use several unimodal encoders and show the difference in performance in Section E.
To encode images, we tested DINOv2-Giant (Oquab et al., )
and the unimodal part of the multimodal encoders previously mentioned.
To encode text, we tested GTR-t5-large (Ni et al., ) and the unimodal part of the multimodal encoders mentioned above.
To show \name’s ability to combine unimodal models, we never tried using the paired unimodal encoders of a multimodal encoder in our experiments, i.e., using CLIP to encode both images and text.

Flickr30k.
We trained ASIF and \nameon the Flickr validation set, which includes $145,000$ images and $5$ captions for each image.
We then validated the models on a test set of $5,000$ images and $25,000$ captions.

COSMOS.
We trained ASIF and \nameon the COSMOS validation set, which includes $41,006$ image-caption pairs. We then validated the models using a test set of 1,700 image-caption pairs, with half of the captions labeled as misinformation by human annotators.

## Appendix B Towards more modalities–-Audio and text

Figure: Figure 8: Classification of YouTube audio and genre tags: \name(blue) performs as well as CLAP, the CLIP-inspired multimodal audio and text encoder, and outperforms ASIF in classifying genre tags of YouTube audio.
Refer to caption: extracted/5915368/figure/musiccaps/ROC_curves_size3777_dim10.png

We now show \name’s generalization ability to more modalities with MusicCaps (Agostinelli et al., ).
We use GTR and CLAP to encode YouTube audio along with the tagged genre descriptions of the audio.
We conducted a classification task in which the models input the audio and a tag and output if the audio aligns with the caption.
Similar to the mislabeled ImageNet experiment, we show the ROC curves and compare the AUC in Figure 8.
We trained ASIF and \namefor $3,777$ data points and tested all methods on $1,625$ data points.
We randomly sampled a tag for each data point during both training and inference.
In Figure 8, we see that \nameperforms as well as CLAP, the CLIP-inspired multimodal audio and text encoder, and outperforms ASIF.
Thus, we conclude that \nameextends its capabilities beyond image and text, effectively handling audio and text as well.

## Appendix C Correlation of Feature Spaces

Figure: Figure 9: Correlation coefficients of COSMOS image and caption features under \name: The data are inherently noisy, as indicated by the correlation coefficients of the unimodal feature spaces, which concentrate on $0.2$ to $0.4$. The unimodal encoders here are GTR and DINOv2.
Refer to caption: extracted/5915368/figure/COSMOS/correlation_coefficient.png

To take a deeper look into unimodal feature spaces, we show the correlation coefficients of COSMOS image and caption features under CSA in Figure 9.
The data are inherently noisy, as indicated by the correlation coefficients of the unimodal feature spaces, which concentrate on $0.2$ to $0.4$.
This distribution of correlation coefficients highlights that, despite the fact that the original multimodal data are noisy and show complex correlations, \namecan effectively map them to a multimodal space where the similarity score remains meaningful for the zero-shot downstream tasks.

## Appendix D Sensitivity to Hyperparameter s 𝑠 s italic_s

**(a)**
| Method | $s$ | mAP | Precision@1 | Precision@5 |
| --- | --- | --- | --- | --- |
| \name | $10$ | $9.5\%$ | $18.1\%$ | $13.4\%$ |
| \name | $50$ | $32.7\%$ | $53.9\%$ | $40.0\%$ |
| \name | $100$ | $36.0\%$ | $58.3\%$ | $42.9\%$ |
| \name | $200$ | $36.6\%$ | $59.3\%$ | $43.4\%$ |
| \name | $500$ | $31.8\%$ | $56.3\%$ | $38.3\%$ |
| \name | $750$ | $27.3\%$ | $50.2\%$ | $33.6\%$ |
| CLIP | ✗ | $73.8\%$ | $92.9\%$ | $77.2\%$ |
| ASIF | ✗ | $14.6\%$ | $25.6\%$ | $20.0\%$ |

We show \name’s sensitivity to the hyperparameter $s$ in terms of the end performance.
In Table 3, CSA achieves optimal performance in $s=200$, and its performance degrades with increases and decreases in $s$, illustrating the trade-off characterized in Section 5.
However, for tasks other than retrieval, we find that a larger $s$ improves performance in image classification, mislabeling detection, and misinformation caption detection.
This phenomenon is likely due to the trade-off between distinguishability and informative embedding features, namely the distance between features.
Although retrieval tasks require a more curated balance between these aspects, other tasks benefit from greater distinguishability of similarity scores.

## Appendix E Sensitivity to Unimodal Encoders

We change the unimodal encoders of ASIF and \nameto showcase their generalization ability to different unimodal encoders.
Figure 11 shows the results on the detection of mislabeled ImageNet data with other encoders, and Figure 12 shows the results on the detection of misinformative captions with other encoders.
\nameagain outperforms ASIF and CLIP while outperforming the results of the combination of GTR and DINOv2 in Section 6.

Figure: (a) Results for a training size of $5000$.
Refer to caption: extracted/5915368/figure/imagenet/ROC_mislabeled_curves_size5000_dim700_clip_gtr.png

Figure: Figure 12: Detecting misinformative COSMOS captions (cont’d): \name(blue) outperforms CLIP, ASIF, and LLaVA with a higher AUC. The supervised-learning method from the original COSMOS paper is the orange cross. It is the only method that outperforms \name, though trained with supervised labels of object locations. The unimodal encoders are GTR and CLIP (image) here.
Refer to caption: extracted/5915368/figure/COSMOS/ROC_ooc_dim750_clip.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: The reviewer will consider increasing the score if the authors can address the weaknesses mentioned above, most likely empirically.
soundness: 3
strengths: S1: (**Data Efficiency and Good Performance**) CSA requires significantly fewer multimodal data pairs by relying on unimodal encoders, which could benefit researchers constrained by data or computational resources. It needs as little as 35,000 image-text pairs to match the performance of CLIP on ImageNet, which is especially notable.

S2: (**Computational Simplicity**) One of the most accessible aspects of CSA is its ability to function effectively without requiring GPU-intensive training, which is a significant advantage for researchers with limited computational resources. 

S3: (**Theoretical Analysis**)The authors include a theoretical analysis for CSA using canonical correlation analysis.
summary: The paper proposes a new approach called Canonical Similarity Analysis (CSA) that addresses the challenge of training multimodal encoders, like CLIP, which typically require large amounts of paired multimodal data. CSA achieves efficient multimodal mapping by using two unimodal encoders, thereby reducing data requirements significantly. CSA utilizes canonical correlation analysis to map unimodal features into a multimodal space and introduces a weighted cosine similarity function to replicate multimodal similarity scores. Experiments across tasks such as image classification, misinformation detection, and cross-modal retrieval demonstrate CSA’s data efficiency and robustness, especially in scenarios with limited or noisy data.
weaknesses: W1: (**Hyperparameter Sensitivity and Limited Justification for $s$ Selection**)
The choice of the hyperparameter $s$ in the canonical similarity metric (Section 4.2) lacks comprehensive justification. While the authors discuss a trade-off in feature distinguishability based on $s$, a more detailed sensitivity analysis showing how varying $s$ affects downstream performance across tasks would add clarity, as the framework is quite sensitive to $s$ indicated by content in Table 3. Table 3 provides some insights, but a broader, task-specific evaluation could make the effects of this parameter choice clearer and improve reproducibility for other researchers.


W2: (**Weakness on Number of Modalities Supported**) While CSA shows promising results with image-text pairs and briefly explores audio-text applications in the appendix, the experimental evaluation of additional modalities remains limited and lacks depth. The audio experiments, while mentioned, do not provide a strong enough demonstration of CSA’s effectiveness beyond the main image-text modality. This limited exploration weakens CSA’s claim of generalizability across modalities and would benefit from more robust testing on diverse modality pairs to strengthen its applicability in multimodal contexts.


W3: (**Scalability Concerns**) While CSA requires fewer training samples to match performance on certain benchmarks (e.g., ImageNet classification with 35,000 image-text pairs), some datasets or applications may still demand larger sample sizes to achieve comparable performance, especially those with more complex or diverse data structures. For instance, datasets like OpenImages (available at https://storage.googleapis.com/openimages/web/index.html), which contain extensive variety in both images and captions and a wide array of object categories, may require larger data samples for CSA to perform effectively. In such cases, the scalability limitations of CSA's matrix decomposition approach become more evident, as the cubic complexity of these operations could lead to significant computational strain on larger datasets. More experimental validation on these varied datasets would clarify the scalability and generalizability of CSA across a broader range of applications.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. In Figure 4, does the smaller subset of ImageNet (5k images) include all of the mislabeled images from ImageNet? A statistic showing the percentage of mislabeled images in the training set for Figures 4(a) and 4(b) would offer more insight into these results.
2. Why did the authors choose not to fine-tune CLIP on ImageNet? Was there some intuition behind this?
soundness: 3
strengths: - The paper is well motivated: Finding data-efficient ways to train multimodal models using existing pretrained unimodal encoders is an important research direction.
- The method used in the paper appears novel, with only one other related work (ASIF) that uses independent unimodal encoders to project embeddings onto a multimodal representational space.
- The proposed approach (CSA) beats or matches CLIP’s performance in image classification tasks using significantly lesser data.
- The paper’s experiments also show that CSA is more robust to mislabeled images in the image classification task. CSA is also slightly more capable in detecting misinformative captions compared to CLIP and other baselines. Finally, the authors show that CSA is more robust to noisy data compared to the previous baseline ASIF.
- Overall, this study could pave the way for more advancements in this space of adapting independent unimodal encoders to multimodal models.
summary: The paper proposes CSA: Canonical Similarity Analysis method to train a multimodal encoder from two independent unimodal encoders with limited data. Unlike traditional methods that train both encoders on multimodal data, CSA only requires the unimodal encoders to generate embeddings, avoiding the need to train them on multimodal pairs.

The authors show that their CSA method matches or beats CLIP and the baseline ASIF on image classification on ImageNet and LeafySpurge datasets. However, the authors’ method falls short of CLIP’s performance on the cross-modal retrieval task. Their experiments also show that CSA outperforms the baselines when detecting mislabeled ImageNet images, and detecting misinformative captions.

The paper concludes that CSA is a robust method to replicate CLIP similarity scores using two pre-trained unimodal encoders and limited data.
weaknesses: - In Figure 3, CSA was only trained on in-distribution image-caption pairs. This may lead to an unfair comparison to CLIP, as the CSA training has seen the ImageNet/Leafy Spurge distribution that its being tested on during the training process. CLIP’s rise to fame is due to its general zero-shot capabilities. The zero-shot capabilities of CSA are not fully evaluated in this paper.
    - A fairer comparison might be to fine-tune CLIP on ImageNet/Leafy Spurge. This could be done by training the last projection layer of CLIP, analogous to keeping the unimodal encoders frozen in CSA training.
    - Additionally, including out-of-distribution datasets would give a clearer view of CSAs true zero-shot capabilities: An example could be to train CSA on MS COCO and then evaluate the performance on ImageNet / Leafy Spurge.

- Similarly, in Figure 4, comparing CLIP to CSA may not be entirely fair. Although CLIP likely encountered ImageNet images in training, they represent a very small fraction of its large dataset. A fairer comparison could be to fine-tune CLIP (again, perhaps just the  projection layer) on ImageNet images, including those with mislabeled data, before proceeding with this analysis (See Question 2).

- The paper's conclusion suggests that CSA outperforms CLIP in cross-modal retrieval ("CSA shows competitive performance compared to CLIP models in image classification, mislabeled data detection, **cross-modality retrieval**, and misinformation detection tasks with
limited data."). However, this is misleading, as Table 2 shows that CSA underperforms CLIP in this task.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. The visualization experiments can show the interpretability of multi-modal learning, I wonder to know the visualization results of embedding spaces.
2. Although the CSA does not require GPUs to optimize the fitting matrix, I wonder to know the run-time cost of the optimization process on different datasets and feature dimensions. Is it faster than the training a traditional network?
soundness: 3
strengths: 1. This paper proposes the canonical similarity analysis (CSA) framework, which can replicate the CLIP multi-modal model. It just uses two unimodal encoders but demands much less computation cost and related data.
2. This paper provides the theoretical analysis on the trade-off of obtaining informative embeddings and distinguishing multimodal data, considering various hyperparameters of CSA.
3. The extensive experiments on various downstream tasks (such as image classification, cross-modal retrieval, and misinformative caption detection) show that CSA outperforms the traditional CLIP model, while requiring much fewer paired multimodal data and fewer unimodal data.
summary: This paper introduces a canonical similarity analysis (CSA) framework, a novel approach that projects two distinct unimodal feature spaces from pre-trained encoders into a unified multimodal feature space. The CSA has great data efficiency and discovers the inherent trade-offs of informative embeddings and distinguishing data. The extensive experiments show that the CSA achieves competitive performances against CLIP models across a range of tasks, including image classification, mislabeled data detection, cross-modality retrieval, and misinformation detection, even with a limited dataset.
weaknesses: 1. This paper proposes a post-tuning mapping framework on the unimodal features, which would compute the matrix optimization without training any encoders. Hence the performances heavily rely on the choices of visual and textual encoders. The experiments part only shows the one encoder situation (gtr + dino). More model encoders analysis for CSA are needed.
2. In the performance comparisons, the ASIF is the only fair baseline method, which lacks of persuasion. It is important to add more comparative experiments for more related methods, e.g., some prompt tuning and adapter tuning work (PEFT methods). What’s more, when compare CSA with CLIP (e.g., Flickr in Tab. 2, ImageNet in Fig. 3), the CSA is the fine-tuned model on the specific dataset, but the CLIP is the zero-shot model without fine-tuning. Such experimental comparison is unreasonable. The CSA should fine-tuning on the ImageNet and then test on the Flickr.
3. The feature dimensions of $q$ and $r$ are the important hyper-parameters for CSA, and the Tab.1 also shows the different dimension choices on different datasets, hence the related ablation study is missing in this paper.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: See weakness.
soundness: 3
strengths: 1. It only requires using a pre-trained unimodal encoder along with CSA, training on a small number of sample pairs (such as a single image and its corresponding caption) to achieve alignment between two modalities, greatly saving computational resources.
2. In image classification tasks, it achieves results comparable to CLIP using fewer resources.
3. Experimental results significantly outperform the baseline model ASIF.
summary: A new similarity calculation strategy, CSA, based on Canonical Correlation Analysis, is proposed to replace the correlation matrix in CLIP for aligning images and text modalities. CSA allows better alignment of features encoded by unimodal encoders for the same samples without requiring as many training samples as CLIP. A theoretical explanation of CSA is provided. Experiments are conducted on a large number of datasets, all of which outperform the baseline model ASIF.
weaknesses: 1. The theoretical discussion of CSA lacks detailed explanation for Equation (9), and the connection to contrastive learning is not clearly established, indicating a lack of novelity in CSA.
2. In comparisons with CLIP for image classification tasks, only a limited amount of training data was used, with no comparisons made on larger-scale training datasets.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - Q1. Have you considered evaluating CSA using the same backbone as CLIP (e.g. using the same image and text encoders but pre-trained on two different datasets from OpenCLIP)? This would empirically strengthen your hypothesis and demonstrate the effectiveness of your method.
- Q2. Just out of curiosity, how scalable is CSA when handling larger training datasets? How much could it benefit from optimizing the matrix decomposition on a large-scale multimodal dataset?
- Q3. Is it possible to possible to solve CSA on a different dataset from the downstream one? It would be interesting to see whether this improves performance in tasks like cross-modal retrieval, where CSA currently underperforms compared to CLIP.
- Q4. Could the authors better explain why the Leafy Spurge dataset was chosen as an evaluation benchmark, and how its characteristics make it particularly suitable for demonstrating CSA's capabilities?

I would be happy to improve my rating if the authors address my concerns, particularly regarding the fairness of the current comparison.

**Minor observations**:
- Error in Table I. In the supplementary material, it is stated that the CLIP model used is the ViT-bigG-14 trained on LAION-2B. However, in Table I, CLIP is attributed to 12B pairs of training data.
- Error in Table I. Moreover CLIP ViT-bigG-14 is 2.5B of params in total.
soundness: 3
strengths: - The paper is well-organized, making it easy to follow the proposed approach and experimental setup. 
- A key strength of CSA is its remarkable data efficiency, achieving effective mapping and strong results with significantly less training data compared to the baselines. 
- Additionally, the method demonstrates robustness in handling noisy and limited multimodal data, which is essential in real-world applications where large correctly labeled datasets are often scarce. 
- Finally, the theoretical analysis offers valuable insights into the trade-offs between reducing noise and preserving meaningful similarity scores, further enhancing the paper's contribution.
summary: The paper proposes a novel method called Canonical Similarity Analysis (CSA) to improve data efficiency when mapping unimodal features (e.g., from image or text) into multimodal feature spaces, effectively replicating models like CLIP but requiring significantly fewer data using pretrained unimodal models. The method leverages two pretrained unimodal encoders and applies Canonical Correlation Analysis (CCA) to align the features using a small portion of the training set. CSA does not involve training neural networks, only solving a matrix decomposition for feature mapping. The authors demonstrate its effectiveness outperforming a pretrained CLIP in some cases with far fewer multimodal data.
weaknesses: - One major issue is that CSA uses different backbones, such as DINOv2-Giant and a GTR-t5-large as unimodal encoders (lines 715-716), compared to the multimodal baseline using a CLIP ViT-bigG/14 pretrained on LAION-2B (line 712). This creates an unfair comparison and makes hard to objectively state the efficacy of the approach. 
- Additionally, the paper's performance on cross-modal retrieval tasks, particularly in text-to-image retrieval, is noticeably weaker than CLIP, which limits its impact in this commonly used task. 
- Moreover, the greatest weakness is that CSA requires a small portion of training data (i.e. 35k pairs on ImageNet) to solve the matrix decomposition for the features mapping, but all the evaluations are compared to zero-shot CLIP (not fine-tuned on the same portion of training data), which leads to an unfair advantage. Evaluation (e.g Figure 3(a)/3(b) and Table 2(a)/2(b)) should include few-shots adaptation baselines (e.g CoOp [1], CLIP-Adapter [2], etc..) or at least the standard CLIP linear evaluation protocol [3] for a fair comparison with the multimodal baseline. Note that using CSA on 800 pairs of Leafy Spurge could be the main reason for its improved performance against zero-shot CLIP. This is because this data is so different from the training distribution that merely adapting to it leads to the observed improvements.

[1] Learning to Prompt for Vision-Language Models (https://arxiv.org/abs/2109.01134)

[2] CLIP-Adapter: Better Vision-Language Models with Feature Adapters (https://arxiv.org/abs/2110.04544)

[3] Learning Transferable Visual Models From Natural Language Supervision (https://arxiv.org/abs/2103.00020)

Meta-review: This paper aims to design an efficient scheme for fast multi-modal learning. With frozen CLIP or the like, they build their model with canonical similarity (deduced from CCA) to measure the similarity from unimodal to multimodal feature spaces. As the authors claimed, the main strength of this paper lies in using fewer multi-modal data pairs to achieve SOTA downstream task performance. 

All the reviewers have main concerns about the extension capabilities of using more modalities or limited kinds of data pairs, as well as the parameter sensitivity of $s$, which is the key hyperparameter for this work. All the reviewers pointed out the advantages of this paper.

To me, if the authors still highlight the $50,000$ times fewer multimodal data pairs, compared to the CLIP, I think it may be misconducted because the proposed method used the pre-trained unimodal encoders, rather than raw features, which is different from training from raw. Therefore, it is suggested that these overexaggerated words be removed, rather than highlighted, in the final version.

Generally, based on the comments and the authors' rebuttal, this is good work to be accepted for ICLR 2025.

Decision: Accept (Poster)