# OneEncoder: a lightweight framework for efficient multimodal training

Bilal Faye, Hanane Azzag, Mustapha Lebbah, Djamel Bouchaffra

► To cite this version:

Bilal Faye, Hanane Azzag, Mustapha Lebbah, Djamel Bouchaffra. OneEncoder: a lightweight framework for efficient multimodal training. Neural Computing and Applications, 2025, 10.1007/s00521-025-11435-8. http://dx.doi.org/10.1007/s00521-025-1141127

HAL Id: hal-05141127

https://hal.science/hal-05141127v1

Submitted on 7 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# OneEncoder: A Lightweight Framework for Progressive Alignment of Modalities

Bilal FAYE¹, Hanane AZZAG², Mustapha Lebbah³, Djamel Bouchaffra

e-mail: faye@lipn.univ-paris13.fr, azzag@univ-paris13.fr, mustapha.lebbah@uvsq.fr

**Abstract**— Cross-modal alignment Learning integrates information from different modalities like text, image, audio and video to create unified models. This approach develops shared representations and learns correlations between modalities, enabling applications such as visual question answering and audiovisual content analysis. Current techniques rely on large modality-specific encoders, necessitating fine-tuning or training from scratch on vast aligned datasets (e.g., text-image, text-audio, image-audio). This approach has limitations: (i) it is very expensive due to the need for training large encoders on extensive datasets, (ii) acquiring aligned large paired datasets is challenging, and (iii) adding new modalities requires retraining the entire framework to incorporate these modalities. To address these issues, we propose OneEncoder, a lightweight framework that progressively represents and aligns *four* modalities (image, text, audio, video). Initially, we train a lightweight Universal Projection module (UP) to align image and text modalities. Then, we freeze the pretrained UP and progressively align future modalities to those already aligned. OneEncoder operates efficiently and cost-effectively, even in scenarios where vast aligned datasets are unavailable, due to its lightweight design. Trained on small paired datasets, it shows strong performance in tasks like classification, querying, and visual question answering, surpassing methods that rely on large datasets and specialized encoders.

## I. INTRODUCTION

Multimodal processing integrates information from various sensory modalities within a single system. Cross-modal alignment learning (CM-AL) involves developing shared representations to enable seamless understanding across different modalities [1]–[3]. Recent advancements in Large Language Models [4, 5], Large Vision Models [6, 7], and Large Audio Models [8, 9] have enabled CM-AL tasks, starting with the alignment of two modalities and extending to multiple modalities [10]–[13]. However, these methods depend on large modality-specific encoders that must be trained on vast aligned datasets to achieve high performance. This approach presents several challenges: large models incur high computational costs; acquiring large aligned datasets, especially for specific modalities, is difficult; furthermore, once trained, adding new modalities requires retraining the entire framework, resulting in significant costs. Thus, a critical challenge is to develop a framework that achieves good alignment results without relying on vast aligned datasets, while progressively aligning diverse modalities.

We draw inspiration from recent research on aligning modalities with language by Han et al. [14]. In their OneLLM study, the authors employ a frozen pretrained

<figure><img src="image_2.png" /><figcaption>**Fig. 1:** Comparison of three modality alignment methods: Standard cross-modal vs. OneEncoder. Standard aligns via simultaneous training of modality-specific encoders. OneEncoder uses frozen, pretrained encoders with a lightweight Universal Projection (UP) module trained on two modalities. For new modalities, UP stays frozen, training only the Alignment Layer. Modality tokens enable efficient switching between modalities. Using this method, video can be aligned with other modalities (image, text, audio) in the same way.</figcaption></figure>

vision-language encoder to encode eight modalities and align them simultaneously with the text modality using a learned mixture of projection experts. This insight suggests that training modality-specific encoders may not be necessary; instead, pretrained encoders can be utilized to represent specific modalities.

In this paper, we introduce OneEncoder, which progressively aligns five modalities (image, text, audio and video) within a single unified framework. Illustrated in Figure 1, OneEncoder comprises frozen pretrained modality-specific encoders, a lightweight Universal Projection module (UP), a compact Alignment Layer (AL), and modality tokens to facilitate modality switching [14]–[17]. Unlike the standard approach,
pretrained modality-specific encoders are frozen and used exclusively for feature extraction, OneEncoder employs a single encoder to achieve greater efficiency. This differs from OneLLM, which uses a mixture of projection experts for the UP module.

We propose a two-step process for progressively training our framework across multiple modalities. In Step 1, we pretrain the UP using image-text data, given that this type of data is more abundant compared to other modalities. In Step 2, which is consistent across all future modalities, we freeze the pretrained UP and train only the compact AL. This step aligns new modalities with the already aligned ones. For example, we first align audio with image and text, followed by aligning video with image, text, and audio.

The role of the AL is not to enhance representation but merely to project the new data into this shared space. Using OneEncoder represents a balanced compromise between alignment performance and complexity, as it minimizes the number of parameters to tune. This approach enables the integration of new modalities at a low cost and ensures good performance even in the absence of vast aligned datasets. Our contributions can be encapsulated in three key points:

* We introduce OneEncoder, an alignment framework designed to progressively align four modalities: image, text, audio and video. To achieve effective alignment without relying on vast aligned datasets, we focus on lightweight models that do not require extensive data like larger models do. We minimize the number of parameters trained in each step. In Step 1, we train only the UP. In Step 2, we train only the AL, which is even lighter than the UP because the UP is already trained and provides a well-defined shared alignment space.

* We leverage lightweight framework architecture to achieve consistent alignment, even in scenarios where vast aligned datasets are unavailable.

* In various downstream tasks such as classification, querying, semantic and visual question answering, our method outperforms classical methods, which trains modality-specific encoders simultaneously.

## II. RELATED WORKS

The progression of large language models (LLMs) has led to their use in diverse domains beyond natural language processing, such as vision and audio. Due to their increasing power, these models are now applied to cross-modal learning tasks to align different modalities in a common semantic space. This alignment enhances representation performance by enabling the learning of one modality using information from other modalities.

**Dual Modality Alignment (DMA).** Integrates representations from distinct modalities into a unified semantic space, promoting seamless interaction. Pioneering

efforts, such as Flamingo [18], focus on aligning image and text modalities by incorporating visual features into LLMs with cross-attention layers, enhancing performance in vision-language tasks. Similarly, ConVIRT [19] employs contrastive learning in small aligned medical datasets, while CLIP [10] scales up contrastive learning in large aligned datasets, advancing the understanding of concepts across modalities. Additionally, ALIGN [20] addresses the challenge of noisy datasets, ensuring robust performance in real-world scenarios.

Building upon promising results in image-text alignment, recent works have sought to expand this approach to additional modalities such as text-audio [21, 22], image-audio [23], text-video [24], and others. While these methods yield powerful representation and alignment, they suffer from two significant limitations: 1) They typically require extensive resources for training on vast aligned datasets, and 2) They are restricted to only two modalities, constraining their generalization to other modalities.

**Multiple Modalities Alignment (MMA).** Extends the concept of DMA by synchronizing representations from more than two distinct modalities into a shared semantic space. For instance, AudioCLIP [12] extends CLIP to handle audio alongside text and images, while ImageBInD [25] aligns six modalities using zero-shot capabilities of vision-language models. NExT-GPT [26] enables any-to-any multimodal understanding and output but still relies on aligned datasets for training, posing challenges similar to DMA. Transitioning from heavyweight to lightweight models becomes crucial to address resource-intensive training requirements and ensure performance on diverse modalities even with limited aligned data.

**Transitioning to Lightweight Models for Modalities Alignment.** Employs frozen pretrained models and modality tokens to handle multiple modalities with a single encoder, reducing the need for large aligned datasets and minimizing parameter learning [27]–[29]. Meta-Transformer [29] demonstrated competitive performance across 12 data modalities with a frozen visual encoder, inspiring Han et al. [14] to propose a unified framework for aligning eight modalities using a frozen CLIP model and a single Universal Projection module (UP) with modality tokens for switching. While addressing the heavyweight nature of previous approaches, these methods struggle to integrate new modalities into existing alignments.

Our work advances the lightweight framework paradigm through two key contributions: proposing an open framework that progressively aligns modalities, enabling seamless integration of new modalities with existing ones without retraining; and leveraging the lightweight nature of our approach to perform effectively even in the absence of vast aligned datasets.
### III. METHOD

In this section, we will begin by introducing the OneEncoder architecture (Section III-A), followed by an overview of the training phase (Section III-B).

#### A. Model Architecture: OneEncoder

Drawing from research by [14, 27]–[29], we capitalize on the robust modality transfer capabilities of pretrained encoders. This approach allows to leverage pretrained modality-specific models, who are trained on large modality-specific datasets, which are more readily available than large aligned datasets. Within OneEncoder, we employ ViT [6] for image encoding, BERT [4] for text encoding, Wav2Vec2 [9] for audio encoding and VideoMAE [30] for video encoding. Each model produces an input token $\mathbf{x} \in \mathbb{R}^{L \times D}$ as its output, where $L$ represents the sequence length and $D$ denotes the token dimension. Consistent with previous research [14, 27]–[29], we also maintain the parameters of these models frozen during training. Figure 2 illustrates the three primary elements comprising OneEncoder: modality-specific encoders, a Universal Projection module (UP), and an Alignment Layer (AL).

**Universal Projection Module (UP).** In contrast to existing methods that rely on modality-specific encoders and multiple projection experts, we introduce an encoder (UP) built using transformer layers [31]. This module is designed to project any modality into a rich-common embedding space. To facilitate modality switching within the UP, we introduce learnable modality tokens $\{\mathbf{t}_m\}_{m \in \mathcal{M}} \in \mathbb{R}^{N \times D}$, as used in [14], which consists of $N$ tokens of dimension $D$, for each modality $m \in \mathcal{M}$. During the forward pass for modality $m$, we input the fusion of input tokens $\mathbf{x}_m \in \mathbb{R}^{L \times D}$ and modality tokens $\mathbf{t}_m$ into the UP module:

$$ \hat{\mathbf{x}}_m = \mathrm{UP}(\mathbf{t}_m \otimes \mathbf{x}_m), \quad (1) $$

where $\otimes$ denotes the fusion operation, which includes concatenation and addition as discussed in [14, 32, 33], as well as cross-attention as detailed in [34], with $\mathbf{t}_m$ serving as the query and $\mathbf{x}_m$ as the key and value. As shown in Figure 2a, the UP module is initially trained to align two modalities. It is then frozen on step 2 for the alignment of additional modalities, as depicted in Figure 2b.

**Alignment Layer (AL).** As shown in Figure 2b, the AL is designed to integrate new modality into the OneEncoder framework efficiently. It consists of a two-layer multi-layer perceptron (MLP). The use of a lighter model compared to the UP module allows for simpler, faster, and more cost-effective alignment of new modality. During the forward pass for a new modality $m$, the fusion of input tokens $\mathbf{x}_m \in \mathbb{R}^{L \times D}$, transformed by the AL component, and modality tokens $\mathbf{t}_m$ is fed into the frozen UP module:

$$ \mathbf{x}_m = \mathrm{AL}(\mathbf{x}_m) \quad (2) $$

$$ \hat{\mathbf{x}}_m = \mathrm{UP}(\mathbf{t}_m \otimes \mathbf{x}_m) \quad (3) $$

Step 2 can be repeated multiple times, each time a new modality needs to be aligned with the already aligned modalities.

#### B. Training Procedure

The OneEncoder alignment process follows a progressive two-step approach. The first step serves as the alignment initialization, while the second step can be repeated to incorporate additional modalities.

* **Step 1: Image-Text Alignment.** Using available aligned image-text datasets and advancements in the field [10, 20], we train the UP module to align image and text modalities in a shared latent space. The UP's parameters are updated using the adapted InfoNCE loss [35] for contrastive (text, image) representation learning by Zhang et al. [19].

During training, we sample a minibatch of $K$ input pairs $(\hat{\mathbf{x}}_{\mathrm{image}}^i, \hat{\mathbf{x}}_{\mathrm{text}}^i)$ from the dataset. The contrastive loss between image and text for each pair $(\hat{\mathbf{x}}_{\mathrm{image}}^i, \hat{\mathbf{x}}_{\mathrm{text}}^i)$ in the minibatch can be formulated as follow:

$$ \ell_{ij} = - \log \frac{\exp\left(\left(\frac{\mathrm{img}_{\mathrm{image}}^{de}, \hat{\mathbf{x}}_{\mathrm{text}}^{j}}{\tau}\right)\right)}{\exp\left(\left(\frac{\hat{\mathbf{x}}_{\mathrm{image}}, \hat{\mathbf{x}}_{\mathrm{text}}^{k}}{\tau}\right)\right)} \quad (4) $$

The term $\langle \hat{\mathbf{x}}^i, \hat{\mathbf{x}}^j \rangle$ represents cosine similarity, with $\mathrm{img}_{\mathrm{image}}$ text

$\tau \in \mathbb{R}^+$ as a temperature parameter. This loss function preserves mutual information between true pairs through representation functions. To ensure symmetry, we introduce a similar contrastive loss from text to image:

$$ \ell_{ji} = - \log \frac{\exp\left(\frac{\langle \hat{\mathbf{x}}^i, \hat{\mathbf{x}}^j \rangle}{\tau}\right)}{\exp\left(\frac{\langle \hat{\mathbf{x}}_{\mathrm{image}}, \hat{\mathbf{x}}_{\mathrm{text}}^j \rangle}{\tau}\right)} \quad (5) $$

The matching pairs are situated along the diagonal of the similarity matrix $(\hat{\mathbf{x}}_{\mathrm{image}}^i, \hat{\mathbf{x}}_{\mathrm{text}}^i)$, which serves as the target for the loss function:

$$ t_{ij} = \frac{\exp\left(\left(\langle \hat{\mathbf{x}}_{\mathrm{image}}^i, \hat{\mathbf{x}}_{\mathrm{text}}^j \rangle + \langle \hat{\mathbf{x}}_{\mathrm{text}}^i, \hat{\mathbf{x}}_{\mathrm{text}}^j \rangle\right)/2 \cdot \tau\right)}{\sum_{k=1}^{K} \exp\left(\left(\langle \hat{\mathbf{x}}_{\mathrm{image}}^k, \hat{\mathbf{x}}_{\mathrm{text}}^i \rangle + \langle \hat{\mathbf{x}}_{\mathrm{text}}^k, \hat{\mathbf{x}}_{\mathrm{text}}^k \rangle\right)/2 \cdot \tau\right)} \quad (6) $$

$$ t_{ji} = \frac{\exp\left(\left(\langle \hat{\mathbf{x}}_{\mathrm{image}}^i, \hat{\mathbf{x}}_{\mathrm{text}}^j \rangle + \langle \hat{\mathbf{x}}_{\mathrm{text}}^i, \hat{\mathbf{x}}_{\mathrm{text}}^j \rangle\right)/2 \cdot \tau\right)}{\sum_{k=1}^{K} \exp\left(\left(\langle \hat{\mathbf{x}}_{\mathrm{image}}^k, \hat{\mathbf{x}}_{\mathrm{text}}^i \rangle + \langle \hat{\mathbf{x}}_{\mathrm{text}}^k, \hat{\mathbf{x}}_{\mathrm{text}}^k \rangle\right)/2 \cdot \tau\right)} \quad (7) $$

The ultimate training loss $L$ (8) is computed by combining the two losses $\ell_{ij}$ and $\ell_{ji}$ and averaging them over all pairs within each minibatch.

$$ L = \frac{1}{2 \cdot K} \sum_{i=1}^{K} \sum_{j=1}^{K} t_{ij} \cdot \mathcal{E}_{ij} + t_{ji} \cdot \mathcal{E}_{ji} \quad (8) $$

* **Step 2: Alignment of Future Modalities.** Once the UP module is trained in Step 1, it is frozen for Step 2. In this step, a new modality $m_i$ is aligned with the already aligned image and text modalities by selecting one (either image or text) for alignment, as illustrated in Figure 2b using the audio modality. The alignment of the selected modality ensures transitive alignment across
<figure><img src="image_3.png" /><figcaption>(a) Step 1: Training the Lightweight UP and Aligning Image-Text Modalities</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>(b) Step 2: Freeze the Pretrained UP, Train the Compact AL, and progressively Align Audio with the Image-Text Modalities from Step 1. This process can be extended to align additional future modalities, such as video.</figcaption></figure>

<caption>**Fig. 2: OneEncoder architecture.** OneEncoder includes frozen pretrained modality-specific encoders, a Universal Projection module (UP), and an Alignment Layer (AL). In step 1, the UP, which consists of a Transformer encoder, is trained to align text and image modalities. In step 2, the pretrained UP is frozen, and the AL, composed of a multi-layer perceptron, is trained to align audio with the text and image modalities. During this step, either image or text is selected to align with audio, indirectly aligning audio with the non-selected modality. The UP fuses input ($x_m$) and modality tokens ($t_m$) to switch between modalities during a forward pass.</caption>

all three modalities (image, text, and $m_i$). During this step, only the AL is trained, using the same loss function as in Step 1 (Equation 8) to update its parameters for consistent input to the UP module. This process is repeated whenever a new modality $m_j$ is introduced (e.g., video).

Algorithm 1 provides a detailed procedure for training the UP on text-image modalities. Once trained, the UP is utilized in Algorithm 2 to align a new modality, denoted as $m_2$, with the set of already aligned modalities, M, using an intermediary modality $m_1$, where $m_1$ must be part of M. This alignment process is achieved by training the AL to project the new modality, $m_2$, into a coherent space compatible with the UP representation. After this process, the expanded set of aligned modalities becomes $M \cup \{m_2\}$. This

alignment can be repeated indefinitely, allowing additional modalities to be aligned with those already in M.

In Algorithm 3, the OneEncoder framework is used to represent any modality in M. For text and image modalities, only the UP is required, while for other modalities, both the UP and AL are necessary. Figure 3 illustrates this process, with Figures 3a and 3b showing how the UP handles text and image representations, and Figures 3c and 3d demonstrating the use of both the UP and AL for audio and video. This approach can be applied post-training for tasks such as zero-shot inference or fine-tuning for domain-specific adaptation.
<figure><img src="image_5.png" /><figcaption>**Fig. 3:** After training, OneEncoder can be used for various downstream tasks: in zero-shot mode by freezing the UP (Universal Projection) and AL (Alignment Layer), or fine-tuned for other tasks.</figcaption></figure>

# IV. EXPERIMENT

In this experiment, we aim to use OneEncoder to align four different modalities: image, text, audio, and video. Given the greater availability of datasets paired with text, we propose leveraging text as the central modality for transitive alignment. The alignment process can be summarized as follows:

1. **Align Image with Text:** Train the UP using Algorithm 1 on the image-text modality pair.

2. **Align Audio with Image and Text:** Train AL<sub>audio</sub> using Algorithm 2 on the text-audio modality pairs.

3. **Align Video with Image, Text, and Audio:** Train AL<sub>video</sub> using Algorithm 2 on the video modality pairs.

The order of alignment steps can be adjusted based on the availability of aligned data and the specific modalities to be aligned.

## A. Datasets

**Training Datasets.** Our goal is to achieve robust performance on downstream tasks using a lightweight framework trained on a modest dataset. Following the approach of virTex [36] and related studies [37, 38], we train the UP module on a combined dataset, which includes COCO Captions [39], Flickr30K [40], and TextCaps [41]. To train the AL<sub>audio</sub>, we utilize the LibriSpeech Speech Recognition Alignment (SRA) [42] Dataset, a corpus containing approximately 1,000 hours of 16kHz recorded English speech.

For the AL<sub>video</sub>, we employ the Microsoft Research Video to Text (MSR-VTT) [43] dataset, a large-scale resource designed for open-domain video captioning.

A detailed description of all datasets used in training the OneEncoder framework is provided in Table I.

**Validation Datasets.** For validating OneEncoder, we use various datasets, tailored either for specific modality-based validation (e.g., classification tasks) or cross-modal validation (e.g., zero-shot tasks). A comprehensive description of the datasets used for validation is provided in Table II.

## B. Implementation Details

**Architecture.** The pretrained encoders for each modality are as follows: ViT-base [58] with 86M parameters for images, BERT-base [4] with 110M parameters for text, Wav2Vec [9] with 317M parameters for audio and VideoMAE-base [30] with 94.2M for video. Additionally, the UP encoder consists of four Transformer block with 4M parameters, while the AL comprises a multi-layer perceptron with 65,792 parameters. The size of modality tokens for each modality is R<sup>1×768</sup>.

**Training Details.** We use the AdamW optimizer [59] with a learning rate of 0.001, $\theta_1 = 0.9$, $\theta_2 = 0.95$, and a weight decay of 0.001. For step 1, we train to align image-text pairs, updating only the UP parameters, on a single A100 GPU for 500 epochs with a batch size of 512. For step 2, to align other modalities (audio and video), we freeze the pretrained UP from step 1, and train only the AL<sub>nv</sub> m ∈ {audio, video} for 100 epochs, using the same parameters as in step 1 with
**Algorithm 1:** Step 1: Training the Universal Projection (UP) model on the image-text modality

**Input** : image encoder; text encoder; **I**: minibatch of aligned images; **T**: minibatch of aligned texts; UP: transformer; M = {image, text} ; {**t**<sub>m</sub>}<sub>m∈M</sub> ∈ ℝ<sup>N×D</sup>; τ: learned temperature parameter; ⊗: fusion operator

**Output:** Trained UP; List of aligned modalities; modality tokens

1 // Freeze the pretrained encoders

- Freeze(imageEncoder)

-Freeze(textEncoder)

2 // Extract feature representations of each modality

- Ximage = imageEncoder(I)

-Xtext = textEncoder(T)

3 // Encode each modality after selection and fusion

- X̂<sub>image</sub> = UP(t<sub>image</sub> ⊗ X<sub>image</sub>)

- X̂<sub>text</sub> = UP(t<sub>text</sub> ⊗ X<sub>text</sub>)

4 // Compute Loss and Update UP Parameters, timage, and ttext

- Compute Loss using Equation 8: L( X̂<sub>image</sub> , X̂<sub>text</sub>, τ)

- Update the UP parameters, **t**<sub>image</sub>, and **t**<sub>text</sub> using an optimizer algorithm based on the computed loss.

5 // Return the trained UP, list of aligned modalities M, modality tokens

- return UP, M, {t_m}m∈{image(text)

a batch size of 64.

We trained two OneEncoder models, each utilizing a different fusion operation: addition and scaled dot product attention [31]. For simplicity, we refer to the model using addition as **OneEncoder-1**, and the model using scaled dot product attention as **OneEncoder-2**.

Our objective is not to achieve state-of-the-art results, which typically demand resource-intensive architectures and extensive hyperparameter tuning. Instead, we aim to explore the behavior of frozen versus non-frozen modality-specific encoders. Specifically, we seek to demonstrate that using frozen encoders within our OneEncoder framework can notably enhance performance and, in many cases, yield better representations for downstream tasks. For a fair comparison, we refer to the baseline approach, which involves training modality-specific encoders, as the **Base** framework.

C. Quantitative Evaluation

1) **UP Validation Following Image-Text Modalities Training:** After training the UP module on a combined dataset of COCO Captions, Flickr30K, and TextCaps, we validate the OneEncoder framework by benchmarking it against the baseline CLIP model [10]. In our method, the

**Algorithm 2:** Step 2: Align a new modality with the previously aligned modalities

**Input** : m₁ encoder; m₂ encoder; **M**₁: minibatch of aligned m₁ modality; **M**₂: minibatch of aligned m₂ modality; UP: pretrained transformer in algorithm 1; M: aligned modalities; {**t**<sub>m</sub>}<sub>m∈{m₁m}</sub> ∈ ℝ<sup>N×D</sup>: modality tokens ; τ: learned temperature parameter; ⊗: fusion operator; AL: Multi-layer Perceptron

**Output:** Trained AL; List of aligned modalities; **t**<sub>m2</sub>

1 // Freeze the pretrained encoders, UP and m1 modality token

-Freeze(m₁Encoder)

-Freeze(m₂Encoder)

-Freeze(UP)

-Freeze(t<sub>m1</sub>)

2 // Extract feature representations of each modality

- X<sub>m1</sub> = AL(X<sub>m1</sub>)

- X<sub>m2</sub> = m2-encoder(M<sub>2</sub>)

3 Project feature representations with the AL

- X<sub>m1</sub> = AL(X<sub>m1</sub>)

- X<sub>m2</sub> = AL(X<sub>m2</sub>)

4 // Encode each modality after selection and fusion

- X̂<sub>m1</sub> = UP(t<sub>m1</sub> ⊗ X<sub>m1</sub>)

- X̂<sub>m2</sub> = UP(t<sub>m2</sub> ⊗ X<sub>m2</sub>)

5 // Compute Loss and Update AL Parameters and t<sub>m2</sub>

- Compute Loss using Equation 8: L( X̂<sub>m1</sub> , X̂<sub>m2</sub> , τ)

- Update the AL parameters and **t**<sub>m2</sub> using an optimizer algorithm based on the computed loss.

6 // Return the trained AL, list of aligned modalities and m2 modality token

- Update list of aligned modalities: M = M ∪ {m₂}

- return AL, M, t<sub>m2</sub>

pretrained ViT and BERT models remain frozen during training, with only the UP module’s 4M parameters being updated. In contrast, the baseline requires training all 196M parameters of the ViT and BERT models. For specific tasks, we employ pretrained models: ResNet-18 [60], EfficientNet-B0 [61], and Swin Transformer [62] for image processing, and RoBERTa [63], DistilBERT [64], and XLNet [65] for text processing.

We encode each modality using Algorithm 3 within the OneEncoder framework and evaluate the performance on various classification tasks.

a) **Zero-shot Classification:** is a task where a model, trained on labeled images, can classify new images from previously unseen classes. It validates the
**Algorithm 3:** Inference: Encoding a Given Modality Using Pretrained UP and AL

**Input** : m: modality of the data to be encoded; **M**: minibatch of data from modality m; UP: Universal Projection model; AL_m: Alignment Layer for modality m; **t_m**: token representing modality m; m encoder: encoder for modality m

**Output:** Encoded representation data

1. // Extract feature representations

$$ \mathbf{X} = \text{m
code}(\mathbf{M}) $$

2. if $m \notin \{\text{image, text}\}$ then

3. // Use AL for feature projection

$$ \mathbf{X} = \text{AL}_m(\mathbf{X}) $$

4. end

5. // Encode with the Universal Projection

$$ \hat{\mathbf{X}} = \text{UP}(\mathbf{t}_m \otimes \mathbf{X}) $$

6. // Return encoded representation of input data

$$ \text{return } \hat{\mathbf{X}} $$

<table><caption>TABLE I: Training datasets</caption><thead><tr><th>Dataset</th><th>Type</th><th>Training Size</th><th>Validation Size</th></tr></thead><tbody><tr><td>COCO Captions [39]</td><td>text-image pairs</td><td>413,915</td><td>202,520</td></tr><tr><td>Flickr30K [40]</td><td>text-image pairs</td><td>158,915</td><td>-</td></tr><tr><td>TextCaps [41]</td><td>text-image pairs</td><td>109,765</td><td>15,830</td></tr><tr><td>SRA [42]</td><td>text-audio pairs</td><td>281,241</td><td>5,559</td></tr><tr><td>MSR-VTT [43]</td><td>text-video pairs</td><td>6,513</td><td>497</td></tr></tbody></table>

<table><caption>TABLE II: Validation datasets</caption><thead><tr><th>Dataset</th><th>Type</th><th>Training Size</th><th>Validation Size</th></tr></thead><tbody><tr><td>CIFAR-10 [44]</td><td>image</td><td>50,000</td><td>10,000</td></tr><tr><td>Oxford-IIIT Pets [45]</td><td>image</td><td>3,680</td><td>3,669</td></tr><tr><td>CIFAR-100 [46]</td><td>image</td><td>50,000</td><td>10,000</td></tr><tr><td>Caltech 101 [47]</td><td>image</td><td>7,659</td><td>3,060</td></tr><tr><td>Tiny ImageNet [48]</td><td>image</td><td>100,000</td><td>10,000</td></tr><tr><td>SST-2 [49]</td><td>text</td><td>67,349</td><td>872</td></tr><tr><td>TREC [50]</td><td>text</td><td>5,452</td><td>500</td></tr><tr><td>Emotion [51]</td><td>text</td><td>16,000</td><td>2,000</td></tr><tr><td>GTZAN [52]</td><td>audio</td><td>1,000</td><td>-</td></tr><tr><td>UrbanSound8K [53]</td><td>audio</td><td>7,980</td><td>1,022</td></tr><tr><td>ESC-50 [54]</td><td>audio</td><td>1,600</td><td>400</td></tr><tr><td>MSVD [55]</td><td>video-text</td><td>48,779</td><td>4,291</td></tr><tr><td>LSMDC [56]</td><td>video-text</td><td>118,081</td><td>-</td></tr><tr><td>DAQUAR [57]</td><td>image-text</td><td>6794</td><td>5,673</td></tr></tbody></table>

model's generalization capability and assesses semantic understanding and transfer learning. Using the CLIP approach, we transform labels into text descriptions ("A photo of a {label}."), encode them with a pretrained model, compute cosine similarity with image embeddings, and use softmax to determine class probabilities.

Zero-shot image classification obviates the need for retraining pretrained models on target datasets, evaluating their ability to generalize to unseen classes. It underscores the importance of the aligned latent space. Results in

<table><caption>TABLE III: Image-Text Alignment Validation: Zero-shot image classification is used to assess the alignment accuracy (%) across five benchmark datasets with varying class counts, providing a measure of the relevance and effectiveness of the image-text alignment.</caption><thead><tr><th>model</th><th>CIFAR-10</th><th>Oxford-IIIT Pets</th><th>CIFAR-100</th><th>Caltech-101</th><th>Tiny ImageNet</th></tr></thead><tbody><tr><td>CLIP</td><td>62.12</td><td>58.27</td><td>53.06</td><td>52.17</td><td>47.15</td></tr><tr><td>OneEncoder-1</td><td><b>78.15</b></td><td><b>69.23</b></td><td><b>58.18</b></td><td><b>56.20</b></td><td><b>52.27</b></td></tr><tr><td>OneEncoder-2</td><td>74.70</td><td>68.98</td><td>57.15</td><td>54.12</td><td>51.12</td></tr></tbody></table>

Table III highlights superior performance of OneEncoder (OneEncoder-1, OneEncoder-2) over the baseline (CLIP) across all datasets, suggesting that training large modality-specific encoders may not always be optimal, as demonstrated by the effectiveness of the lightweight OneEncoder framework.

b) **Linear Classification and Fine-Tuning:** involve adding a linear classifier to a pretrained model, freezing the pretrained weights and training only the linear classifier for linear classification, while training both the pretrained model and the linear classifier for fine-tuning. Linear classification allows for the assessment of the quality of the extracted features from the pretrained model, while fine-tuning simulates the practical use of pretrained weights. In OneEncoder, we always freeze the modality-specific encoders; in the fine-tuning task, we train only the UP for image and text datasets. In each case (Linear Classification and Fine-Tuning), we train models for 100 epochs without using any data augmentation strategy.

The results presented in Table IV demonstrate the performance of various models on image and text classification tasks using two training strategies: linear classification and fine-tuning. These approaches allow us to evaluate the models' ability to generalize to new data, providing a comprehensive comparison between OneEncoder, CLIP, and other baselines.

In image classification, OneEncoder consistently outperforms the CLIP model, which uses CLIP-ViT on image datasets. For linear classification, OneEncoder-1 achieves the highest accuracy on CIFAR-10 (90.16%), Oxford-IIIT Pets (86.23%), and Caltech-101 (68.23%), closely rivaling Swin Transformer, which leads in CIFAR-100 (71.12%) and Tiny ImageNet (62.30%). This highlights the efficiency of OneEncoder, especially considering that it only updates the 4M parameters of the UP module, unlike CLIP, which retrains its larger 196M parameters.

In text classification tasks, where CLIP-BERT is used as the baseline for CLIP, OneEncoder again demonstrates superior performance. OneEncoder-1 achieves the best results across all datasets: SST-2 (82.12%), TREC (79.10%), and Emotion (63.09%) in the linear classification setup. This shows its robust ability to handle diverse text modalities, outperforming specialized models like RoBERTa, DistilBERT, and XLNet.

The fine-tuning results further emphasize the effectiveness of OneEncoder. For image classification, OneEncoder-1
<table><thead><tr><th rowspan="3">Model</th><th colspan="5">Linear Classification</th><th colspan="3">Text Classification</th></tr><tr><th colspan="5">Image Classification</th><th rowspan="2">SST-2</th><th rowspan="2">TREC</th><th rowspan="2">Emotion</th></tr><tr><th>CIFAR-10</th><th>Oxford-IIIT Pets</th><th>CIFAR-100</th><th>Caltech-101</th><th>Tiny ImageNet</th></tr></thead><tbody><tr><td>ResNet-18</td><td>89.15</td><td>84.98</td><td>68.10</td><td>63.45</td><td>59.11</td><td>-</td><td>-</td><td>-</td></tr><tr><td>EfficientNet-B0</td><td>89.87</td><td>85.12</td><td>70.15</td><td>64.87</td><td>60.27</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Swin Transformer</td><td>90.17</td><td>86.05</td><td><strong>71.12</strong></td><td>65.10</td><td><strong>62.30</strong></td><td>-</td><td>-</td><td>-</td></tr><tr><td>RoberTa</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>76.04</td><td>77.34</td><td>59.06</td></tr><tr><td>DistilBERT</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>77.15</td><td>76.14</td><td>68.11</td></tr><tr><td>XLNet</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>79.27</td><td>78.11</td><td>60.10</td></tr><tr><td>CLIP</td><td>81.21</td><td>78.16</td><td>60.12</td><td>60.14</td><td>58.14</td><td>80.15</td><td>78.24</td><td>60.23</td></tr><tr><td>OneEncoder-1</td><td><strong>90.16</strong></td><td>86.23</td><td>70.10</td><td><strong>68.23</strong></td><td>62.12</td><td><strong>82.12</strong></td><td><strong>79.10</strong></td><td><strong>63.09</strong></td></tr><tr><td>OneEncoder-2</td><td>89.18</td><td><strong>86.78</strong></td><td>68.27</td><td>65.05</td><td>60.10</td><td>80.87</td><td>78.06</td><td>61.89</td></tr></tbody></table>

<table><caption>**TABLE IV:** Linear classification and fine-tuning accuracy (%) across various benchmark datasets in image and text modalities. Linear classification involves training only a linear classifier while keeping all pretrained models frozen. Fine-tuning entails training both the pretrained models and the linear classifier. For OneEncoder models (OneEncoder-1 and OneEncoder-2), only the UP (Universal Projection) component is trained during fine-tuning, with modality-specific encoders remaining frozen. In contrast, CLIP and other baseline models (ResNet-18, EfficientNet-B0, Swin Transformer, RoberTa, DistilBERT and XLNet) are retrained during fine-tuning.</caption><thead><tr><th rowspan="3">Model</th><th colspan="5">Fine-Tuning</th><th colspan="3">Text Classification</th></tr><tr><th colspan="5">Image Classification</th><th rowspan="2">SST-2</th><th rowspan="2">TREC</th><th rowspan="2">Emotion</th></tr><tr><th>CIFAR-10</th><th>Oxford-IIIT Pets</th><th>CIFAR-100</th><th>Caltech-101</th><th>Tiny ImageNet</th></tr></thead><tbody><tr><td>ResNet-18</td><td>93.23</td><td>90.19</td><td>82.37</td><td>78.12</td><td>67.89</td><td>-</td><td>-</td><td>-</td></tr><tr><td>EfficientNet-B0</td><td>94.56</td><td>92.23</td><td>80.11</td><td>79.98</td><td>68.10</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Swin Transformer</td><td>95.27</td><td>92.11</td><td><strong>82.02</strong></td><td>79.15</td><td>69.09</td><td>-</td><td>-</td><td>-</td></tr><tr><td>RoberTa</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>83.24</td><td>85.45</td><td>66.13</td></tr><tr><td>DistilBERT</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>82.56</td><td>83.27</td><td>63.15</td></tr><tr><td>XLNet</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>84.72</td><td>85.67</td><td>64.11</td></tr><tr><td>CLIP</td><td>86.76</td><td>81.90</td><td>70.87</td><td>69.67</td><td>60.15</td><td>85.15</td><td>84.24</td><td>64.56</td></tr><tr><td>OneEncoder-1</td><td><strong>96.01</strong></td><td>92.32</td><td>81.10</td><td><strong>80.11</strong></td><td>69.12</td><td><strong>86.11</strong></td><td><strong>86.12</strong></td><td><strong>67.12</strong></td></tr><tr><td>OneEncoder-2</td><td>95.98</td><td><strong>93.12</strong></td><td>80.21</td><td>78.23</td><td><strong>69.15</strong></td><td>85.12</td><td>86.00</td><td>66.78</td></tr></tbody></table>

delivers the highest accuracy on CIFAR-10 (96.01%), Oxford-IIIT Pets (92.32%), and Caltech-101 (80.11%), while also performing competitively on Tiny ImageNet (69.12%), narrowly surpassed by Swin Transformer. In text classification, OneEncoder-1 achieves the best performance on SST-2 (86.11%), TREC (86.12%), and Emotion (67.12%), surpassing the fine-tuned CLIP-BERT and other text-specific models.

Overall, the results illustrate that OneEncoder, with its efficient training approach and minimal parameter updates, outperforms CLIP and other models in both image and text tasks, demonstrating its superior generalization and adaptability across multiple modalities.

2) AL<sub>audio</sub> Validation Following Text-Audio Modalities Training: After training the UP on image-text modalities, it is frozen and then used for aligning other modalities. Specifically, for audio alignment, only the AL<sub>audio</sub> with 65,792 parameters is trained within the OneEncoder framework. This process uses a text-audio modality dataset and follows Algorithm 2 on the ROCO dataset. For comparison, we also train AudioCLIP, an extended version of CLIP that aligns image, text, and audio using ViT for images, BERT for text, and Wav2Vec for audio, with a total of 513M parameters to tune.

Table V compares the performance of AudioCLIP and OneEncoder models (OneEncoder-1 and OneEncoder-2) in text-audio retrieval. This task validates the alignment between text and audio. Evaluated using Top-1 Precision/Recall (P@1, R@1) and mean Average Precision

<table><caption>**TABLE V:** Performance metrics for text-audio retrieval tasks on the AudioSet, UrbanSound8K, and ESC-50 datasets. The evaluation includes Top-1 Precision (P@1), Top-1 Recall (R@1), and mean Average Precision (mAP) for the models: AudioCLIP, OneEncoder-1, and OneEncoder-2.</caption><thead><tr><th rowspan="2">Model</th><th colspan="2">AudioSet</th><th colspan="2">UrbanSound8K</th><th colspan="2">ESC-50</th></tr><tr><th>P@1</th><th>R@1</th><th>P@1</th><th>R@1</th><th>P@1</th><th>R@1</th></tr></thead><tbody><tr><td>AudioCLIP</td><td>4.27</td><td>75.37</td><td>27.12</td><td>40.10</td><td>45.11</td><td>78.27</td><td><strong>48.90</strong></td><td>78.21</td><td>75.12</td></tr><tr><td>OneEncoder-1</td><td><strong>5.37</strong></td><td><strong>76.10</strong></td><td><strong>28.37</strong></td><td><strong>41.11</strong></td><td><strong>46.12</strong></td><td><strong>79.65</strong></td><td>47.98</td><td><strong>80.12</strong></td><td><strong>75.57</strong></td></tr><tr><td>OneEncoder-2</td><td>5.10</td><td>76.06</td><td>28.10</td><td>40.89</td><td>45.78</td><td>79.23</td><td>47.87</td><td>78.12</td><td>74.98</td></tr></tbody></table>

(mAP), OneEncoder consistently outperforms AudioCLIP across all datasets. This highlights OneEncoder's efficient latent space and its ability to handle cross-modal retrieval effectively. Unlike AudioCLIP, which requires extensive encoder training, OneEncoder achieves superior results with a lightweight framework, demonstrating its robustness with minimal dataset-specific training.

To validate transitive alignment between audio and

<table><caption>**TABLE VI:** Image-Audio Alignment Validation: Zero-shot image classification is used to assess the alignment accuracy (%) across five benchmark datasets with varying class counts, providing a measure of the relevance and effectiveness of the image-audio alignment.</caption><thead><tr><th>model</th><th>CIFAR-10</th><th>Oxford-IIIT Pets</th><th>CIFAR-100</th><th>Caltech-101</th><th>Tiny ImageNet</th></tr></thead><tbody><tr><td>AudioCLIP</td><td>61.28</td><td>58.15</td><td>52.27</td><td>51.10</td><td>46.04</td></tr><tr><td>OneEncoder-1</td><td><strong>77.01</strong></td><td><strong>69.02</strong></td><td><strong>56.07</strong></td><td><strong>55.37</strong></td><td><strong>50.18</strong></td></tr><tr><td>OneEncoder-2</td><td>74.07</td><td>66.56</td><td>55.18</td><td>53.11</td><td>50.06</td></tr></tbody></table>
image, we apply the zero-shot classification method as described in Section IV-C.1, replacing text descriptions ("A photo of a {label}.") with corresponding audio. Comparing Table VI with Table III, which uses text descriptions, demonstrates that the OneEncoder framework maintains strong alignment between image and audio, even without direct image-audio alignment. This approach is more efficient and powerful than the resource-intensive AudioCLIP, offering a cost-effective solution with superior performance. For representation learning model validation,

<table><caption>TABLE VII: Representation learning evaluation using fine-tuning on the UrbanSound8K and ESC-50 datasets.</caption><thead><tr><th>model</th><th>UrbanSound8K</th><th>ESC-50</th></tr></thead><tbody><tr><td>Piczak-CNN [66]</td><td>73.70</td><td>64.50</td></tr><tr><td>SB-CNN [67]</td><td>79.00</td><td>-</td></tr><tr><td>ESResNet [68]</td><td>85.42</td><td>91.50</td></tr><tr><td>AST [69]</td><td>-</td><td>95.60</td></tr><tr><td>ERANN [70]</td><td>-</td><td>96.10</td></tr><tr><td>AudioCLIP</td><td>88.32</td><td>96.12</td></tr><tr><td>OneEncoder-1</td><td>89.23</td><td>96.87</td></tr><tr><td>OneEncoder-2</td><td>88.86</td><td><b>97.02</b></td></tr></tbody></table>

we fine-tune the models on the UrbanSound8K and ESC-50 datasets. Unlike AudioCLIP, which requires retraining all Wav2Vec parameters, OneEncoder only fine-tunes the UP and the (AL<sub>audio</sub>) for 100 epochs. Table VII shows that OneEncoder-1 and OneEncoder-2 outperform AudioCLIP on both datasets, with OneEncoder-2 achieving the highest accuracy on ESC-50 (97.02%) and OneEncoder-1 leading on UrbanSound8K (89.23%). This demonstrates the efficiency of the OneEncoder framework, achieving superior performance with fewer retrained parameters compared to the more resource-intensive AudioCLIP. These results underscore the robustness of OneEncoder for fine-tuned representation learning across diverse audio classification tasks.

3) AL<sub>video</sub> Validation Following Text-Video Modalities Training: After aligning the audio with both image and text modalities (see Section IV-C.2), we further integrate the video modality and align it with image, text, and audio. This alignment is performed using Algorithm 2, following a similar approach as in audio alignment, where only the AL<sub>video</sub> is trained while keeping the UP frozen. The OneEncoder model is trained for 100 epochs on the MSR-VTT dataset, utilizing the text modality to align with the video modality, which consequently aligns the audio and image modalities with the video through transitive alignment.

For evaluating OneEncoder in the context of text-video alignment, we benchmark its performance against X-CLIP [71], an extended version of CLIP designed for text-video alignment.

Results on Table VIII demonstrate the superior performance of OneEncoder in aligning text and video across both MSVD and LSMDC datasets. On MSVD, OneEncoder-1 outperforms all models with a notable Recall at rank 5 (R@5) of 80.76 and Mean Rank (MnR) of 7.98 in text-to-video retrieval. Similarly, in video-to-text retrieval,

<table><caption>Retrieval performance comparison on LSMDC</caption><caption>Retrieval performance comparison on MSVD</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">Text-to-Video</th><th colspan="3">Video-to-Text</th></tr><tr><th>R@1↑</th><th>R@5↑</th><th>MnR↓</th><th>R@1↑</th><th>R@5↑</th><th>MnR↓</th></tr></thead><tbody><tr><td>Multi Cues [72]</td><td>20.3</td><td>47.8</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>CE [73]</td><td>19.8</td><td>49.0</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>SSB [74]</td><td>28.4</td><td>60.0</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>NoiseE [75]</td><td>20.3</td><td>49.0</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>CLIP-straight [10]</td><td>37.0</td><td>64.1</td><td>-</td><td>59.9</td><td>85.2</td><td>-</td></tr><tr><td>Frozen [76]</td><td>33.7</td><td>64.7</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>TT-CE+ [77]</td><td>25.4</td><td>56.9</td><td>-</td><td>27.1</td><td>55.3</td><td>-</td></tr><tr><td>CLIP4Clip-MeanP (ViT-B/32) [78]</td><td>46.2</td><td>76.1</td><td>10.0</td><td>56.6</td><td>79.7</td><td>7.6</td></tr><tr><td>CLIP4Clip-seqTransf (ViT-B/32) [78]</td><td>45.2</td><td>75.5</td><td>10.3</td><td>62.0</td><td>87.3</td><td>4.3</td></tr><tr><td>CLIP4Clip-MeanP (ViT-B/16) [78]</td><td>47.3</td><td>77.7</td><td>9.1</td><td>62.9</td><td>87.2</td><td>4.2</td></tr><tr><td>CLIP4Clip-seqTransf (ViT-B/16) [78]</td><td>47.2</td><td>77.7</td><td>9.1</td><td>63.2</td><td>87.2</td><td>4.2</td></tr><tr><td>X-CLIP (ViT-B/32)</td><td>47.1</td><td>77.8</td><td>9.5</td><td>60.9</td><td>87.8</td><td>4.7</td></tr><tr><td>X-CLIP (ViT-B/16)</td><td><b>50.4</b></td><td>80.6</td><td>8.4</td><td><b>66.8</b></td><td>90.4</td><td>4.2</td></tr><tr><td>OneEncoder-1</td><td>49.21</td><td><b>80.76</b></td><td><b>7.98</b></td><td>65.89</td><td><b>91.62</b></td><td><b>3.98</b></td></tr><tr><td>OneEncoder-2</td><td>47.02</td><td>79.27</td><td>8.88</td><td>65.23</td><td>89.78</td><td>4.65</td></tr></tbody></table>

<table><caption>TABLE VIII: Comparison of text and video retrieval performance on the MSVD and LSMDC datasets. The reported metrics are Recall at Rank 1 (R@1) and Rank 5 (R@5), where higher values indicate better performance, and Mean Rank (MnR), where lower values are better.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">Text-to-Video</th><th colspan="3">Video-to-Text</th></tr><tr><th>R@1↑</th><th>R@5↑</th><th>MnR↓</th><th>R@1↑</th><th>R@5↑</th><th>MnR↓</th></tr></thead><tbody><tr><td>CT-SAN [79]</td><td>5.1</td><td>16.3</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>JSFusion [79]</td><td>9.1</td><td>21.2</td><td>-</td><td>12.3</td><td>28.6</td><td>-</td></tr><tr><td>CE [73]</td><td>11.2</td><td>26.9</td><td>96.8</td><td>-</td><td>-</td><td>-</td></tr><tr><td>MMT [77]</td><td>12.9</td><td>29.9</td><td>75.0</td><td>-</td><td>-</td><td>-</td></tr><tr><td>NoiseE [75]</td><td>6.4</td><td>19.8</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>CLIP-straight [10]</td><td>11.3</td><td>22.7</td><td>-</td><td>6.8</td><td>16.4</td><td>-</td></tr><tr><td>MDMMT [80]</td><td>18.8</td><td>38.5</td><td>58.0</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Frozen [76]</td><td>15.0</td><td>30.8</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>HiT [81]</td><td>14.0</td><td>31.2</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>TT-CE+ [77]</td><td>17.2</td><td>36.5</td><td>-</td><td>17.5</td><td>36.0</td><td>-</td></tr><tr><td>CLIP4Clip-MeanP (ViT-B/32) [78]</td><td>20.7</td><td>38.9</td><td>65.3</td><td>20.6</td><td>39.4</td><td>56.7</td></tr><tr><td>CLIP4Clip-seqTransf (ViT-B/32) [78]</td><td>22.6</td><td>41.0</td><td>61.0</td><td>20.8</td><td>39.0</td><td>54.2</td></tr><tr><td>CLIP4Clip-MeanP (ViT-B/16) [78]</td><td>23.5</td><td>43.2</td><td>54.8</td><td>22.6</td><td>50.5</td><td>50.3</td></tr><tr><td>CLIP4Clip-seqTransf (ViT-B/16) [78]</td><td>23.5</td><td>45.2</td><td>51.6</td><td>23.2</td><td>42.4</td><td>47.4</td></tr><tr><td>X-CLIP (ViT-B/32)</td><td>23.3</td><td>43.0</td><td>56.0</td><td>22.5</td><td>42.2</td><td>50.7</td></tr><tr><td>X-CLIP (ViT-B/16)</td><td>26.1</td><td><b>48.4</b></td><td>46.7</td><td>26.9</td><td>46.2</td><td><b>41.9</b></td></tr><tr><td>OneEncoder-1</td><td><b>26.12</b></td><td>48.23</td><td><b>46.11</b></td><td><b>27.01</b></td><td><b>46.67</b></td><td>42.3</td></tr><tr><td>OneEncoder-2</td><td>25.32</td><td>46.76</td><td>50.19</td><td>25.67</td><td>44.15</td><td>42.10</td></tr></tbody></table>

it achieves the best R@5 score (91.62) and the lowest MnR (3.98), surpassing strong baselines like CLIP4Clip and X-CLIP.

To validate the transitive alignment between audio and video, we convert each text description into audio and perform audio-video retrieval to assess alignment. Table IX compares these results with those in Table VIII, demonstrating successful audio-video alignment. This confirms the effectiveness of the progressive alignment process, which requires minimal computational resources
<figure><img src="image_6.png" /><figcaption>**Fig. 4:** Qualitative results across three modalities. For each query, OneEncoder retrieves the most relevant data from the available dataset, showcasing the effectiveness of the alignment.</figcaption></figure>

<table><caption>**TABLE IX:** Audio and video retrieval performance on the MSVD and LSMDC datasets. The reported metrics are Recall at Rank 1 (R@1) and Rank 5 (R@5), where higher values indicate better performance, and Mean Rank (MnR), where lower values are better.</caption><thead><tr><th colspan="7">Retrieval performance comparison on MSVD</th></tr><tr><th rowspan="2">Model</th><th colspan="3">Audio-to-Video</th><th colspan="3">Video-to-Audio</th></tr><tr><th>R@1↑</th><th>R@5↑</th><th>MnR↓</th><th>R@1↑</th><th>R@5↑</th><th>MnR↓</th></tr></thead><tbody><tr><td>OneEncoder-1</td><td>46.34</td><td>78.45</td><td>8.13</td><td>63.43</td><td>89.12</td><td>4.15</td></tr><tr><td>OneEncoder-2</td><td><strong>46.78</strong></td><td>77.32</td><td>8.97</td><td><strong>63.78</strong></td><td>87.78</td><td>4.98</td></tr></tbody><thead><tr><th colspan="7">Retrieval performance comparison on LSMDC</th></tr><tr><th rowspan="2">Model</th><th colspan="3">Audio-to-Video</th><th colspan="3">Video-to-Audio</th></tr><tr><th>R@1↑</th><th>R@5↑</th><th>MnR↓</th><th>R@1↑</th><th>R@5↑</th><th>MnR↓</th></tr></thead><tbody><tr><td>OneEncoder-1</td><td>24.37</td><td>46.32</td><td>47.32</td><td>25.23</td><td>44.56</td><td>44.20</td></tr><tr><td>OneEncoder-2</td><td>23.32</td><td>43.13</td><td>49.32</td><td>23.21</td><td>42.12</td><td>46.57</td></tr></tbody></table>

while maintaining the strong performance of the OneEncoder framework.

These quantitative results highlight the efficiency of OneEncoder's lightweight architecture, which achieves high-performance alignment across multiple modalities at low computational cost. This framework can be effectively applied post-training in various tasks such as classification and retrieval, or fine-tuned for specific domains to enhance representation quality.

## D. Qualitative Analysis

Figure 4 presents qualitative results of OneEncoder across image, text, and audio modalities. In step 1 (see Algorithm 1), we demonstrate that OneEncoder effectively retrieves images using text queries and vice versa, indicating that the UP module can comprehend visual and textual content, resulting in relevant retrievals due to well-aligned latent space. In step 2 (see Algorithm 2), we show that image retrieval via audio inputs provides coherent data to the frozen UP, ensuring coherent alignment and making retrievals interesting. These illustrations, along with quantitative analysis, highlight OneEncoder's excellent performance in progressively aligning modalities using a lightweight framework. This framework enables coherent results with small aligned dataset training by utilizing frozen pretrained modality-specific encoders.

## V. ONEENCODER ON VISUAL QUESTION ANSWERING

In Section IV, we demonstrated that OneEncoder can be efficiently trained using a contrastive learning approach to align multiple modalities at a low computational cost. In this section, we introduce an alternative alignment method tailored for Visual Question Answering (VQA) tasks to further train OneEncoder. The goal is to illustrate the versatility of our proposed framework, showing its ability to be applied across various domains while utilizing different
<figure><img src="image_7.png" /><figcaption>(b) **OneEncoder**: The parameters of both the Image Encoder and Text Encoder are frozen, with only the parameters of the Universal Projection (UP) module being trained.</figcaption></figure>

<caption>**Fig. 5:** OneEncoder architecture for the Visual Question Answering (VQA) task. The OneEncoder framework in 5b trains only the UP module to align the textual answer with both the image and the textual question, unlike the baseline method in 5a, which trains all specific encoders (image encoder and text encoder), making it more computationally expensive. Both approaches use a "Prediction Head" to generate textual answers.</caption>

alignment strategies during training.

VQA is a complex task that involves understanding both visual content and textual questions, requiring the model to align and reason across these modalities to generate accurate answers. By employing a specialized alignment mechanism for VQA, we aim to demonstrate OneEncoder's ability to handle cross-modal reasoning tasks beyond retrieval, further highlighting its adaptability across different types of multimodal learning challenges. Figure 5 presents a comparison between the classical VQA approach 5a and the OneEncoder framework 5b. As discussed in IV, OneEncoder trains only the UP module to align the textual answer with the image and question inputs, significantly reducing the number of parameters compared to the Baseline method 5a, which requires training both the image and text encoders. Both methods utilize a "Prediction Head" module to generate the textual answer.

<figure><img src="image_8.png" /><figcaption>**Fig. 6:** Comparison of Trainable Parameters (in millions) between Baseline Models and OneEncoder Variants (OneEncoder-1 and OneEncoder-2).</figcaption></figure>

To train both the Baseline and OneEncoder frameworks, we utilize the DAQUAR (Dataset for Question Answering on Real-world images) [57]. For modality-specific encoders, we employ BEiT-base [82], DEiT-base [83], and ViT-base [58]
<figure><img src="image_9.png" /><figcaption>**Fig. 7:** Validation Performance of Baseline Models and OneEncoder Variants (OneEncoder-1, OneEncoder-2) on the DAQUAR dataset, evaluated using Wu-Palmer Similarity (WUPS), Accuracy, and F1 Score.</figcaption></figure>

models as image encoders, each with 86M parameters, while ALBERT [84] (60M parameters), BERT-base [4] (110M parameters), and RoBERTa-base [63] (125M parameters) serve as text encoders. We construct 9 VQA models for each method (Baseline and OneEncoder) by combining these encoder pairs: (BEiT, ALBERT), (BEiT, BERT), (BEiT, RoBERTa), (DEiT, ALBERT), (DEiT, BERT), (DEiT, RoBERTa), (ViT, ALBERT), (ViT, BERT), and (ViT, RoBERTa).

Since the DAQUAR dataset features simple vocabulary tokens as answers, we reformulate the task as a classification problem, using a linear layer as the "Prediction Head," where the output dimension matches the vocabulary size, and applying cross-entropy loss. Unlike the Baseline, which fine-tunes the entire pretrained modality-specific encoders, OneEncoder freezes these encoders and focuses solely on training the UP module. The goal of this application, using the smaller DAQUAR dataset, is to demonstrate that our framework can achieve strong performance with limited paired data, significantly reducing the number of parameters

to optimize and shortening the training time required for convergence. We use four Transformer blocks with a total of 4M parameters for the UP module, and modality tokens of size $R^{1×768}$. All models are trained for 100 epochs without any data augmentation techniques.

Figure 6 provides a detailed comparison of the number of trainable parameters between Baseline models and OneEncoder variants. Specifically, OneEncoder-1 utilizes addition-based fusion, while OneEncoder-2 employs an attention-based fusion mechanism. Unlike the Baseline models, which train all parameters, the OneEncoder versions use Baseline models for feature extraction but keep them frozen during training.

Figure 7 demonstrates that the OneEncoder architecture (OneEncoder-1, OneEncoder-2) consistently outperforms baseline models across the three key metrics: Wu-Palmer Similarity (WUPS) [85], Accuracy, and F1 Score. These results indicate that retraining specialized encoders may not be essential for achieving strong performance. By
<figure><img src="image_10.png" /><figcaption>**Fig. 8:** Example VQA Results Using the OneEncoder-1 Model.</figcaption></figure>

freezing the encoders and only training the UP on a small paired dataset, we can significantly reduce the number of parameters to optimize, minimize the need for large datasets, and shorten training times—all while yielding superior outcomes as illustrated in Figure 8.

The VQA experiment further validates the findings in Section IV, focused on contrastive learning. OneEncoder, with its efficient and lightweight design, can be effectively integrated into any alignment-based approach, reducing parameter complexity, data requirements, and surpassing traditional methods that rely on retraining modality-specific encoders.

## VI. CONCLUSION

In this paper, we introduce a novel approach to multimodal representation learning: OneEncoder. Our method aims to reduce the training cost of multimodal systems by minimizing the number of parameters and the reliance on large datasets. OneEncoder leverages pretrained modality-specific encoders as fixed feature extractors, while only a lightweight, shared Universal Projection (UP) module is trained across all modalities. A modality token is incorporated before each representation to ensure consistent multimodal mapping within the UP.

For contrastive learning, OneEncoder aligns text and image embeddings within the same projection space, demonstrating superior performance on smaller datasets compared to more resource-intensive models like CLIP,

which require separate training for each modality. To extend this framework to additional modalities such as audio and video, we propose a progressive alignment strategy. Here, the UP remains fixed while a compact Alignment Layer is introduced to map the output of modality-specific feature extractors into a UP-compatible space, providing a highly flexible and scalable solution without the need for large-scale retraining.

We further demonstrate OneEncoder's versatility through its application to visual question answering (VQA), where it achieves better performance and lower training costs compared to baseline models.

In conclusion, OneEncoder offers a scalable and parameter-efficient approach for multimodal representation learning, significantly reducing the need for extensive datasets while maintaining strong performance. Future work will explore its application to tasks like open-vocabulary object detection, where the model identifies relevant concepts based on rich semantic prompts before performing object detection.

## REFERENCES

[1] T. Baltrus'aitis, C. Ahuja, and L.-P. Morency, "A review of multimodal fusion for video understanding," *International Journal of Computer Vision*, vol. 128, pp. 1039-1052, 2020.
[2] T. Chen, Y. Xu, T. Luo, C. Li, Y. Xu, and D. Tao, "Multimodal sentiment analysis with word-level fusion and reinforcement learning," in *Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI)*, pp. 6094-6101, 2018.
[3] N. Srivastava and R. R. Salakhutdinov, "Multimodal learning with deep boltzmann machines," in Advances in neural information processing systems, pp. 2222–2230, 2012.

[4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding," Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pp. 4171–4186, 2019.

[5] OpenAI, "Gpt-4 technical report," 2024.

[6] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, B. Steiner, M. Hein, H. Touvron, A. Hervieu, et al., "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint arXiv:2010.11929, 2020.

[7] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," 2021.

[8] J. Li, T. N. Sainath, K. C. Sim, Y. Wu, and Z. Yang, "Jasper: An end-to-end convolutional neural acoustic model," in Proceedings of the Interspeech, pp. 71–75, 2019.

[9] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, "wav2vec 2.0: A framework for self-supervised learning of speech representations," Advances in neural information processing systems, vol. 33, pp. 12449–12460, 2020.

[10] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., "Learning transferable visual models from natural language supervision," arXiv preprint arXiv:2103.00020, 2021.

[11] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, "Zero-shot text-to-image generation," arXiv preprint arXiv:2102.12092, 2021.

[12] A. Guzhov, F. Raue, J. Hees, and A. Dengel, "Audioslip: Extending clip to image, text and audio," arXiv preprint arXiv:2106.13043, 2021.

[13] H. Akbari, L. Yuan, R. Qian, W.-H. Chuang, S.-F. Chang, Y. Cui, and B. Gong, "Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text," arXiv preprint arXiv:2104.11178, 2021.

[14] J. Han, K. Gong, Y. Zhang, J. Wang, K. Zhang, D. Lin, Y. Qiao, P. Gao, and X. Yue, "OneLLM: One Framework to Align All Modalities with Language," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[15] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, "Minigpt-4: Enhancing vision-language understanding with advanced large language models," 2023.

[16] H. Liu, C. Li, Q. Wu, and Y. J. Lee, "Visual instruction tuning," 2023.

[17] J. Li, D. Li, S. Savarese, and S. Hoi, "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models," arXiv preprint arXiv:2301.12597, 2023.

[18] W. Yin, E. Sheng, and Y.-G. Jiang, "Flamingo: Learning a vision-language embedding for Browsing fashion images," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13546–13555, 2020.

[19] L. Zhang, M. A'. Carreira-Perpinha', and A. Lavie, "Convirt: Contrastive visual representation learning for medical imaging," IEEE Transactions on Medical Imaging, vol. 41, no. 2, pp. 519–531, 2022.

[20] J. Chung, B. Zoph, A. Srinivas, L. Zhang, W. Yang, B. Zhang, A. Howard, M. A'. Carreira-Perpinha', and E. D. Cubuk, "Align: Activating latent innovations for multimodal contrastive learning," in Advances in Neural Information Processing Systems, vol. 34, 2021.

[21] N.-C. Ristea, A. Anghel, and R. T. Ionescu, "Cascaded cross-modal transformer for audio-textual classification," 2024.

[22] Z. Wang, C. Subakan, K. Subramani, J. Wu, T. Tavares, F. Ayres, and P. Smaragdis, "Unsupervised improvement of audio-text cross-modal representations," 2023.

[23] H. Chung, J. Shim, and J.-K. Kim, "Cross-modal contrastive representation learning for audio-to-image generation," 2022.

[24] H. Jiang, J. Zhang, R. Huang, C. Ge, Z. Ni, J. Lu, J. Zhou, S. Song, and G. Huang, "Cross-modal adapter for text-video retrieval," 2022.

[25] R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra, "Imagebind: One embedding space to bind them all," 2023.

[26] S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua, "Next-gpt: Any-to-any multimodal llm," 2023.

[27] S. Mirchandani, F. Xia, P. Florence, B. Ichter, D. Driess, M. G. Arenas, K. Rao, D. Sadigh, and A. Zeng, "Large language models as general pattern machines," 2023.

[28] L. Zhou, Y. Wang, H. Zhou, H. Lin, H. Ling, and G. Hu, "Cross-modal attention network for image and text retrieval," in Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pp. 2379–2385, 2022.

[29] Y. Zhang, K. Gong, K. Zhang, H. Li, Y. Qiao, W. Ouyang, and X. Yue, "Meta-transformer: A unified framework for multimodal learning," 2023.

[30] Z. Tong, Y. Song, J. Wang, and L. Wang, "Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training," 2022.

[31] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in neural information processing systems, pp. 5998–6008, 2017.

[32] C.-Y. Wang, I.-H. Yeh, and H.-Y. M. Liao, "You only learn one representation: Unified network for multiple tasks," arXiv preprint arXiv:2105.04206, 2021.

[33] J. Smith and et al., "Multimodal fusion techniques for image and text data," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

[34] A. Johnson and et al., "Cross-modal attention mechanisms for multimodal embedding," in Advances in Neural Information Processing Systems (NeurIPS), 2019.

[35] A. van den Oord and et al., "Representation learning with contrastive predictive coding," arXiv preprint arXiv:1807.03748, 2018.

[36] J. Lu, V. Goswami, D. Parikh, and S. Lee, "Virtex: Learning visual representations from textual annotations," in European Conference on Computer Vision, pp. 183–199, Springer, 2020.

[37] A. Karpathy and L. Fei-Fei, "Deep visual-semantic alignments for generating image descriptions," 2015.

[38] K. Xu and et al., "Show, attend and tell: Neural image caption generation with visual attention," in Proceedings of the International Conference on Machine Learning (ICML), 2015.

[39] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick, "Microsoft coco captions: Data collection and evaluation server," 2015.

[40] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, "From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions," in Transactions of the Association for Computational Linguistics, vol. 2, pp. 67–78, 2014.

[41] N. Sidorov, E. Mallas, Y. Liao, A. Potechin, S. Vempala, X. Luo, M. Bansal, and C. L. Zitnick, "Textcaps: a dataset for image captioning with reading comprehension," in European Conference on Computer Vision, pp. 742–758, Springer, 2020.

[42] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, "Librispeech: An asr corpus based on public domain audio books," in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206–5210, IEEE, 2015.

[43] J. Xu, T. Mei, T. Yao, and Y. Rui, "Msr-vtt: A large video description dataset for bridging video and language," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288–5296, 2016.

[44] A. Krizhevsky and G. Hinton, "Cifar-10 (canadian institute for advanced research)," tech. rep., Technical Report, 2009.

[45] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar, "Cats and dogs," in 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3498–3505, IEEE, 2012.

[46] A. Krizhevsky and G. Hinton, "Cifar-100 (canadian institute for advanced research)," tech. rep., Technical Report, 2009.

[47] L. Fei-Fei, R. Fergus, and P. Perona, "Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories," in IEEE. CVPR Workshop on Generative-Model Based Vision, 2004.

[48] Y. Le and X. Yang, "Tiny imagenet visual recognition challenge," in CS231N course, Stanford, 2015.

[49] R. Socher, A. Perelygin, J. Y. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts, "Recursive deep models for semantic compositionality over a sentiment treebank," in Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631–1642, Citeseer, 2013.

[50] E. M. Voorhees and D. K. Harman, "The trec-8 question answering track report," in TREC, 1999.

[51] E. Saravia, H.-C. T. Liu, Y.-C. Huang, J. Wu, and Y. Chen, "Carer: Contextualized affect representations for emotion recognition," in
*Proceedings of the 2018 conference on empirical methods in natural language processing*, pp. 3687–3697, 2018.

[52] G. Tzanetakis and P. Cook, “Musical genre classification of audio signals,” 2002.

[53] J. Salamon, C. Jacoby, and J. P. Bello, “A dataset and taxonomy for urban sound research,” in *Proceedings of the 22nd ACM international conference on Multimedia*, pp. 1041–1044, 2014.

[54] K. J. Piczak, “Environmental sound classification with convolutional neural networks,” *Proceedings of the IEEE 25th International Workshop on Machine Learning for Signal Processing (MLSP)*, pp. 1–6, 2015.

[55] D. Chen and W. B. Dolan, “Collecting highly parallel data for paraphrase evaluation,” in *Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies*, pp. 190–200, 2011.

[56] Y. Liu, S. Albanie, A. Nagrani, and A. Zisserman, “Use what you have: Video retrieval using representations from collaborative experts,” 2020.

[57] M. Malinowski and M. Fritz, “A multi-world approach to question answering about real-world scenes based on uncertain input,” in *Advances in Neural Information Processing Systems (NeurIPS)*, pp. 1682–1690, 2014.

[58] I. J. Orton, “Vision based body gesture meta features for affective computing,” *arXiv preprint arXiv:2003.00809*, 2020.

[59] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” *arXiv preprint arXiv:1711.05101*, 2019.

[60] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2015.

[61] M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” 2020.

[62] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” 2021.

[63] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert approach,” *arXiv preprint arXiv:1907.11692*, 2019.

[64] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” 2020.

[65] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” 2020.

[66] K. J. Piczak, “Environmental sound classification with convolutional neural networks,” in *2015 IEEE 25th international workshop on machine learning for signal processing (MLSP)*, pp. 1–6, IEEE, 2015.

[67] J. Salamon and J. P. Bello, “Deep convolutional neural networks and data augmentation for environmental sound classification,” *IEEE Signal processing letters*, vol. 24, no. 3, pp. 279–283, 2017.

[68] A. Guzhov, F. Raue, J. Hees, and A. Dengel, “Esresnet: Environmental sound classification based on visual domain models,” in *2020 25th international conference on pattern recognition (ICPR)*, pp. 4933–4940, IEEE, 2021.

[69] Y. Gong, Y.-A. Chung, and J. Glass, “Ast: Audio spectrogram transformer,” *arXiv preprint arXiv:2104.01778*, 2021.

[70] S. Verbitskiy, V. Berikov, and V. Vyshegorodtsev, “Eranns: Efficient residual audio neural networks for audio pattern recognition,” *Pattern Recognition Letters*, vol. 161, pp. 38–44, 2022.

[71] Y. Ma, G. Xu, X. Sun, M. Yan, J. Zhang, and R. Ji, “X-clip: End-to-end multi-grained contrastive learning for video-text retrieval,” 2022.

[72] L. A. Hendricks, O. Wang, E. Shechtman, J. Sivic, T. Darrell, and B. Russell, “Localizing moments in video with natural language,” in *Proceedings of the IEEE international conference on computer vision*, pp. 5803–5812, 2017.

[73] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in *International conference on machine learning*, pp. 1597–1607, PMLR, 2020.

[74] X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,” *arXiv preprint arXiv:2003.04297*, 2020.

[75] E. Amrani, R. Ben-Ari, D. Rotman, and A. Bronstein, “Noise estimation using density estimation for self-supervised multimodal learning,” *arXiv preprint arXiv:2003.03186*, vol. 8, 2020.

[76] M. Bain, A. Nagrani, G. Varol, and A. Zisserman, “Frozen in time: A joint video and image encoder for end-to-end retrieval,” *arXiv preprint arXiv:2104.00650*, 2021.

[77] V. Gabeur, C. Sun, K. Alahari, and C. Schmid, “Multimodal transformer for video retrieval,” in *Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16*, pp. 214–229, Springer, 2020.

[78] H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan, and T. Li, “Clip4clip: An empirical study of clip for end to end video clip retrieval,” *arXiv preprint arXiv:2104.08860*, 2021.

[79] J. Dong, X. Li, C. Xu, S. Ji, Y. He, G. Yang, and X. Wang, “Dual encoding for zero-example video retrieval,” in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 9346–9355, 2019.

[80] M. Dzabraev, M. Kalashnikov, S. Komkov, and A. Petiushko, “Mdmmt: Multidomain multimodal transformer for video retrieval,” in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 3354–3363, 2021.

[81] G. Bertasius, H. Wang, and L. Torresani, “Is space-time attention all you need for video understanding?,” *arXiv preprint arXiv:2102.05095*, 2021.

[82] H. Bao, L. Dong, S. Piao, and F. Wei, “Beit: Bert pre-training of image transformers,” in *International Conference on Learning Representations (ICLR)*, 2022.

[83] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Je'gou, “Training data-efficient image transformers and distillation through attention,” in *International Conference on Machine Learning (ICML)*, 2021.

[84] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representations,” in *International Conference on Learning Representations (ICLR)*, 2020.

[85] Z. Wu and M. Palmer, “Verbs semantics and lexical selection,” in *Proceedings of the 32nd annual meeting on Association for Computational Linguistics*, pp. 133–138, Association for Computational Linguistics, 1994.