Enhancing Multimodal Unified Representations for Cross Modal Generalization
Authors: Hai Huang, Yan Xia, Shengpeng Ji, Shulei Wang, Hanting Wang, Minghui Fang, Jieming Zhu, Zhenhua Dong, Sashuai Zhou, Zhou Zhao
Accepted at: acl25

Enhancing Multimodal Unified Representations for Cross Modal Generalization

Hai Huang1†, Yan Xia1†, Shengpeng Ji1, Shulei Wang1, Hanting Wang1, Minghui Fang1, Jieming Zhu2*, Zhenhua Dong2, Sashuai Zhou1, Zhou Zhao1‡

1Zhejiang University, 2Huawei Noah’s Ark Lab
haihuangcode@outlook.com

Abstract

To enhance the interpretability of multimodal unified representations, many studies have focused on discrete unified representations. These efforts typically start with contrastive learning and gradually extend to the disentanglement of modal information, achieving solid multimodal discrete unified representations. However, existing research often overlooks two critical issues: 1) The use of Euclidean distance for quantization in discrete representations often overlooks the important distinctions among different dimensions of features, resulting in redundant representations after quantization; 2) Different modalities have unique characteristics, and a uniform alignment approach does not fully exploit these traits. To address these issues, we propose Training-free Optimization of Codebook (TOC) and Fine and Coarse cross-modal Information Disentangling (FCID). These methods refine the unified discrete representations from pretraining and perform fine- and coarse-grained information disentanglement tailored to the specific characteristics of each modality, achieving significant performance improvements over previous state-of-the-art models. The code is available at https://github.com/haihuangcode/CMG.

1 Introduction

Humans’ capacity to integrate multimodal information, such as text, audio, and visual, has inspired research on extracting unified information from multimodal data (Harwath et al., 2018; Miech et al., 2019; Shvetsova et al., 2022; Monfort et al., 2021). Researchers aim to develop models that learn unified representations across modalities, using techniques like contrastive learning to map semantically similar multimodal data closer in the embedding space (Radford et al., 2021; Luo et al., 2022; Xu et al., 2021), achieving notable results in downstream tasks like zero-shot cross-modal retrieval. However, the unbounded nature of the continuous embedding space poses challenges in interpretability. To address this, recent works have explored constructing discrete embedding spaces with prototypes or codebooks, enhancing cross-modal learning and model interpretability (Liu et al., 2021a; Lu et al., 2022; Zhao et al., 2022; Xia et al., 2024).

While recent works has demonstrated incredible achievements in multimodal unified representation, there are limitations in terms of the efficiency of embedding space utilization and the granularity of alignment. 1) According to previous work (Breiman, 2001; Wojtas and Chen, 2020), the significance of features varies across different dimensions, and selecting the appropriate dimensions can optimize the feature space, thereby speeding up inference and improving model performance. However, existing multimodal unified representation methods, whether through contrastive learning (Liu et al., 2021a), teacher-student distillation (Duan et al., 2022), or information disentanglement (Xia et al., 2024; Huang et al., 2025), overlook this issue due to the inherent constraints of the codebook and the quantization method based on Euclidean distance. 2) In the unified discrete representation of multimodal data, some studies focus on coarse-grained semantic alignment (Duan et al., 2022), others on fine-grained alignment (Xia et al., 2024), and yet others consider both fine and coarse alignments simultaneously (Liu et al., 2021a). However, these approaches align text with audiovisual data in the same granularity, overlooking the inherent differences between modalities: audiovisual data have temporal fine-grained connections, whereas text represents holistic semantics.

To address the aforementioned issues, we propose two techniques: TOC and FCID.

1) Training-free Optimization of Codebook (TOC), inspired by training-free adapters (Zhang
Figure 1: (a) Pretrained multimodal unified discrete representation. (b) vanilla downstream experiments using the quantized code from the unified representation. (c) After refinement with TOC, downstream experiments are conducted using only a subset of the dimensions.

Figure 2: Multimodal Unified Representation of Vanilla method, DCID, FCID.

et al., 2022; Zhu et al., 2023) and drawing on the concept of feature importance (Breiman, 2001; Wojtas and Chen, 2020; Xue et al., 2022; Zhu et al., 2023), as illustrated in Figure 1, enables the use of a refined code for downstream tasks compared to the conventional approach, which directly uses the full quantized code during inference. The TOC calculation is independent of downstream tasks and does not require additional training. It only needs to be computed once, and the resulting dimensions can be reused for all subsequent inferences.

2) We adjust the order of multimodal alignment based on the inherent differences between text and audio-visual modalities and propose a Fine and Coarse Cross-modal Information Disentangling (FCID) architecture. As shown in Figure 2, vanilla methods for learning multimodal unified representations often retain modality-specific information. DCID (Xia et al., 2024) introduces disentanglement to separate modality-specific features; however, it does so by repeatedly aligning and decoupling pairs of modalities. For example, when aligning and disentangling audio and text, residual information such as at and avt is left behind. Similarly, when aligning audio and video, residuals like av and avt remain. These residuals, at and av, suggest that the disentanglement process is not fully effective. In contrast, FCID first aligns and disentangles audio and video while retaining fine-grained av and avt temporal information. It then performs coarse-grained disentangling with text, preserving the shared avt information, resulting in a more unified multimodal representation. The main contributions are summarized as follows:

• We propose TOC, a novel method for accurately identifying the importance of feature dimensions through efficient calculations, without the need for additional training. This versatile approach can be seamlessly applied to both multimodal unified and single-modal codebooks, offering a promising and adaptable solution.

• We introduce FCID, which disentangles information based on the distinct characteristics of text and audiovisual modalities, preserving both temporal and semantic information across all three modalities, and achieved a more unified multimodal representation.

• Our method outperforms the state-of-the-art across various cross-modal generalization tasks. Specifically, FCID, TOC, and their combination improve upon the SOTA by 2.16%, 1.06%, and 2.96%, respectively, across four downstream tasks. Additionally, we validated our approach on zero-shot cross-modal retrieval and cross-modal generation tasks.

2 Related Work

Multi-Modal Unified Representation: Recent work on multi-modal unified representations includes approaches that align modalities into a shared latent space (Petridis et al., 2018; Sarkar and Etemad, 2022; Andonian et al., 2022) and train modal-general encoders for cross-modal extraction (Chen et al., 2020b; Wang et al., 2022). Cross-modal distillation enables knowledge transfer between modalities (Sarkar and Etemad, 2022; Pedersoli et al., 2022), while bridging techniques connect representation spaces for improved unified representations (Wang et al., 2023). To enhance interpretability, many works use codebooks or prototypes (Duan et al., 2022; Lu et al., 2022; Liu et al., 2021a; Zhao et al., 2022; Xia et al., 2024; Huang et al., 2024; Fang et al., 2024). For instance, Duan et al. (2022) applies Optimal Transport to map features to prototypes, Xia et al. (2024) maps multimodal sequences to a common discrete semantic space. Our FCID framework addresses the inherent differences between text and audio-visual modalities through decoupling, enhancing multi-modal unified representations.
Training Free Optimization: Recent works have explored various training-free methods to boost model performance. Tip-Adapter(Zhang et al., 2022) and APE(Zhu et al., 2023) enhance CLIP’s few-shot classification, while a training-free method for diffusion models (Chen et al., 2024a) optimizes time steps and architecture for efficient image generation. The FuseDream (Liu et al., 2021b) combines CLIP and GANs for robust text-to-image generation. TEEN (Wang et al., 2024) offers a training-free solution for few-shot class-incremental learning. SCG-Diffusion (Wang et al., 2025) proposes a novel training-free method to improve alignment in Transformer-based Text-Guided Diffusion Models. Recent research in video generation (Chen et al., 2024b; Yang et al., 2024; Peng et al., 2024; Zhang et al., 2023) and multi-modal large language models (Wu et al., 2024) also focuses on training-free techniques. We introduce TOC, the first training-free optimization method for discrete representation, broadening the scope of training-free approaches in the field.

3 Backgroud

Cross Modal Generalization (CMG) is a task introduced by Xia et al. (2024) that evaluates a model’s ability to map diverse modalities into a unified discrete latent space. The model’s ability for cross-modal zero-shot knowledge transfer is assessed in a setup where training is conducted on modality \( m_1 \) and testing is performed on modality \( m_2 \).

During training, the model learns a representation for inputs from one modality using the encoder \( \Phi^{m_1} \) and the downstream decoder D:

\[
\mathbf{E}(\mathbf{D}(VQ(\Phi^{m_1}(\mathbf{x}_i^{m_1}))), \mathbf{y}_i^{m_1}),
\]

where \( \mathbf{x}_i^{m_1} \) is the input, \( \mathbf{y}_i^{m_1} \) is the label, and \( \mathbf{E} \) is the evaluation function. During testing, the model is evaluated on a different modality \( m_2 \), demonstrating its ability to generalize:

\[
\mathbf{E}(\mathbf{D}(VQ(\Phi^{m_2}(\mathbf{x}_i^{m_2}))), \mathbf{y}_i^{m_2}).
\]

Here, \( m_1, m_2 \in a, b, c \) and \( m_1 \neq m_2 \). The parameters of both \( \Phi^{m_1} \) and \( \Phi^{m_2} \) are parameters frozen during training and testing, while only the parameters of \( \mathbf{D} \) are updated during training.

4 Method

4.1 Training-free Optimization Codebook

Discrete unified representation spaces commonly employ a codebook structure, where modalities are updated based on the Euclidean distance between their features and the codebook codes. This dimension-equal-weighted update strategy does not consider the varying importance of feature dimensions, leading to redundancy in the final discrete space. According to previous work (Breiman, 2001; Wojtas and Chen, 2020), the importance of features varies across different dimensions. Eliminating redundant dimensions can help improve performance and accelerate computation. Therefore, we propose two metrics, Code Similarity and Code Variance, to refine features in the unified space.

Code Similarity: This metric aims to enhance the distinctiveness of codes by extracting feature dimensions that minimize code similarity. We represent the unified representation codebook of modalities as \( \mathbf{e} \in \mathbb{R}^{H \times D} \), where \( H, D \) denote the size of the codebook and hidden dimension, respectively.

Assuming the existence of a classification dataset with \( C \) categories, acquiring its complete data enables the calculation of the average similarity, denoted as \( S \). In an open-world setting, we may assume that the prior probabilities of all categories are equal, denoted as \( \frac{1}{C} \). We adopt cosine similarity, \( \delta(\cdot, \cdot) \), as the chosen metric:

\[
S = \frac{1}{C^2} \sum_{i=1}^C \sum_{j=1}^C \frac{1}{N^i N^j} \sum_{m=1}^{N^i} \sum_{n=1}^{N^j} \delta(\mathbf{x}_i^{m}, \mathbf{x}_j^{n}),
\]

(3)

where \( \mathbf{x}_i^{m} \) and \( \mathbf{x}_j^{n} \) denote the input features for the \( m \)-th and \( n \)-th samples of categories \( i \) and \( j \), respectively. \( N^i \) and \( N^j \) represent their respective total number of training samples.

Each code in the pretrained codebook, \( \mathbf{e}^i \in \mathbb{R}^D, i \in [0, H] \), can be considered as a distinct semantic cluster center, representing a category. Therefore, we can simplify the average similarity calculation:

\[
S = \frac{1}{H^2} \sum_{i=1}^H \sum_{j=1, j \neq i}^H \delta(\mathbf{e}^i, \mathbf{e}^j),
\]

(4)

Our goal is to select \( Q \) dimensions out of \( D \) to enhance the distinctiveness of the codes. We introduce a binary flag \( \mathbf{F} \in \{0, 1\}^D \), where \( F_k = 1 \) (\( k = 1, ..., D \)) indicates that the \( k^{th} \) dimension \( \mathbf{e}_k^i \) is selected, and \( \mathbf{F}^\top = Q \). Our objective now
becomes finding the optimal \( \mathbf{F} \) to minimize the Code Similarity:

\[
\min_{\mathbf{F}} \quad S = \frac{1}{H^2} \sum_{i=1}^{H} \sum_{j=1, j \neq i}^{H} \delta(\mathbf{e}^i \odot \mathbf{F}, \mathbf{e}^j \odot \mathbf{F}), \tag{5}
\]

where \( \odot \) denotes element-wise multiplication.

We further suppose the Codebook has been L2-normalized, meaning that each code vector \( \mathbf{e}^i \in \mathbb{R}^D \) has a unit length. Under this assumption, the cosine similarity between two code vectors \( \mathbf{e}^i \) and \( \mathbf{e}^j \) can be simplified as their dot product:

\[
\delta(\mathbf{e}^i, \mathbf{e}^j) = \mathbf{e}^i \cdot \mathbf{e}^j, \tag{6}
\]

where \( \cdot \) denotes the dot product of two vectors. Then we can simplify the cosine similarity as

\[
S = \sum_{k=d_1}^{d_Q} S_k = \sum_{k=d_1}^{d_Q} \left( \frac{1}{H^2} \sum_{i=1}^{H} \sum_{j=1, j \neq i}^{H} \mathbf{e}_k^i \cdot \mathbf{e}_k^j \right), \tag{7}
\]

where \( k = \{ d_1, d_2, ..., d_Q \} \) denotes the indices of selected feature dimensions with \( F_k = 1 \), and \( S_k = \frac{1}{H^2} \sum_{i=1}^{H} \sum_{j=1, j \neq i}^{H} \mathbf{e}_k^i \cdot \mathbf{e}_k^j \) represents the average inter-class similarity of the \( k^{th} \) dimension. Through straightforward derivation, we observe that solving the optimization problem is equivalent to selecting Q elements with the smallest average similarity.

**Code Variance:** Our goal is to reduce redundancy by removing feature dimensions with low variance across codewords, as these dimensions offer minimal discriminative value. The variance for the \( k^{th} \) feature dimension is formulated as:

\[
V_k = \frac{1}{L} \sum_{i=1}^{L} (\mathbf{e}_k^i - \bar{\mathbf{e}}_k)^2, \tag{8}
\]

where \( \bar{\mathbf{e}}_k = \frac{1}{L} \sum_{i=1}^{L} \mathbf{e}_k^i \) represents the mean of the \( k^{th} \) dimension across all codewords. Similar to Code Similarity, we select the top \( Q \) dimensions with the highest variance to enhance discriminative power.

To combine the criteria of similarity and variance, a balance factor \( \lambda \) is introduced to compute the final metric for each feature dimension:

\[
U_k = \lambda V_k - (1-\lambda) S_k, \tag{9}
\]

where \( k = 1, \ldots, D \). The dimensions corresponding to the top-Q biggest values of \( U_k \) are chosen as the refined features.

4.2 Fine and Coarse cross-modal Information Disentangling

As shown in Figure 3, the Vanilla method uses contrastive learning (Liu et al., 2021a) or distillation (Duan et al., 2022) for cross-modal representation unification. DCID (Xia et al., 2024) introduces decoupling to separate modality-specific information. In FAC (Alayrac et al., 2020), audio and video are first aligned temporally and then with text for semantic alignment. We propose FCID, which combines FAC and DCID. It decouples and aligns audio and video, isolates modality-specific data, and then further aligns with text, discarding non-shared information for a unified multimodal representation. FCID is the first method to simultaneously tackle both modality differences and redundancy issues, inspired by FAC and DCID.

4.2.1 Fine cross-modal Information Disentangling

Given paired audio-video modalities, \( (\mathbf{x}_i^m, \mathbf{x}_i^n)_{i=1}^N \), we utilize two fine modal-general encoders, \( \Phi_g^m \) and \( \Phi_g^n \), to extract fine modal-general features \( \mathbf{f}_i^m \) and \( \mathbf{f}_i^n \in \mathbb{R}^{T \times D} \), and employ two fine modal-specific encoders, \( \Psi_f^m \) and \( \Psi_f^n \), to obtain fine modal-specific features \( \overline{\mathbf{f}}_i^m \) and \( \overline{\mathbf{f}}_i^n \in \mathbb{R}^{T \times D} \) from the audio and video modalities, respectively. Here, \( N, T, \) and \( D \) represent the number of samples, the length of audio-video sequences, and the feature dimension, respectively. In subsequent equations, \( m, n \in \{audio, video\} \):

\[
\mathbf{f}_i^m = \Phi_g^m(\mathbf{x}_i^m), \quad \overline{\mathbf{f}}_i^m = \Psi_f^m(\mathbf{x}_i^m).
\] (10)

Subsequently, we minimize the mutual information between the fine modal-specific features \( \mathbf{f}_i^m \) and \( \overline{\mathbf{f}}_i^m \). At the same time, maximize the mutual information between \( \mathbf{f}_i^m \) and \( \overline{\mathbf{f}}_i^n \). The details of this approach are outlined below:

**Mutual Information Minimization:** CLUB (Cheng et al., 2020) could optimize the mutual information upper bound, demonstrating superior advantages in information disentanglement. Given two variables x and y, the objective function of CLUB is defined as:

\[
I_{vCLUB}(x; y) := \mathbb{E}_{p(x,y)}[\log q_\theta(y|x)] \\
- \mathbb{E}_{p(x)}\mathbb{E}_{p(y)}[\log q_\theta(y|x)]. \tag{11}
\]

We use CLUB to optimize the mutual information upper bound between fine modal-general features \( \mathbf{f}_i^m \) and fine modal-specific features \( \overline{\mathbf{f}}_i^m \),
Figure 3: (a) A simple demonstration of four models: Vanilla method, FAC, DCID, and our proposed FCID. (b) Details of the FCID encoder. On the left, audio and video undergo fine-grained mutual information separation and alignment using modal-general encoders \( \Phi^g_f, \Phi^v_f \) and modal-specific encoders \( \Psi^g_f, \Psi^v_f \). The CLUB module separates specific information \( \tilde{\mathbf{f}}_i^g, \tilde{\mathbf{f}}_i^v \) from general information \( \mathbf{f}_i^g, \mathbf{f}_i^v \), while CrossCPC aligns the general information across modalities. This is followed by compressing the features into unified audiovisual representations. On the right, coarse-grained mutual information separation and alignment are conducted with audiovisual data and text, resulting in a unified discrete representation across all three modalities. The decoder combines the quantized code with modality-specific features and computes the reconstruction loss, omitted in the figure.

where \( q_\theta \) is the variational approximation of ground-truth posterior of y given x and can be parameterized by a network \( \theta \).

\[
\hat{I}_{vCLUB_f} = \frac{1}{N} \sum_{i=1}^N \left[ \frac{1}{T} \sum_{t=1}^T \log q_\theta(\tilde{\mathbf{f}}_i^m | \mathbf{f}_i^m) \right. \\
- \left. \frac{1}{N} \frac{1}{T} \sum_{j=1}^N \sum_{t=1}^T \log q_\theta(\tilde{\mathbf{f}}_j^m | \mathbf{f}_i^m) \right].
\]

Mutual Information Maximization: Contrastive Predictive Coding (CPC) (Oord et al., 2018) aims to maximize the mutual information between sequence items by predicting future samples using autoregressive models and is widely adopted in self-supervised learning. Given fine general features \( \mathbf{f}^g, \mathbf{f}^v \in \mathbb{R}^{T \times D} \), a prediction horizon of R steps, and a random time moment \( t \in (0, T-R] \), two single-layer unidirectional LSTMs are used to summarize the information of all \( \mathbf{f}_{<t}^g, \mathbf{f}_{<t}^v \), yielding three context representations as \( \mathbf{o}_t^m = \mathrm{LSTM}(\mathbf{f}_{<t}^m) \).

For modality M, we first select a set \( Z_{neg} \) of N-1 random negative samples and one positive sample \( \mathbf{f}_{t+r}^n \) from modality N, then use \( \mathbf{o}_t^m \) to predict r-th future step \( \mathbf{f}_{t+r}^n \) in modality N, and the loss for all modality can be optimized as:

\[
L_{cpc}^{m2n} = - \frac{1}{R} \sum_{r=1}^R \log \left[ \frac{\exp (\mathbf{f}_{t+r}^n W_r^m \mathbf{o}_t^m)}{\sum_{j \in Z_{neg}} \exp (\mathbf{f}_j^n W_r^m \mathbf{o}_t^m)} \right].
\]

4.2.2 Coarse Cross-modal Information Disentangling

CCID initially sets up two projectors, \( P^{te} \) for compressing textual features and \( P^{va} \) for compressing the audiovisual modal-general features obtained from FCID. Subsequently, it configures two coarse modal-specific encoders, \( \Psi_c^{av} \) and \( \Psi_c^{te} \), to extract coarse modal-specific features \( \overline{\mathbf{c}}_i^{av} \) and \( \overline{\mathbf{c}}_i^{te} \in \mathbb{R}^D \), and two coarse modal-general encoders, \( \Phi_c^{av} \) and \( \Phi_c^{te} \), are employed to derive coarse modal-general features \( \mathbf{c}_i^{av} \) and \( \mathbf{c}_i^{te} \in \mathbb{R}^D \) from the audiovisual and textual modalities, respectively. In subsequent equations, \( \mathcal{M}, \mathcal{N}, \in \{audiovisual, text\} \):

\[
\mathbf{c}_i^{\mathcal{M}} = \Phi_c^{\mathcal{M}}(P^{\mathcal{M}}(\mathbf{x}_i^{\mathcal{M}})),
\]
\[
\overline{\mathbf{c}}_i^{\mathcal{M}} = \Psi_c^{\mathcal{M}}(P^{\mathcal{M}}(\mathbf{x}_i^{\mathcal{M}})),
\]

The subsequent process of information disentanglement and alignment is similar to that of Fine cross-modal Information Disentangling.

Mutual Information Minimization: We use CLUB to optimize the mutual information upper bound between coarse modal-general features \( \mathbf{c}_i^{\mathcal{M}} \) and fine modal-specific features \( \overline{\mathbf{c}}_i^{\mathcal{M}} \), similar to \( \hat{I}_{vCLUB_f} \) in FCID:

\[
\hat{I}_{vCLUB_c} = \frac{1}{N} \sum_{i=1}^N \left[ \frac{1}{T} \sum_{t=1}^T \log q_\theta(\overline{\mathbf{c}}_i^{\mathcal{M}} | \mathbf{c}_i^{\mathcal{M}}) \right. \\
- \left. \frac{1}{N} \frac{1}{T} \sum_{j=1}^N \sum_{t=1}^T \log q_\theta(\overline{\mathbf{c}}_j^{\mathcal{M}} | \mathbf{c}_i^{\mathcal{M}}) \right].
\]
Mutual Information Maximization: Since the coarse information lacks a sequential structure, we transitioned the contrastive learning approach from CPC to InfoNCE, as described below:

\[
L_{nce} = - \frac{1}{N} \sum_{i=1}^{N} \log \left[ \frac{\exp(\mathrm{sim}(c_i^M, c_i^N)/\tau)}{\sum_{j=1}^{N} \exp(\mathrm{sim}(c_i^M, c_j^N)/\tau)} \right]
\]

(16)

4.3 Final Loss

Then, we use the codebook to explicitly represent the unified multimodal representation, the latent codebook \( e \in R^{H \times D} \) is shared across modalities audio, video, and text, where \( T, H, D \) represent time, size of the discrete latent space, and hidden dimension, respectively. Apply vector quantized operation to map coarse model-general feature \( f_i^{av}, f_i^{te} \) to discrete latent codes, \( t \in [0, T] \):

\[
\hat{c}_{i,t}^M = VQ(\Phi_c^M(x_i^M)) = VQ(c_{i,t}^M) = e_l,
\]
where \( l = argmin_j \| \Phi_c(x) - e_j \|_2 \).

(17)

Then, we combine \( \hat{c}_i^m \) with \( \bar{c}_i^m \) together to reconstruct original features:

\[
\frac{\| x_i^M - D(\hat{c}_i^M; \bar{c}_i^M) \|_2^2 + \beta \| \phi_k^M(x_i^M) - sg[e] \|_2^2 }{\text{reconstruction loss}} + \text{commitment loss}
\]

(18)

and employ Multimodal Exponential Moving Average (MMEMA) strategy to update codebook.

The overall objective of FCID is a combination of these loss functions across both layers:

\[
L = L_{\text{recon}} + L_{\text{commit}} + L_{\text{contra}} + L_{\text{club}},
\]

(19)

where \( L_{\text{recon}} \) is the reconstruction loss that merges the modal-specific and modal-general results for each modality and compares them with the original input using MSE loss, \( L_{\text{commit}} \) is the commitment loss that computes the MSE loss between the modal-general results and their quantized codes, \( L_{\text{contra}} = L_{\text{cpc}} + L_{\text{nce}} \) is the loss that enhances cross-modal alignment and inference by predicting future samples in one modality using information from another, and \( L_{\text{club}} = I_{cCLUB_f} + I_{cCLUB_e} \) represents the mutual information loss concerning the modal-specific and modal-general results within each modality.

5 Experiment

5.1 Datasets and Tasks

5.1.1 Pretrain

Multimodal Unified Representation: The pre-training dataset uses VGGSound-AVEL40K (Chen et al., 2020a; Zhou et al., 2022) with text from Xia et al. (2024). Single-modal Representation: We trained a VQVAE (Van Den Oord et al., 2017) on the CelebA-HQ 30K (Karras et al., 2017) dataset and evaluated TOC’s effect on selecting feature dimensions for reconstruction, assessing its transferability to other domains using the codebook.

5.1.2 Downstream

The unified representation pre-trained models will be evaluated on several downstream tasks using different datasets. Cross-modal event classification on AVE dataset: (Tian et al., 2018) training on one modality and evaluating on another. Cross-modal event localization on AVVP dataset: (Tian et al., 2020) localizing events in one modality and transferring to the other. Cross-dataset localization/classification: training on classification in AVE and evaluating localization in AVVP, transferring across datasets. Cross-modal classification between UCF-101 (Soomro et al., 2012) visual clips and VGGSound (Chen et al., 2020a) audio clips. The decoder in all of the above experiments consists of a single linear layer. Cross-modal Zero-shot Retrieval: We adopt a process similar to the test set (Yu et al., 2018), which consists of 500 pairs from MSCOCO (Chen and Dolan, 2011), assessing zero-shot retrieval capability for visual-text alignment. Clotho (Drossos et al., 2020) assesses zero-shot retrieval capability for audio-text alignment. Flickr Sound (Senocak et al., 2018) assesses zero-shot retrieval capability for audio-visual alignment. Cross-modal Generation: We use a 2-layer MLP and the IP-Adapter (Ye et al., 2023) as downstream decoders. The MLP is trainable during training but frozen during testing, while the IP-Adapter remains frozen throughout. By leveraging IP-Adapter’s image-to-image capability, the MLP bridges multi-modal unified representation and image generation, enabling audio-to-image and text-to-image generation during testing. The model was fine-tuned on 4,500 FlickrSound (Senocak et al., 2018) image-audio pairs over 80,000 steps with a batch size of 8, and evaluated on 500 additional pairs. Please refer to Appendix A for details on the compared works, evaluation metrics, and hyperparameter settings.
Figure 4: Example results of reconstructions using random and TOC masking.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">AVE</th>
    <th colspan="3">AVVP</th>
    <th colspan="3">UCF101+VOG16</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>V→A</th>
    <th>A→V</th>
    <th>V→A</th>
    <th>A→V</th>
    <th>V→A</th>
    <th>A→V</th>
    <th>V→A</th>
    <th>A→V</th>
    <th>V→A</th>
    <th>A→V</th>
  </tr>
  <tr>
    <td>CODIS</td>
    <td>36.8</td>
    <td>39.7</td>
    <td>32.7</td>
    <td>32.6</td>
    <td>40.8</td>
    <td>40.6</td>
    <td>50.8</td>
    <td>45.2</td>
    <td>39.9</td>
    <td>39.9</td>
  </tr>
  <tr>
    <td>TURN</td>
    <td>37.6</td>
    <td>39.5</td>
    <td>32.2</td>
    <td>32.5</td>
    <td>40.6</td>
    <td>41.1</td>
    <td>50.4</td>
    <td>46.1</td>
    <td>39.9</td>
    <td>39.9</td>
  </tr>
  <tr>
    <td>CMCM</td>
    <td>40.8</td>
    <td>41.5</td>
    <td>35.1</td>
    <td>35.2</td>
    <td>45.2</td>
    <td>45.2</td>
    <td>52.3</td>
    <td>44.1</td>
    <td>41.3</td>
    <td>41.3</td>
  </tr>
  <tr>
    <td>SimMMMDG</td>
    <td>49.5</td>
    <td>51.7</td>
    <td>39.3</td>
    <td>39.7</td>
    <td>52.9</td>
    <td>52.7</td>
    <td>64.5</td>
    <td>58.8</td>
    <td>51.4</td>
    <td>51.4</td>
  </tr>
  <tr>
    <td>DCID</td>
    <td>54.1</td>
    <td><b>55.0</b></td>
    <td>40.0</td>
    <td>40.8</td>
    <td>53.0</td>
    <td>52.4</td>
    <td>67.1</td>
    <td>60.6</td>
    <td>52.9</td>
    <td>52.9</td>
  </tr>
  <tr>
    <td>FCID</td>
    <td>55.2</td>
    <td>54.0</td>
    <td>42.6</td>
    <td>44.1</td>
    <td>55.2</td>
    <td>57.4</td>
    <td>69.4</td>
    <td>61.6</td>
    <td>56.0</td>
    <td>56.0</td>
  </tr>
  <tr>
    <td>CODES+TOC</td>
    <td>54.2</td>
    <td>54.3</td>
    <td>42.7</td>
    <td>43.9</td>
    <td>55.1</td>
    <td>56.3</td>
    <td>71.3</td>
    <td>64.0</td>
    <td>57.1</td>
    <td>57.1</td>
  </tr>
  <tr>
    <td>TURN+TOC</td>
    <td>38.3</td>
    <td>40.5</td>
    <td>33.2</td>
    <td>32.9</td>
    <td>41.5</td>
    <td>43.3</td>
    <td>51.5</td>
    <td>46.8</td>
    <td>41.00</td>
    <td>41.00</td>
  </tr>
  <tr>
    <td>CMCM+TOC</td>
    <td>46.9</td>
    <td>47.2</td>
    <td>37.9</td>
    <td>36.2</td>
    <td>49.8</td>
    <td>50.1</td>
    <td>52.3</td>
    <td>49.1</td>
    <td>46.19</td>
    <td>46.19</td>
  </tr>
  <tr>
    <td>DCID+TOC</td>
    <td>51.5</td>
    <td>55.0</td>
    <td>40.9</td>
    <td>41.6</td>
    <td>56.5</td>
    <td>53.6</td>
    <td>68.1</td>
    <td>61.7</td>
    <td>53.99</td>
    <td>53.99</td>
  </tr>
  <tr>
    <td>FCID+TOC</td>
    <td>55.9</td>
    <td>55.0</td>
    <td>43.6</td>
    <td>45.1</td>
    <td>57.4</td>
    <td>58.5</td>
    <td>62.0</td>
    <td>58.89</td>
    <td>58.89</td>
    <td>58.89</td>
  </tr>
</table>

Table 1: Comparison with SOTA Methods on four audio-visual tasks. (SimMMMDG represents recent great work in multimodal domain generalization, is incompatible with TOC as it does not use discrete representations.)

5.2 Performance Analysis

In the all tables, bold numbers indicate the best results, while green values in parentheses show the performance improvement attributed to the TOC.

TOC: As shown in Table 1 and Table 2, TOC optimizes methods with discrete representation spaces, facilitating at least a 0.80% improvement in average results for cross-modal generalization tasks, and a minimum average increase of 0.41% for cross-modal zero-shot retrieval tasks. Additionally, as illustrated in Table 4 and Figure 5, TOC excels in cross-modal generation tasks, improving image-to-image (\( I2I \)), audio-to-image (\( A2I \)), and text-to-image (\( T2I \)) generation performance.

We also explored extending TOC to unimodal discrete representation space. Using a VQVAE model trained on the CelebA-HQ 30K dataset (Karras et al., 2017), we tested reconstructions with a subset of the codeword dimensions. Table 3 shows that R100-avg represents the average outcome of 100 random codeword dimension selections for reconstruction, where TOC masks the least important dimensions. The MSE for TOC reconstructions with 25% to 87.5% of dimensions was significantly lower than the average MSE from random selections. 'Count' represents the number of times the MSE from 100 random selections is greater than the MSE from TOC. Figure 4 displays reconstruction samples, for all columns except the 'origin' column, with the left half showing reconstructions with randomly masked dimensions and the right half showing TOC-reconstructed images, demonstrating the superior effectiveness of TOC-selected dimensions. For more results, see Appendix B.5.

We also conducted experiments on the impact of TOC on the cosine similarity of the codebook. Please refer to Appendix B.1 for details.

FCID: Reviewing Tables 1 and 2 clearly shows that FCID and FCID+TOC consistently outperform all other methods across a variety of tasks. Compared to the previous SOTA, FCID achieves an average improvement of 2.16% in four cross-modal generalization tasks and an average improvement of 0.75% in three cross-modal zero-shot retrieval tasks. As shown in Table 4, these approaches also demonstrate a clear advantage in cross-modal generation tasks. All results suggest that our methods can more effectively process and understand cross-modal information.

As shown in Figure 5, the top row displays four image-audio pairs, and the three rows below show images generated from these samples. FCID, fine-tuned only with images, achieves A → I results that closely resemble I → I outcomes. Notably, in the last two examples, the generated images are identical, indicating that these image-audio pairs map to the same code in the codebook, demonstrating strong modal alignment. For additional T → I examples and results, see Appendix B.6.

We demonstrate multimodal quantization activations in the discrete representation spaces of DCID (Xia et al., 2024) and FCID using tri-modal audio-video-text data from VALOR32K (Chen et al., 2023). Figures 6 and 7 show the code activations in the DCID and FCID codebooks. Red points indicate single modality activations > 95%, green points show activation across all three modalities \( \geq 5\% \), and blue points fall between these categories. More green dots indicate closer alignment of discrete representations across modalities in the

Figure 5: Example results of cross-modal image-generation experiments conducted by FCID+TOC.

![Example results of cross-modal image-generation experiments conducted by FCID+TOC.](page_1012_120_627_180.png)
<table>
  <tr>
    <th>Method</th>
    <th>V→T (R@10)</th>
    <th>A→T (R@10)</th>
    <th>V→A (R@10)</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>CMCM</td>
    <td>7.20</td>
    <td>14.87</td>
    <td>13.60</td>
    <td>7.11</td>
  </tr>
  <tr>
    <td>DCID</td>
    <td>8.30</td>
    <td>16.70</td>
    <td>17.20</td>
    <td>8.14</td>
  </tr>
  <tr>
    <td>FCID</td>
    <td><b>9.60</b></td>
    <td><b>18.19</b></td>
    <td><b>17.50</b></td>
    <td><b>8.89</b></td>
  </tr>
  <tr>
    <td>CMCM+TOC</td>
    <td>7.70</td>
    <td>15.33</td>
    <td>16.10</td>
    <td>7.52 (<b>+0.41</b>)</td>
  </tr>
  <tr>
    <td>DCID+TOC</td>
    <td>8.80</td>
    <td>17.08</td>
    <td>17.80</td>
    <td>8.56 (<b>+0.42</b>)</td>
  </tr>
  <tr>
    <td>FCID+TOC</td>
    <td><b>10.40</b></td>
    <td><b>19.04</b></td>
    <td><b>18.40</b></td>
    <td><b>9.42 (<b>+0.53</b>)</b></td>
  </tr>
</table>

Table 2: Results on three cross-modal zero-shot retrieval tasks, averaged across two directions, with Avg. as the average of R@1, R@5, and R@10 (details in Table 10).

<table>
  <tr>
    <th>Mask (%)</th>
    <th>TOC↓</th>
    <th>Count↑</th>
  </tr>
  <tr>
    <td>87.5</td>
    <td>0.0621</td>
    <td>0.0231</td>
    <td>100</td>
  </tr>
  <tr>
    <td>75.0</td>
    <td>0.0477</td>
    <td>0.0159</td>
    <td>100</td>
  </tr>
  <tr>
    <td>62.5</td>
    <td>0.0335</td>
    <td>0.0109</td>
    <td>100</td>
  </tr>
  <tr>
    <td>50.0</td>
    <td>0.0229</td>
    <td>0.0086</td>
    <td>100</td>
  </tr>
  <tr>
    <td>37.5</td>
    <td>0.0141</td>
    <td>0.0062</td>
    <td>96</td>
  </tr>
  <tr>
    <td>25.0</td>
    <td>0.0075</td>
    <td>0.0039</td>
    <td>90</td>
  </tr>
</table>

Table 3: Reconstruction errors under random vs. TOC masking.

<table>
  <tr>
    <th>Method</th>
    <th>I2I↓</th>
    <th>A2I↓</th>
    <th>T2I↓</th>
  </tr>
  <tr>
    <td>CMCM</td>
    <td>129.56</td>
    <td>130.93</td>
    <td>148.93</td>
  </tr>
  <tr>
    <td>DCID</td>
    <td>121.44</td>
    <td>123.28</td>
    <td>141.16</td>
  </tr>
  <tr>
    <td>FCID</td>
    <td><b>116.06</b></td>
    <td><b>117.26</b></td>
    <td><b>135.52</b></td>
  </tr>
  <tr>
    <td>CMCM+TOC</td>
    <td>124.25</td>
    <td>125.37</td>
    <td>144.93</td>
  </tr>
  <tr>
    <td>DCID+TOC</td>
    <td>118.30</td>
    <td>119.96</td>
    <td>135.93</td>
  </tr>
  <tr>
    <td>FCID+TOC</td>
    <td><b>113.95</b></td>
    <td><b>115.14</b></td>
    <td><b>130.98</b></td>
  </tr>
</table>

Table 4: Cross-modal image generation results (lower is better).

codebook, while more red dots reflect greater divergence. FCID outperforms DCID in aligning the three modalities, with fewer codes activated by a single modality, demonstrating its improved ability to learn unified tri-modal representations.

For details on the computational efficiency and loss variation of FCID pre-training, please refer to Appendix B.2 and B.3, respectively.

5.3 Ablation Study

Loss: We conducted ablation experiments on all losses in Equation 19, excluding the contrastive loss, as it is fundamental to multimodal alignment in this work. Without contrastive learning, alignment cannot be achieved, rendering its ablation unnecessary. As shown in Table 5, we observe that \( L_{commit} \) has a minor impact on results, as its primary role is to align unified discrete representations with quantized features. In contrast, \( L_{recon} \) is crucial for ensuring semantic completeness after disentanglement, minimizing information incompleteness during this process. Finally, \( L_{club} \) directly evaluates the success of information disentangling, and its absence significantly degrades model performance.

<table>
  <tr>
    <th>Method</th>
    <th>V→A</th>
    <th>AVVP</th>
    <th>AVE→AVVP</th>
    <th>UCF(v)→VGG(a)</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Ours</td>
    <td>55.0</td>
    <td>45.6</td>
    <td>45.7</td>
    <td>57.4</td>
    <td>58.9</td>
  </tr>
  <tr>
    <td>w/o \( L_{club} \)</td>
    <td>51.3</td>
    <td>42.6</td>
    <td>45.7</td>
    <td>58.3</td>
    <td>57.1</td>
  </tr>
  <tr>
    <td>w/o \( L_{recon} \)</td>
    <td>53.3</td>
    <td>42.5</td>
    <td>43.3</td>
    <td>56.1</td>
    <td>57.5</td>
  </tr>
  <tr>
    <td>w/o \( L_{commit} \)</td>
    <td>54.6</td>
    <td>42.9</td>
    <td>44.7</td>
    <td>57.2</td>
    <td>56.8</td>
  </tr>
</table>

Table 5: Ablation studies on the impact of different loss.

TOC: As presented in Table 6, the two components of TOC, when applied individually, led to improvements in FCID performance by 0.54% and 0.10% across eight metrics, respectively. The combined effect of both components resulted in a total improvement of 0.80%. Refining the discrete representation space using either Code Similarity or Code Variance alone effectively reduces feature redundancy and enhances model performance in downstream tasks. However, Code Similarity demonstrates a greater impact than Code Variance. When both components are combined, the best performance is achieved, highlighting the effectiveness of our proposed TOC design.

<table>
  <tr>
    <th>S</th>
    <th>R</th>
    <th>AVE</th>
    <th>AVVP</th>
    <th>AVE→AVVP</th>
    <th>UCF(v)→VGG(a)</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>-</td>
    <td>-</td>
    <td>55.2</td>
    <td>54.9</td>
    <td>42.4</td>
    <td>44.5</td>
    <td>55.3</td>
    <td>57.4</td>
    <td>69.4</td>
    <td>61.6</td>
    <td>55.09</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>-</td>
    <td>55.8</td>
    <td>54.5</td>
    <td>43.6</td>
    <td>45.7</td>
    <td>56.8</td>
    <td>58.3</td>
    <td>69.2</td>
    <td>61.1</td>
    <td>55.63</td>
  </tr>
  <tr>
    <td>-</td>
    <td>✓</td>
    <td>55.6</td>
    <td>55.0</td>
    <td>43.4</td>
    <td>44.8</td>
    <td>56.2</td>
    <td>54.8</td>
    <td>69.8</td>
    <td>61.9</td>
    <td>55.19</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>55.9</td>
    <td>55.0</td>
    <td>43.6</td>
    <td>45.1</td>
    <td>57.4</td>
    <td>58.5</td>
    <td>69.6</td>
    <td>62.0</td>
    <td>55.89</td>
  </tr>
</table>

Table 6: Ablation studies on the impact of TOC (S and R represent Code Similarity and Code Variance, respectively)

FCID: We focus our ablation studies on the key disentanglement components, \( IvCLUB_f \) and \( IvCLUB_c \), which involve \( ACCLUB, VCCLUB \) and \( AVCLUB, TECLUB \), respectively. Table 7 shows that \( ACCLUB \) and \( VCCLUB \) significantly impact audiovisual-related downstream tasks. \( TECLUB \) also influences results, as improperly disentangled textual information can affect performance by containing irrelevant or missing AV data. Similarly, using only \( AVCLUB \) still retains some modality-specific information, but the disentanglement and alignment with text help separate these features.

<table>
  <tr>
    <th>ACCLUB</th>
    <th>VCCLUB</th>
    <th>AVCLUB</th>
    <th>TECLUB</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>50.71</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>52.79</td>
  </tr>
  <tr>
    <td>-</td>
    <td>✓</td>
    <td>-</td>
    <td>-</td>
    <td>53.40</td>
  </tr>
  <tr>
    <td>-</td>
    <td>-</td>
    <td>✓</td>
    <td>-</td>
    <td>51.78</td>
  </tr>
  <tr>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>✓</td>
    <td>51.59</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>-</td>
    <td>-</td>
    <td>54.34</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>-</td>
    <td>✓</td>
    <td>-</td>
    <td>52.46</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>55.09</td>
  </tr>
</table>

Table 7: Ablation studies on the impact of FCID (detailed results for each task are provided in Table 11)
6 Conclusion

Inspired by feature importance and training-free optimization, we propose TOC, the first training-free optimization method for discrete representation space, enhancing both multimodal and single-modal representations. We also introduce FCID, a framework that integrates disentanglement with modality-specific characteristics to achieve fine-grained audio-video temporal alignment and coarse-grained text semantic alignment. This disentanglement separates modality-specific information, yielding a unified multimodal representation. Extensive experiments on cross-modal classification, localization, retrieval, and generation tasks validate the effectiveness of our approach.

Limitations

TOC: We assumed equal prior probabilities for different classes in the open-world setting, a reasonable assumption in many cases. However, in real-world applications, class distributions can be imbalanced, with some classes having an excessive number of instances, while others may have too few. Consequently, for broader practical applicability, TOC requires further optimization and deeper exploration to account for such variabilities.

FCID: is specifically designed for tri-modal representations (audio, video, and text). However, in scenarios involving only two modalities, only the Fine or Coarse components of FCID’s disentanglement and alignment processes are applicable.

Acknowledgments

We thank MindSpore (http://mindspore.cn) for the partial support of this work, which is a new deep learning computing framework.

References

Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. 2020. Self-supervised multimodal versatile networks. Advances in neural information processing systems, 33:25–37.

Alex Andonian, Shixing Chen, and Raffay Hamid. 2022. Robust cross-modal representation learning with progressive self-distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16430–16441.

Leo Breiman. 2001. Random forests. Machine learning, 45:5–32.

David Chen and William B Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 190–200.

Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. 2020a. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721–725. IEEE.

Minghao Chen, Iro Laina, and Andrea Vedaldi. 2024a. Training-free layout control with cross-attention guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5343–5353.

Pengtao Chen, Mingzhu Shen, Peng Ye, Jianjian Cao, Chongjun Tu, Christos-Savvas Bouganas, Yiren Zhao, and Tao Chen. 2024b. Delta-dit: A training-free acceleration method tailored for diffusion transformers. arXiv preprint arXiv:2406.01125.

Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu. 2023. Valor: Vision-audio-language omni-perception pre-training model and dataset. arXiv preprint arXiv:2304.08345.

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020b. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104–120. Springer.

Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. 2020. Club: A contrastive log-ratio upper bound of mutual information. In International conference on machine learning, pages 1779–1788. PMLR.

Hao Dong, Ismail Nejjar, Han Sun, Eleni Chatzi, and Olga Fink. 2024. Simmdgd: A simple and effective framework for multi-modal domain generalization. Advances in Neural Information Processing Systems, 36.

Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. 2020. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 736–740. IEEE.

Jiali Duan, Liqun Chen, Son Tran, Jinyu Yang, Yi Xu, Belinda Zeng, and Trishul Chitilimi. 2022. Multi-modal alignment using representation codebook. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15651–15660.

Minghui Fang, Shengpeng Ji, Jialong Zuo, Hai Huang, Yan Xia, Jieming Zhu, Xize Cheng, Xiaoda Yang, Wenrui Liu, Gang Wang, et al. 2024. Ace: A generative cross-modal retrieval framework with coarse-to-fine semantic modeling. arXiv preprint arXiv:2406.17507.
David Harwath, Adria Recasens, Dídac Surís, Galen Chuang, Antonio Torralba, and James Glass. 2018. Jointly discovering visual objects and spoken words from raw sensory input. In Proceedings of the European conference on computer vision (ECCV), pages 649–665.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30.

Hai Huang, Shulei Wang, and Yan Xia. 2025. Semantic residual for multimodal unified discrete representation. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE.

Hai Huang, Yan Xia, Shengpeng Ji, Shulei Wang, Hanting Wang, Jieming Zhu, Zhenhua Dong, and Zhou Zhao. 2024. Unlocking the potential of multimodal unified discrete representation through training-free codebook optimization and hierarchical alignment. arXiv preprint arXiv:2403.05168.

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196.

Alexander H Liu, SouYoung Jin, Cheng-I Jeff Lai, Andrew Rouditchenko, Aude Oliva, and James Glass. 2021a. Cross-modal discrete representation learning. arXiv preprint arXiv:2106.05438.

Xingchao Liu, Chengyue Gong, Lemeng Wu, Shujian Zhang, Hao Su, and Qiang Liu. 2021b. Fuse-dream: Training-free text-to-image generation with improved clip+ gan space optimization. arXiv preprint arXiv:2112.01573.

Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. 2022. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916.

Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2022. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508:293–304.

Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2630–2640.

Mathew Monfort, SouYoung Jin, Alexander Liu, David Harwath, Rogério Feris, James Glass, and Aude Oliva. 2021. Spoken moments: Learning joint audio-visual representations from video descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14871–14881.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.

Fabrizio Pedersoli, Dryden Wiebe, Amin Banitalebi, Yong Zhang, George Tzanetakis, and Kwang Moo Yi. 2022. Estimating visual information from audio through manifold learning. arXiv preprint arXiv:2208.02337.

Bo Peng, Xinyuan Chen, Yaohui Wang, Chaochao Lu, and Yu Qiao. 2024. Conditionvideo: Training-free condition-guided video generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4459–4467.

Stavros Petridis, Themos Stafylakis, Pingchuan Ma, Georgios Tzimiropoulos, and Maja Pantic. 2018. Audio-visual speech recognition with a hybrid ctc/attention architecture. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 513–520. IEEE.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.

Pritam Sarkar and Ali Etemad. 2022. Xkd: Crossmodal knowledge distillation with domain alignment for video representation learning. arXiv preprint arXiv:2211.13929.

Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. 2018. Learning to localize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4358–4366.

Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas, Brian Kingsbury, Rogério S Feris, David Harwath, James Glass, and Hilde Kuehne. 2022. Everything at once-multi-modal fusion transformer for video retrieval. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 20020–20029.

Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402.

Yapeng Tian, Dingzeyu Li, and Chenliang Xu. 2020. Unified multisensory perception: Weakly-supervised audio-visual video parsing. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pages 436–454. Springer.

Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. 2018. Audio-visual event localization in unconstrained videos. In Proceedings of the European Conference on Computer Vision (ECCV), pages 247–263.
Aaron Van Den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning. Advances in neural information processing systems, 30.

Qi-Wei Wang, Da-Wei Zhou, Yi-Kai Zhang, De-Chuan Zhan, and Han-Jia Ye. 2024. Few-shot class-incremental learning via training-free prototype calibration. Advances in Neural Information Processing Systems, 36.

Shulei Wang, Wang Lin, Hai Huang, Hanting Wang, Sihang Cai, WenKang Han, Tao Jin, Jingyuan Chen, Jiacheng Sun, Jieming Zhu, et al. 2025. Towards transformer-based aligned generation with self-coherence guidance. arXiv preprint arXiv:2503.17675.

Teng Wang, Wenhao Jiang, Zhichao Lu, Feng Zheng, Ran Cheng, Chengguo Yin, and Ping Luo. 2022. Vlmixer: Unpaired vision-language pre-training via cross-modal cutmix. In International Conference on Machine Learning, pages 22680–22690. PMLR.

Zehan Wang, Yang Zhao, Haifeng Huang, Jiageng Liu, Aoxiong Yin, Li Tang, Linjun Li, Yongqi Wang, Zhang Zhang, and Zhou Zhao. 2023. Connecting multi-modal contrastive representations. Advances in Neural Information Processing Systems, 36:22099–22114.

Maksymilian Wojtas and Ke Chen. 2020. Feature importance ranking for deep learning. Advances in Neural Information Processing Systems, 33:5105–5114.

Mingrui Wu, Xinyue Cai, Jiayi Ji, Jiale Li, Oucheng Huang, Gen Luo, Hao Fei, Xiaoshuai Sun, and Rongrong Ji. 2024. Controlmlm: Training-free visual prompt learning for multimodal large language models. arXiv preprint arXiv:2407.21534.

Yan Xia, Hai Huang, Jieming Zhu, and Zhou Zhao. 2024. Achieving cross modal generalization with multimodal unified representation. Advances in Neural Information Processing Systems, 36.

Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. 2021. Video-clip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.14084.

Zihui Xue, Zhengqi Gao, Sucheng Ren, and Hang Zhao. 2022. The modality focusing hypothesis: Towards understanding crossmodal knowledge distillation. arXiv preprint arXiv:2206.06487.

Shaoshu Yang, Yong Zhang, Xiaodong Cun, Ying Shan, and Ran He. 2024. Zerosmooth: Training-free diffuser adaptation for high frame rate video generation. arXiv preprint arXiv:2406.00908.

Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. 2023. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721.

Youngjae Yu, Jongseok Kim, and Gunhee Kim. 2018. A joint sequence fusion model for video question answering and retrieval. In Proceedings of the European conference on computer vision (ECCV), pages 471–487.

Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. 2022. Tip-adapter: Training-free adaption of clip for few-shot classification. In European Conference on Computer Vision, pages 493–510. Springer.

Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. 2023. Controlvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305.13077.

Yang Zhao, Chen Zhang, Haifeng Huang, Haoyuan Li, and Zhou Zhao. 2022. Towards effective multi-modal interchanges in zero-resource sounding object localization. Advances in Neural Information Processing Systems, 35:38089–38102.

Jinxing Zhou, Dan Guo, and Meng Wang. 2022. Contrastive positive sample propagation along the audio-visual event line. IEEE Transactions on Pattern Analysis and Machine Intelligence.

Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijie Hao, and Meng Wang. 2021. Positive sample propagation along the audio-visual event line. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8436–8444.

Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, and Peng Gao. 2023. Not all features matter: Enhancing few-shot clip with adaptive prior refinement. arXiv preprint arXiv:2304.01195.

A Implementation Details

Compared Works: The models we compare include the most outstanding recent developments in multimodal unified discrete representations and models that excel in multimodal domain generalization: CODIS (Duan et al., 2022), TURN (Zhao et al., 2022), CMCN (Liu et al., 2021a), SimMDG (Dong et al., 2024), and DCID (Xia et al., 2024). These methods are implemented on our tasks, and their performance is evaluated on multi downstream tasks.

Evaluation Metrics: For the AVE (Tian et al., 2018), VGGSound-AVEL (Zhou et al., 2022, 2021), and UCF101 (Soomro et al., 2012) datasets, precision is used as the metric. The F1-score is utilized for assessing the AVVP (Tian et al., 2020) and AVE→AVVP generalization task, and recall is utilized for zero-shot retrieval (Chen and Dolan, 2011; Drossos et al., 2020). Mean Square Error (MSE) is employed to evaluate the reconstruction quality of
TOC on the CelebA-HQ 30K dataset (Karras et al., 2017). Additionally, Fréchet Inception Distance (FID) (Heusel et al., 2017) is used to assess the model’s capability in cross-modal generalization.

Hyperparameter Settings: The \( \beta \) of \( L_{commit} \) is set to 0.25, and in the TOC formulation, the parameter \( \lambda \) is set to 0.3, and in \( L_{nce} \), the parameter \( \tau \) is set to 1.0. All results presented in table 1, 2, 4, 6, 7 were obtained with a codebook size set to 400 and an embedding dimension set to 256. The table 3 involves VQVAE with a codebook size of 128 and an embedding dimension of 128. The ablation study on codebook size is discussed in Table 9.

B Experimental Supplement

B.1 The Impact of TOC on Cosine Similarity of Codebook

![Left: Cosine similarity in the original codebook. Right: Cosine similarity after TOC.](page_184_682_495_246.png)

Figure 8: Left: Cosine similarity in the original codebook. Right: Cosine similarity after TOC.

We conducted an evaluation of TOC on the open-source pre-trained model of DCID (Xia et al., 2024). As depicted in Figure 8, the distinctiveness of the codes with the features obtained after TOC computation is notably enhanced.

B.2 Computational efficiency

Comparing computational efficiency and resource requirements provides a more comprehensive view of the trade-off between performance improvement and computational cost. As shown in Table 8, we present a comparison of the computational efficiency of CMC (Liu et al., 2021a), DCID (Xia et al., 2024), and our method:

<table>
  <tr>
    <th>Model</th>
    <th>GPU Memory Usage</th>
    <th>Time per Epoch</th>
    <th>Total Epochs</th>
  </tr>
  <tr>
    <td>DCID</td>
    <td>100%</td>
    <td>100%</td>
    <td>5</td>
  </tr>
  <tr>
    <td>CMCM</td>
    <td>81%</td>
    <td>76%</td>
    <td>8</td>
  </tr>
  <tr>
    <td>FCID</td>
    <td>96%</td>
    <td>99%</td>
    <td>5</td>
  </tr>
</table>

Table 8: Model Comparison: Relative GPU Memory Usage and Time per Epoch (Compared to DCID), and Total Training Epochs

Under the same pretraining conditions with a batch size of 80 and using an RTX 3090 GPU, the results are shown in Table 8. In CMC, contrastive learning is applied to both fine-grained and high-level representation for each modality. DCID performs alignment and disentanglement of all modalities at the fine-grained level (Alignment of pairwise combinations of the three modalities, along with their respective disentanglement). Our proposed FCID first applies disentangling and contrastive learning to fine-grained features for audio and video, then applies coarse-grained disentangling and contrastive learning to the compressed audio-visual and text features. Specifically, there are 2 alignment operations and 4 disentangling operations (fine-grained alignment of audio and video features, coarse-grained alignment of audiovisual and text features, and disentangling of these 4 feature types).

For GPU Memory Usage and Time per Epoch, CMC has the lowest consumption in both cases. FCID uses less GPU memory than DCID, while Time per Epoch is nearly the same, mainly due to the projection compressing the temporal dimension before performing coarse-grained alignment and disentanglement, reducing computational data. DCID and FCID benefit from faster convergence due to disentangling and only require 5 epochs to achieve optimal performance, while CMC needs warm-start strategy and takes 8 epochs to converge, resulting in the longest total training time.

During downstream inference, no disentangling or alignment is required. Only the corresponding encoders and quantization are used. Compared to DCID, our FCID adds only simple projection layers and coarse-grained general and specific encoders constructed with MLP for audiovisual data, making the added inference time almost negligible. The text modality, on the other hand, requires less time than DCID due to coarse-grained compression. The inference speed difference between CMC, DCID, and FCID is minimal.

Another module we propose, TOC, only requires a single computation of less than 10 seconds after obtaining the discrete representations, and it only requires CPU computation. By reducing the feature dimensions, it can also accelerate training and inference speeds in downstream tasks.

B.3 Loss Changes During Pre-training

As shown in Equation 19, our model involves a total of four losses. Among them, \( L_{commit} \) is a
loss commonly used in VQ-related models. Its purpose is to ensure that the model’s output maintains continuity in the latent space when mapped to the discrete space. This is not the main contribution of this paper. Here, we focus on the changes in the other three losses. As illustrated in the Figure 9, the overall trend shows a clear downward slope. Among them, \( L_{club} \) converges first, and when it reaches zero, it indicates successful decoupling of the modalities. The decrease in \( L_{contra} \) reflects an increase in the alignment of multimodal information, while the decrease in \( L_{recon} \) signifies improved preservation of complete semantics after the information is decoupled and re-merged.

![Line plots showing changes in losses during pre-training](page_184_627_573_312.png)

Figure 9: Changes in Losses During Pre-training

<table>
  <tr>
    <th rowspan="2">Codebook Size</th>
    <th colspan="2">AVE</th>
    <th colspan="2">AVVP</th>
    <th colspan="2">UCT(V)→VGG(G)</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>V→A→A→V</th>
    <th>Y→A→A→V</th>
    <th>V→A→A→V</th>
    <th>Y→A→A→V</th>
    <th>V→A→A→V</th>
    <th>Y→A→A→V</th>
  </tr>
  <tr>
    <td>256</td>
    <td>52.9</td>
    <td>52.3</td>
    <td>38.8</td>
    <td>43.2</td>
    <td>53.7</td>
    <td>53.9</td>
    <td>70.8</td>
    <td>56.4</td>
    <td>52.75</td>
  </tr>
  <tr>
    <td>300</td>
    <td>52.8</td>
    <td>54.1</td>
    <td>42.1</td>
    <td>44.1</td>
    <td>54.1</td>
    <td><b>58.5</b></td>
    <td>69.6</td>
    <td>60.4</td>
    <td>54.46</td>
  </tr>
  <tr>
    <td>400</td>
    <td><b>55.2</b></td>
    <td><b>54.9</b></td>
    <td><b>42.4</b></td>
    <td><b>44.5</b></td>
    <td><b>55.3</b></td>
    <td>57.4</td>
    <td>69.4</td>
    <td><b>61.6</b></td>
    <td><b>55.47</b></td>
  </tr>
  <tr>
    <td>512</td>
    <td>54.1</td>
    <td>52.4</td>
    <td>40.6</td>
    <td>42.6</td>
    <td>53.1</td>
    <td>56.7</td>
    <td>69.6</td>
    <td>59.3</td>
    <td>53.79</td>
  </tr>
  <tr>
    <td>800</td>
    <td>52.2</td>
    <td>54.6</td>
    <td>41.6</td>
    <td>43.9</td>
    <td>53.1</td>
    <td>56.7</td>
    <td>69.6</td>
    <td>59.7</td>
    <td>53.93</td>
  </tr>
  <tr>
    <td>1024</td>
    <td>52.8</td>
    <td>54.5</td>
    <td>40.4</td>
    <td>41.6</td>
    <td><b>55.3</b></td>
    <td>55.9</td>
    <td>65.8</td>
    <td>58.6</td>
    <td>53.11</td>
  </tr>
</table>

Table 9: Ablation Studies on the Impact of Codebook Size

B.4 Codebook size

Table 9 presents the performance of the FCID model across various codebook sizes. It is observed that the model achieves the best average results when the codebook size is set to 400. Conversely, using either a excessively large or small codebook size may lead to insufficient semantic learning or inadequate semantic expression, resulting in decreased model performance.

B.5 Reconstruction

As shown in Figure 10, for all columns except the 'origin' column, the images on the left represent reconstructions with random masks, while the images on the right illustrate reconstructions using the dimensions with the highest TOC retention scores. It is evident that TOC significantly outperforms random masking in reconstructions with mask ratios ranging from 25.0% to 87.5%, with the performance gap becoming increasingly pronounced as the mask ratio increases.

![Grid of reconstructed images with varying mask ratios](page_1012_627_377_312.png)

Figure 10: More results of reconstructions using random and TOC masking.

B.6 Generation

As shown in Figure 11, thanks to multimodal unified representations, the results of cross-modal image generation from audio and text closely resemble actual images. As evident in samples 2 and 6, despite the audio not mentioning specific details

![Grid of cross-modal generation results](page_1012_1047_377_312.png)

Figure 11: More results of cross-modal generation.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">MSCOCO(V→T)</th>
    <th colspan="3">Clotho(A→T)</th>
    <th colspan="3">FlickrSound(V→A)</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>R@1</th><th>R@5</th><th>R@10</th>
    <th>R@1</th><th>R@5</th><th>R@10</th>
    <th>R@1</th><th>R@5</th><th>R@10</th>
  </tr>
  <tr>
    <td>CMCM (<a href="https://doi.org/10.48550/arXiv.2106.07945">Liu et al., 2021a</a>)</td>
    <td>0.50</td><td>4.20</td><td>7.20</td>
    <td>1.62</td><td>8.04</td><td>14.87</td>
    <td>2.20</td><td>9.80</td><td>15.60</td>
    <td>7.11</td>
  </tr>
  <tr>
    <td>DCID (<a href="https://doi.org/10.48550/arXiv.2306.04247">Xia et al., 2024</a>)</td>
    <td>0.80</td><td>5.00</td><td>8.30</td>
    <td>2.06</td><td>9.00</td><td>16.70</td>
    <td>3.10</td><td>11.10</td><td>17.20</td>
    <td>8.14</td>
  </tr>
  <tr>
    <td>FCID</td>
    <td>1.30</td><td>4.90</td><td>9.60</td>
    <td>2.87</td><td>10.73</td><td>18.19</td>
    <td>3.10</td><td>11.80</td><td>17.50</td>
    <td>8.89</td>
  </tr>
  <tr>
    <td>CMCM+TOC</td>
    <td>0.70</td><td>4.50</td><td>7.70</td>
    <td>1.93</td><td>8.43</td><td>15.33</td>
    <td>2.40</td><td>10.60</td><td>16.10</td>
    <td>7.52<sup>(+0.41)</sup></td>
  </tr>
  <tr>
    <td>DCID+TOC</td>
    <td>1.10</td><td>5.30</td><td>8.80</td>
    <td>2.59</td><td>9.00</td><td>17.08</td>
    <td>3.60</td><td>11.80</td><td>17.80</td>
    <td>8.56<sup>(+0.42)</sup></td>
  </tr>
  <tr>
    <td>FCID+TOC</td>
    <td>1.50</td><td>5.10</td><td>10.40</td>
    <td>3.16</td><td>11.15</td><td>19.04</td>
    <td>3.80</td><td>12.20</td><td>18.40</td>
    <td>9.42<sup>(+0.53)</sup></td>
  </tr>
</table>

Table 10: Details of comparison with SOTA methods on three cross-modal zero-shot retrieval tasks, all results are calculated as the mean across two directions.

<table>
  <tr>
    <th rowspan="2">A<sub>CLUB</sub></th>
    <th rowspan="2">V<sub>CLUB</sub></th>
    <th rowspan="2">AV<sub>CLUB</sub></th>
    <th rowspan="2">TE<sub>CLUB</sub></th>
    <th colspan="3">AVE</th>
    <th colspan="3">AVVP</th>
    <th colspan="3">AVE→AVVP</th>
    <th colspan="3">UCF(v)→VGG(a)</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>V→A</th><th>A→V</th><th>V→A→V</th>
    <th>V→A</th><th>A→V</th><th>V→A→V</th>
    <th>V→A</th><th>A→V</th><th>V→A→V</th>
    <th>V→A</th><th>A→V</th><th>V→A→V</th>
  </tr>
  <tr>
    <td>-</td><td>-</td><td>-</td><td>-</td>
    <td>51.3</td><td>51.6</td><td>39.5</td>
    <td>40.7</td><td>50.6</td><td>51.1</td>
    <td>63.3</td><td>57.6</td><td>50.7</td>
  </tr>
  <tr>
    <td>✓</td><td>-</td><td>-</td><td>-</td>
    <td>52.4</td><td>53.5</td><td>40.9</td>
    <td>42.4</td><td>53.1</td><td>54.2</td>
    <td>66.0</td><td>59.8</td><td>52.79</td>
  </tr>
  <tr>
    <td>-</td><td>✓</td><td>-</td><td>-</td>
    <td>53.1</td><td>53.4</td><td>41.7</td>
    <td>43.2</td><td>53.9</td><td>54.7</td>
    <td>67.1</td><td>60.1</td><td>53.40</td>
  </tr>
  <tr>
    <td>-</td><td>-</td><td>✓</td><td>-</td>
    <td>52.2</td><td>51.9</td><td>40.2</td>
    <td>41.7</td><td>52.4</td><td>52.5</td>
    <td>64.2</td><td>59.1</td><td>51.78</td>
  </tr>
  <tr>
    <td>-</td><td>-</td><td>-</td><td>✓</td>
    <td>51.7</td><td>51.5</td><td>40.6</td>
    <td>41.8</td><td>52.5</td><td>52.9</td>
    <td>63.5</td><td>58.2</td><td>51.59</td>
  </tr>
  <tr>
    <td>✓</td><td>✓</td><td>-</td><td>-</td>
    <td>54.2</td><td>54.0</td><td>41.4</td>
    <td>43.9</td><td>55.9</td><td>56.1</td>
    <td>67.9</td><td>61.3</td><td>54.34</td>
  </tr>
  <tr>
    <td>-</td><td>✓</td><td>✓</td><td>-</td>
    <td>52.9</td><td>52.6</td><td>40.8</td>
    <td>42.1</td><td>52.5</td><td>53.9</td>
    <td>65.7</td><td>59.2</td><td>52.46</td>
  </tr>
  <tr>
    <td>✓</td><td>✓</td><td>✓</td><td>✓</td>
    <td><b>55.2</b></td><td><b>54.9</b></td><td><b>42.4</b></td>
    <td><b>44.5</b></td><td><b>55.3</b></td><td><b>57.4</b></td>
    <td><b>69.4</b></td><td><b>61.6</b></td><td><b>55.09</b></td>
  </tr>
</table>

Table 11: Details of ablation studies on the impact of FCID

such as the color of clothing and trains, these elements are still accurately generated, which can be attributed to the discrete unified representation serving as a central semantic hub for multiple modalities. In contrast, the results from Text-to-Image (T→I) are noticeably inferior to those from Image-to-Image (I→I) and Audio-to-Image (A→I). This difference is exemplified in the first image generated from sample 1’s text, where the action of a car mowing grass is mistakenly transformed into a man mowing grass. This discrepancy arises because the semantic connections between images and audio are stronger than those generated through model-based text, which merely mentioned ‘man’ and ‘plowing grass’ without specifying the tool used for plowing.

Reviewer paper summaries: This is a work about Multimodal Unified Representation Learning, aiming to optimize the generalization of unified modal features across text, audio and vision modalities. The motivation and idea are well presented, and experiments are extensive to demonstrate the work's contribution. 

However, the codebook is generally used in multimodal learning and multimodal tasks, as shown in the summary of weakness, which causes limited contribution and novelty.  Additionally, the method is not well-organized, showing a confusing and hard-readable presentation, hoping further revision in the next version.

This paper aims to learn more effective multimodal unified representations for cross-modal generalization.The authors propose Training-Free Optimization of Codebook (TOC) to resolve the issues that previous methods often overlook the important distinctions among different dimensions of features. And then they propose Fine and Coarse cross-modal Information Disentangling (FCID) to perform feature disentanglement during alignment and conduct alignment in a fine to coarse manner. The experiments are extensive, verifying the effectiveness of the proposed method.
Summary of strengths: 1. The ideas of code similarity  and code variance are good, which filters out the irrelevant features and reduces redundancy. It seems to bring improvements in cross-modal tasks. Experiments also present the transferability of TOC.

2. Extensive experiments on various downstream tasks.

1. Technical Quality: The methods are technically sound and well-motivated. The use of Code Similarity and Code Variance metrics in TOC is a reasonable and intuitive way to refine features in the unified space. The combination of disentanglement and alignment in FCID is also reasonable, leading to a more unified multimodal representation.

2.Empirical Validation: The authors provide extensive experiments across various tasks such as cross-modal classification, localization, retrieval, and generation. The results demonstrate the effectiveness of the proposed methods, outperforming previous state-of-the-art models.

3. Clarity and Presentation: The paper is well-written and easy to follow. The figures and tables are clear and help illustrate the key points of the methods and results.
Summary of weaknesses: 1. Codebook is widely used in multimodal task which limits the novelty and contribution of this works. I listed some multimodal works that have introduced codebook, as follows. 

[1]Cross-Modal Discrete Representation Learning, ACL 2022

[2]Achieving Cross Modal Generalization with Multimodal Unified Representation, NIPS 2023

[3]Multi-modal Alignment using Representation Codebook, CVPR 2022

It is necessary to claim the difference between proposed work and prior work and re-organize the architecture of presentation for clarity. 

2. This work seems to combine two modules: TOC and FCID. TOC has good motivation but FCID just borrows two modules (DCID and FAC) of prior work (line: 92-95). FCID tries to align and distangle modal features, which has been explored for a long time. Thus, authors should make it clear about this module's novelty and contribution.

3. The writing of method is not well and it is obscure and difficult to understand. 

4. Figure 4 fails to present the difference between four models including the proposed model. I suggest there should have a overview or detailed pipeline architecture for the proposed model in the next revision. 

5. I am not quite sure the meaning of the expression " still requires refinement to improve robustness in diverse real-world scenarios" in limitation section.

1. Model Complexity: While the paper compares the proposed methods with several state-of-the-art models, a more detailed comparative analysis of the computational efficiency and resource requirements of the proposed methods would be beneficial. This includes a discussion on the trade-offs between performance gains and computational costs. The model combines feature alignment and disentanglement with various encoders and decoders, and conducts alignment and disentanglement for two times for each sample. It seems that the running time and model complexity might increase significantly.

2.Training Losses: The training of the model relies on multiple losses, which can be difficult to optimized. Perhaps showing the learning curves of these losses can give insight to the readers about which losses actually take effect.

3.Visualization: The authors are encouraged to provide more t-SNE visualizations about the modality-specific and modality-general features of each modality or audio-video data to analyze whether the modality-general features are aligned or whether the features are actually disentangled.
Comments, suggestions and typos: None

Please see the weaknesses.
Overall assessment scores: 3

3.5
Metareview score: 2
Metareview: All the reviewers think that this paper is well-motivated and appreciate the efforts the authors have put into it. The proposed problem and approach are certainly sound. However, the reviewers unanimously find that the paper lacks some important experimental result analysis, the explanation of the approach is not clear enough, and some experimental details are insufficient. Additionally, the paper may require some restructuring beyond the details already presented by the authors. Therefore, I do not recommend accepting the current version of the paper and believe it would merit another round of review. I encourage the authors to consider all the suggestions the reviewers have provided in the next version.
Summary of reasons to publish: The motivation of this paper is interesting.
The proposed Training-Free Optimization of Codebook (TOC) approach is both technically sound and well-motivated.