SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The modified commitment loss  $L_{commit}^{b}$  is also the same process with modality A. During training, the distance between different modalities in the latent space gradually decreases.
Reset inactivated code: In order to alleviate the codebook collapse problem existing in VQ process, which implies that only a subset of codes are extensively utilized while the majority of codes remain inactive, we adopt a code reset strategy: for a given code vector  $\mathbf{e}_i$ , it will be retained only if it has been selected at least once by both modality A and modality B in  $N_{re}$  consecutive batches, whereas the remaining codes will be re-initialized. This strategy can effectively mitigate the issue of some code being perpetually unexploitable due to poor initialization, as well as the problem of generating redundant code during the process of aligning modalities A and B (see appendix for more details).
4.4 Training and Downstream Tasks
The full objective of our pre-training framework is the combination of all above objection functions:  $L = L_{recon} + L_{commit} + L_{cpc} + L_{cmcm} + L_{MI}$ , where  $L_{MI}$  is  $\hat{I}_{\mathrm{vCLUB}}$ ,  $L_{cmcm}$  is the objective loss proposed in [17], which can also promote the alignment among modalities. All these objectives except  $L_{cmcm}$  integrate two modalities. After training, we can obtain a unified discrete latent space.
In this paper, we investigate the unified representation of various modalities, including audio-visual, audio-text, visual-text, and even more, the tri-modal combination of audio-visual-text. Then we can apply the pre-trained multi-modal encoder in various downstream tasks, i.e., cross-modal event classification, event localization, query-based video segmentation and cross both domain & modal event localization. Note that during downstream tasks, the parameters of the pre-trained encoder are frozen.
5.1 Datasets and Tasks
Pre-train: We use VGGsound-AVEL [44, 45] to pre-train our unified representation, and divide it into several different sizes: 24K, 40K, 81K. Considering that the VGGsound-AVEL dataset only contains audio, video and event label, we design prompts for these labels and modify them into descriptive sentences, see Appendix for more details.
Table 1: Compared with state-of-the-art methods on two downstream tasks. We use precision to indict the performance of the models on AVE tasks, and use accuracy for AVVP tasks.
Method	VGGsounds-AVEL 24K	VGGsounds-AVEL 40K	VGGsounds-AVEL 81K
AVE	AVVP	AVE	AVVP	AVE	AVVP
V→A A→V	V→A A→V	V→A A→V	V→A A→V	V→A A→V	V→A A→V
Baseline	4.4	5.9	7.6	8.4	5.5	5.4	6.9	8.7	7.1	9.3	5.6	7.2
S-Mit[43]	12.7	16.9	17.2	22.8	14.4	15.9	19.0	22.3	13.4	17.0	20.9	22.8
MST[13]	13.3	19.0	25.7	29.1	19.5	23.1	22.7	24.5	18.6	20.5	19.1	24.8
CODIS[16]	18.5	22.0	29.4	33.7	20.8	26.4	35.1	37.9	28.5	30.2	34.0	37.8
TURN[20]	17.7	21.0	29.4	32.4	19.1	24.3	36.9	39.3	27.6	31.4	33.8	38.1
CMCM[17]	28.9	35.9	42.6	50.4	32.7	36.8	41.9	45.1	31.1	34.0	39.3	44.8
DCID+S-Mit	28.1	32.3	45.9	49.2	32.2	34.0	47.8	53.0	34.8	37.6	51.9	53.5
DCID+MST	31.2	35.0	50.7	52.1	34.9	37.8	54.4	59.1	33.5	35.4	57.1	59.2
DCID+TURN	29.4	35.3	53.4	56.0	29.7	36.9	55.2	58.2	31.9	36.8	56.2	60.9
DCID+CODIS	33.4	36.0	53.8	60.2	36.7	41.0	52.6	62.0	35.9	40.1	54.3	59.0
DCID+CMCM	34.1	38.8	57.6	60.8	36.4	42.9	58.7	62.8	38.8	41.4	57.5	60.5
Uni-Code	44.0	49.7	61.9	65.7	47.7	52.3	64.0	65.6	41.2	45.6	60.5	61.7
Table 2: Ablation studies of audio-visual pre-training on AVE and AVVP tasks.
CLUB	Cross-CPC	MM-EMA	Reset code	\( {\mathrm{L}}_{\text{cmcm }} \)	VGGsounds-AVEL 24K	VGGsounds-AVEL 40K
AVE	AVVP	AVE	AVVP
\( V \rightarrow A \ A \rightarrow V \)	\( V \rightarrow A \ A \rightarrow V \)	\( V \rightarrow A \ A \rightarrow V \)	\( V \rightarrow A \ A \rightarrow V \)
-	✓	✓	✓	✓	34.9	35.1	50.6	54.0	37.2	40.3	52.9	59.5
✓	-	✓	✓	✓	4.6	5.8	10.9	24.6	5.2	7.1	12.3	24.1
-	-	✓	✓	✓	29.8	34.6	30.4	32.5	35.2	36.9	32.3	34.0
✓	✓	-	✓	✓	34.1	38.8	57.6	60.8	36.4	42.9	58.7	62.8
✓	✓	✓	-	✓	37.8	41.2	59.1	61.5	38.9	40.3	55.4	62.1
✓	✓	✓	✓	-	39.7	42.6	58.2	62.1	41.3	46.0	58.7	62.8
✓	✓	-	-	✓	28.2	30.9	41.4	49.2	31.5	33.8	46.0	48.3
✓	✓	✓	✓	✓	44.0	49.7	61.9	65.7	47.7	52.3	64.0	65.6
Downstream: we evaluate the pre-trained models on several downstream tasks using different datasets: Cross-modal event classification (AVE [46]): In AVE, each video contains a primary event that is present in both audio and visual information. Therefore, we can train an event classifier using a single modality (e.g., video) and directly evaluate the classifier's performance on another modality (e.g., audio). Cross-modal event localization (AVVP [47]): In AVVP, a portion of the data is annotated with fine-grained event labels for both audio and visual modalities. Similar to AVE, we can perform event localization on one modality and then directly transfer the model to the other modality for testing. Cross both modal and dataset localization/classification: In order to prove that our model can also achieve cross-modal transfer on different downstream datasets, we use classification tasks for one modality in AVE to train, and directly test fine-grained event localization ability of the model on another modality in AVVP (from AVE to AVVP). Also we test the cross-modal classification tasks between the visual modality in part of UCF-101 (16 classes) and audio modality in part of VGGSound-AVEL (16 classes). Cross-modal video segmentation (AVSBench-S4 [48]): In AVS, each piece of AVS data contains a piece of audio, a video clip of the object that emits the sound, a text label corresponding to the object, and a manually labeled mask. We expand each text label into a descriptive sentence. We use one modality (e.g., audio) to train the query-based video segmentation, and directly test the segmentation ability in another modality (e.g., text). Cross-modal retrieval (VGGSound-AVEL) We select retrieval task to demonstrate that our method can also be applied to visual-text generalization as well. We use audio as an inter-medium to measure the generalization ability of our model across these two modalities and implement an X-to-audio retrieval task. To be detailed, in the first stage, we train visual-text unified representation learning using VGGSound24K dataset, and then in the second stage, during downstream training, we let the model learn text(video)-audio retrieval, finally during inference, we directly test the generalization ability of the model on video(text)-audio retrieval.
5.2 Baselines and Implementation Details
We use the method depicted in Section 4.1 as our baseline. We also compare our methods with several state-of-the-art unified representation methods: MST [13], CODIS [16], S-MiT [43], CMCM [17],
Table 3: Ablation studies of audio-visual-text pre-training on three downstream tasks.
CLUB	Cross-CPC	MM-EMA	Reset code	Lcmcm	VGGsounds-AVEL 40K
AVE	AVVP	AVE→AVVP	UCF(v)←VGG(a)
V→A A→V	V→A A→V	V→A A→V	V→A A→V
-	✓	✓	✓	✓	50.2	51.8	62.4	66.2	50.1	51.2	9.87	9.59
✓	-	✓	✓	✓	43.8	49.2	59.3	61.1	45.5	50.6	60.6	54.6
✓	✓	-	✓	✓	52.9	49.9	62.0	67.3	48.1	46.8	66.5	60.5
✓	✓	-	-	✓	33.0	35.5	56.7	61.2	7.4	12.6	43.3	35.2
✓	✓	✓	-	✓	50.8	47.8	56.4	61.1	47.9	50.4	60.0	49.8
✓	✓	✓	✓	-	52.4	54.5	57.5	72.9	50.5	48.7	69.9	59.7
✓	✓	✓	✓	✓	54.1	55.0	63.4	71.0	53.0	52.4	67.1	60.6
Evaluation results of the labeled modality	64.8	65.8	71.0	72.9	-	-	80.0	85.4
(a) Pretrained on audio-visual.
(b) Pretrained on audio-visual-text.
Figure 3: The effect of different codebook sizes on two pre-training tasks.
TURN [20]. We implement these methods on our tasks and test their performance on two downstream tasks. We use the perception to evaluate the performance on AVE, VGGSound-AVEL and UCF101 datasets, and use accuracy  $(\%)$  to evaluate the performance on AVVP datasets, and use F1-score  $(\%)$  to evaluate the performance for AVE to AVVP generalization task, and use mIoU and F-score (as same as AVS[48]) for AVS-S4 dataset. The implementation details are provided in Appendix.
5.3 Compared with State-of-the-art Methods
We mainly compare our model with these leading methods on two downstream tasks, cross-modal event classification and localization, all these models are pre-trained using audio-visual modalities on three different sizes of datasets. As we can see in Table 1, our methods can outperform all these previous works on three settings. The baseline results indicate that without any constraints, it is challenging for the model to learn the alignment relationship between different modalities during pre-training, leading to poor performance in downstream tasks. Moreover, the experiment results also demonstrate that merely employing these state-of-the-art methods in such unconstrained audio-visual alignment has considerable limitations. However, when incorporated with our proposed DCID module, these models exhibit substantial improvements in downstream tasks. This further suggests that our proposed DCID can effectively decouple the semantic information of different modalities.
In addition, we can observe from Table 1 that when the scale of pre-training data increases from  $24\mathrm{K}$  to  $40\mathrm{K}$ , there is a significant improvement in the performance of the model on downstream tasks. However, when the scale reaches  $81\mathrm{K}$ , the performance of the model does not improve much and even declines. This also indicates that excessive pre-training data may not yield desirable results, only an appropriate amount of data can enable the model to learn the correct alignment relationships.
5.4 Ablation Studies
Main Components We use the audio-visual unified representation as pre-training task and test the performance on the downstream AVE and AVVP tasks by removing each main component of our model, the results are depicted in Table 2. We can observe that the performances all decline after removing these components. In the DCID module, the impact of removing Cross-CPC is far greater than that of removing CLUB, which indicates that without the fine-grained cross-modal alignment constraints provided by Cross-CPC, it is difficult for CLUB alone to extract effective semantic
information from the original modal features. When MM-EMA is removed, the model performance also declines significantly, demonstrating that MM-EMA can effectively utilize the Cross-Attention mechanism between different modalities to gradually map the relevant information together, helping to achieve a unified representation. The experimental results of removing the "reset code" indicate that this module can effectively remove the redundancy latent codes caused by MM-EMA.
The Effect of Codebook Size Different codebook sizes may affect the effectiveness of modality aggregation, thereby influencing the performance of downstream cross-modal generalization tasks. Here we investigate how the size of codebook influences pre-training performance. From Fig 3, we can see that the appropriate codebook size can achieve the best pre-training performance, while too large size (which may cause signals from different modalities to aggregate at different positions in the codebook) and too small size (where semantically distinct signals are erroneously mapped together due to limited capacity) will both degrade the performance of pre-trained models.
Unified Representation of Three Modalities Compared with previous works that can only align two modalities together, our method can disentangle and align the semantic information of three modalities, as shown in Table 3. From the results we can see that the removal of these modules will lead to a decline in the performance of the pre-trained model on downstream tasks, similar to the results in Table 2. However, compared to the unified representation of the two modalities, the removal of these modules leads to a smaller decline in the model performance. We think the reason is that the introduction of the third modality can facilitate the other two modalities approaching each other, and the performance of the model pre-trained on three modalities is significantly higher than that of the model pre-trained on two, which also illustrate this point.
Results on AVS-S4 As we can see in Table 4, our pre-trained model can effectively transfer the segmentation ability across modals. The visualization of segmentation results in Fig 5, our model can accurately identify vocal regions in video frames for unknown modalities. Furthermore, the performance of our method is closer to the evaluation results of the AVS model on source modality. More visualization results and analysis can be found in Appendix.
Results on Cross Modal Retrieval Tasks We also test the performance of X-to-audio retrieval task under cross modal generalization setting, where X can be visual or text, the results are as shown in Table 5. Compared with baseline model, our methods can greatly advance the audio retrieval ability in both two generalization directions.
(a) baseline
(b) CMCM
Figure 4: Visualization of the discrete codes. The blue dot means that the number of visual modal quantization accounts for more than  $80\%$  of the total number of audio and video quantification, the red dot means that the audio station accounts for more than  $80\%$ , and the green dot means that audio and video each account for more than  $20\%$ . As we can seen in (b), the discrete codes obtained by CMCM still contain a little single modality mapped codes, while ours (c) are almost all green codes which can be mapped by both two modalities.
(c) Uni-Code
5.5 Qualitative Analyses
To further illustrate the effectiveness of our model during pre-training phase, we take audio and video as an example and visualize the latent discrete codes obtained from two modalities. As we can see in Fig 4(a), when there is no additional supervisory signal, different modalities are difficult to map to the same discrete code, which also proves why the baseline performs poorly in downstream tasks. The CMCM [17] method can effectively alleviate this problem, but there are still many codes that can only be mapped by single modality. However, our model enables information from different modalities but with the same semantics to be mapped into the same codes. (as shown in Fig 4(c)).
Table 4: Performance on AVS-S4 datasets (pre-trained on audio-visual-text modalities).
Methods	A2T	T2A
mIoU	F-score	mIoU	F-score
Baseline	69.8	81.4	69.9	81.3
Our full model	78.0	87.1	77.7	86.7
SST [49] (A2A)	60.3	80.1	-	-
AVS [48] (A2A)	78.7	87.9	-	-
Table 5: Performance of audio retrieval tasks under cross modal generalization directions.
Methods	V2T	T2V
R@5	R@10	R@5	R@10
Baseline	0.47	1.03	0.62	0.85
Our full model	10.3	21.9	8.47	16.7
Figure 5: Visualization results of A2T (left) and T2A (right) of our model on AVS-S4 dataset. We compare our method with the baseline model.
6 Conclusion and Discussion
In this paper, we propose a new Cross Modal Generalization task, and bring up two innotative modules, DCID and MM-EMA, which can achieve multi-modal unified representation. We achieve unified representation of different multi-modal combinations, including audio-visual, audio-text and audio-visual-text. Also, we conduct extensive experiments on various downstream tasks. The extensive experiments demonstrate the effectiveness of our proposed methods. Furthermore, our model can help existing models to extend into other modalities, achieving more extensive applications. Limitations and Boarder Impact: In our work we only focus on the unified representation of three modalities. However, our method can inspire future works to explore the combination of more modalities. Our work do not contain potential negative social impact.
7 Acknowledgments
This work is supported by National Key R&D Program of China under Grant No.2022ZD0162000, National Natural Science Foundation of China under Grant No. 62222211 and No.61836002. We also gratefully acknowledge the support of MindSpore (https://www.mindspore.cn), which is a new deep learning computing framework.
[1] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015.
[2] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. arXiv preprint arXiv:1809.01696, 2018.
[3] Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19108-19118, 2022.
[4] Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, and Cees GM Snoek. Actor and action video segmentation from a sentence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5958-5966, 2018.
[5] Seonguk Seo, Joon-Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with a large-scale benchmark. In Computer Vision-ECCV 2020: 16th
[37] Lingpeng Kong, Cyprien de Masson d'Autume, Wang Ling, Lei Yu, Zihang Dai, and Dani Yogatama. A mutual information maximization perspective of language representation learning. arXiv preprint arXiv:1910.08350, 2019.
[38] Malik Boudiaf, Imtiaz Zika, Jérôme Rony, José Dolz, Pablo Piantanida, and Ismail Ben Ayed. Information maximization for few-shot learning. Advances in Neural Information Processing Systems, 33:2445-2457, 2020.
[39] Wei Zhu, Haitian Zheng, Haofu Liao, Weijian Li, and Jiebo Luo. Learning bias-invariant representation by cross-sample mutual information minimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15002-15012, 2021.
[40] Solomon Kullback. Information theory and statistics. Courier Corporation, 1997.
[41] Avraham Ruderman, Mark Reid, Dario García-García, and James Petterson. Tighter variational representations of f-divergences via restriction to probability measures. arXiv preprint arXiv:1206.4664, 2012.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[43] Mathew Monfort, SouYoung Jin, Alexander Liu, David Harwath, Rogerio Feris, James Glass, and Aude Oliva. Spoken moments: Learning joint audio-visual representations from video descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14871–14881, 2021.
[44] Jinxing Zhou, Dan Guo, and Meng Wang. Contrastive positive sample propagation along the audio-visual event line. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[45] Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijie Hao, and Meng Wang. Positive sample propagation along the audio-visual event line. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8436–8444, 2021.
[46] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European Conference on Computer Vision (ECCV), pages 247–263, 2018.
[47] Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly-supervised audio-visual video parsing. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pages 436–454. Springer, 2020.
[48] Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio-visual segmentation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXVII, pages 386–403. Springer, 2022.
[49] Brendan Duke, Abdalla Ahmed, Christian Wolf, Parham Aarabi, and Graham W Taylor. Sstvos: Sparse spatiotemporal transformers for video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5912-5921, 2021.