Achieving Cross Modal Generalization with Multimodal Unified Representation
Authors: Yan Xia
Hai Huang
Jieming Zhu
Zhou Zhao
Venue: NeurIPS 2023 poster (2023)
Keywords: multi-modal
discrete representation
mutual information estimation

Abstract: This paper introduces a novel task called Cross Modal Generalization (CMG), which addresses the challenge of learning a unified discrete representation from paired multimodal data during pre-training. Then in downstream tasks, the model can achieve zero-shot generalization ability in other modalities when only one modal is labeled. Existing approaches in multimodal representation learning focus more on coarse-grained alignment or rely on the assumption that 
 information from different modalities is completely aligned, which is impractical in real-world scenarios. To overcome this limitation, we propose \textbf{Uni-Code}, which contains two key contributions: the Dual Cross-modal Information Disentangling (DCID) module and the Multi-Modal Exponential Moving Average (MM-EMA). These methods facilitate bidirectional supervision between modalities and align semantically equivalent information in a shared discrete latent space, enabling fine-grained unified representation of multimodal sequences. During pre-training, we investigate various modality combinations, including audio-visual, audio-text, and the tri-modal combination of audio-visual-text. Extensive experiments on various downstream tasks, i.e., cross-modal event classification, localization, cross-modal retrieval, query-based video segmentation, and cross-dataset event localization, demonstrate the effectiveness of our proposed methods. The code is available at https://github.com/haihuangcode/CMG.

Achieving Cross Modal Generalization with Multimodal Unified Representation
Yan Xial*
Hai Huang
Jieming Zhu
Zhou Zhao 1.2f
"hejiann University
Shanghai Artificial Intelligence Laboratory
Huawei Noah's ' Ark Lab
xiayan . 2 zju@gmail. com jiemingzhu@ieee.com org
haihuanggood@@uutlodo com
com
zhouzhao@zju. edu.
Abstract
This paper introduces novel task called Cross Modal Generalization (CMG), which addresses the challenge of learning unified discrete representation from paired multimodal data during pre-training. Then in downstream tasks, the model can achieve zero-shot generalization ability in other modalities when only one modal is labeled. Existing approaches in multimodal representation learning fo- cus more on coarse- grrsied alignment or rely on the assumption that information from different modalities is completely aligned, which is impractical in real-world scenarios. To overcome this limitation, we propose Uni-Code, which contains key contributions: the Dual Coosl Information Disentangling (DCID) module and the Multi-Modal Exponential Moving Average (IIM-MMA)) These methods facilitate bidirectional supervision between modalities and align semanti- cally equivalent information shared discrete latent space, enabling fine-grained unified representation of multimodal sequences. During pre-training, we investi- gate various modality combinations, including audio-visual, audio-text, and the tri-modal combination of audio-visual-texx Extensive experiments various downstream tasks, i.c., cross-modal event classification, localization, cross-modal retrieval, query-basee video segmentation, and cross-dataset event localization, demonstrate the effectiveness of our proposed methods. The code is available at https//gtthob.commmaaauunggcccc
1 Introduction
Although recent years have witnessed significant achievements many multimodal areas, e.g., multi-modality question-answering [I] 2 3. query-based segmentation [ 5. audio-visual event localization [6] 77), labeling these tasks always consumes extensive human resources. Moreover, the labeling cost for different modalities
vary significantly [8]. leading scenarios where only a subset of modalities is labeled, while others remain scarce [9] For example, text-based visual segmentation is common while the label data for audio-based visual segmentation is rare. Consequently, models trained on such limited data are restricted to specific scenarios, hindering their broader applicability. Fortunately, unannotated paired multimodal data is readily accessible, such as the abundant image-caption [10] and audio-visual pairs [[]] available on the Internet. Hence, in this paper, we develop new task, Cross-Modal Generalization (CMG), for investigating how to learn unified discrete representations from these unlabelled multi-modal data pairs. Our objective is transfer the knowledge acquired from labeled modalities to other unseen modalities in downstream tasks, enabling models to generalize effectively.
"Equal Contribution
t "oorreppnnding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Uiiited Repreentation
Good Unifeed Representatin
Downstccom Task
United
laok
laking www.eneeeeeeeeeeeeeee..o.. ceceonseeeeeeneeennn
Cross Modal Gentrallaation
Dowrstroam inforence
United Dncrete trandler Tack Spoc fic Docoder
Figure 1: The overview of our proposed CMG task, different colors in the left and middle parts mean different semantics. The left part is the illustration of bad multi-modal unified representation, where features from various modalities sharing the same semantic meaning are mapped into disparate latent codes, while the good unified representation (middle part) totally different. The right part shows that in downstream tasks, the model will be directly transferred to unseen modalities.
Humans possess a natural ability to associate different modalities with similar semantics, facilitating moosl knowledge transfer based known modalities. Inspired by this, numerous studies have explored the integration of diverse multimodal information into a unified semantic space, which can be categorized into two types: implicit representations [12]
13 14] and explicit representations
15 16 17.. For implicit representations, several methods utilize a lagnosii- encoder [1] 13
to represent different modalities, employ contrastive learning [8 19 to bring different modalities closer in high-dimensional semantic space. In contrast, explicit representations aim to use a unified codebook 117 or prototype [16] to represent different modalities, serving bridge to facilitate robust alignment across modalities. Furthermore, the use of discrete space allows for the aggregation of similar input features in high-dimensional space, enabling complex feature representation with reduced number of latent codes. However, these studies predominantly compress sequential features within modalities into single vectors before quantization [16] 20 rely on the assumption that information from different modalities is completely aligned [7 21. As a result, these methods are typically limited simple tasks such as cross-modal retrieval and demonstrate poor performance in more complex scenarios.
As shown in [ the visual information in unconstrained videos comprises events of cat meowing and dog barking, while the audio information comprises events of woman speaking and dog barking. Directly applying previous methods [13 17
16 20] would lead to inaccurate mapping of multimodal information lacking shared semantics. To address this limitation, our paper primarily focuses on the implementation of fine- grained, unified representation for multimodal sequences. We address this novel problem from two key aspects: 1) extracting information with identical semantics across different modalities while mitigating the influence of modality-spectie details, and 2) representing these diverse modalities with shared semantics using unified codebook. Most of the previous works primarily focus on the second aspect while neglecting the importance of the first aspect, which we argue is crucial for achieving cross-modal generalization.
Our work builds upon the advantages of explicit representations and proposes two modules to address the aforementioned challenges. To address the first aspect, we propose Dual Cross-modal Information Disentangling (DCID) module, which combines Contrastive Log-ratio Upper Bound (CLUB) 22 with our proposed Cross-modal Contrastive Predictive Coding (23) (Cross-CPC). Concretely, we employ CLUB to optimize the upper bound of mutual information, efficiently distinguishing the modal-agnoste semantic information (conveying primary content or events) from the modal-specific information (additional details unrelated to semantics, such light and perspective in vision, or timbre and pitch in audio, etc.) across various modalities. However, without effective guidance, it is challenging for the model to identify which aspects of semantic information are useful (24]. Motivated by the complementary guidance nature of multi-modal information [25], we propose novel Cross-CPC method, to predict the future information in other modalities based on the known sequence information of the current modality, which can effectively maximize the fine- tne-grained mutual information between different modalities. To address the second aspects, we propose Multi-Modal Exponential Moving Average (MM-EMA), which employs a teacher-studen mechanism to facilitate

---

cross-modal guidance during pretraining, This mechanism encourages the aggregation of quantized vectors encoded from different modalities with shared semantics into the same latent code space. Otherwise, distinct modalities would aggregate in separate regions of the discrete space, rather than being mapped together 20 To summarize, our main contributions are threefold:
We introduce new task, CMG, which enhances the applicability of existing models in the realm of multimodal learning, addressing the challenges arising from modality scarcity and the substantial cost of annotating certain modalities.
We propose a novel framework named Uni-Code, which can effectively extract shared se- mantic information from paired multi-modal data and project them into a common quantized latent space in a fine-grained level.
We investigate the unified representation of various modalities, including audio-visual. audio-text, visual-text, and even the challenging tri- mo-dm combination of andio-visuall textt Extensive experiments on various downstream tasks, e-g., multimodal event elassification, localization, cross modal retrieval and video segmentation, demonstrate the effectiveness of our proposed methods.
2
Related Work
Implicit Multi-Modal Unified Representation. The past few years have witnessed remarkable achievements in the implicit multi-modal unified representation, which aims to align diverse modali- ties within a shared latent space [26] 27 14] or try to learn a modality
encoder for extracting information across various modal 28 29 These methods investigate the unified representation for
various modality combinations, i.e., Speech- Text 226 30). Video-Audio [1] 32 (3) Vision- Text
To achieve these, different kinds of methods have been proposed. Pedersoli et al. (31) and Sarkar et al. (27) introduce a cross-modal knowledge distillation to transfer knowledge across modalities. CLIP-based methods [8 14 use contrastive loss to learn image-text consistency from large paired dataset, and have gained incredible zero-shot ability in various downstream tasks.
18 14 29]
Explicit Multi-Modal Unified Representation. Recently, some works investigate how achieve multimodal explicit unified representation by utilizing a universal codebook [15] 17 20] or prototype [6] to explicitly express multimodal content. Duan et al. [16] apply Optimal Transport the feature vectors extracted from different modalities to the prototypes. Zhao et al. 20 use self-
map
coss-recontttutttonn to enhance the mutual information between different modals. However, these coarse-grained alignment methods [6 20 are only suitable for simple tasks such as retrieval, and are unable to accomplish fine-grained comprehension in downstream tasks. In recent years, align the text and speech temporally with a unified discrete space. Liu et al. [17 use the same scheme to align short videos and speech/text. However, a key assumption underlying their success that the modalities they chose have a one-to-one alignment (i.e. text-to-speech) while in more general case, it is hard to guarantee the semantic information in the two paired modalities is completely consistent (i.c. unconstrained video and audio). In this paper, we mainly investigate how to map paired multimodal sequence into a common discrete semantic space, where the information in the multimodal sequence is unconstrained and may not be perfectly aligned.
(34 21
Mutual Information Estimation. Mutual information (MI) estimation aims measure the depen- dence between two random variables. Recent works primarily concentrate on how to combine MI including MI maximization and minimization. MI maximization aims to learn representations that capture meaningful and useful information about the input data, leading to improved performance in various downstream tasks [36] 37 38 In order to
estimation with deep neural networks 35
22 2
maximize mutual information in a sequence, Contrastive Predictive Coding (CPC) 23 employs an autoregressive model and contrastive estimation to capture long-term relations while maintaining local features within sequence. MI minimization tries to reduce the dependency between two random variables while preserving the relevant information, has been successfully applied in disen- tangled representation learning [39 24]. Belghaz et al. [35] propose a Mutual Information Neural Estimator (MINE), which builds a neural network to estimate mutual information based on dual representations of the KL-ddvergence 40 41 Chen et al. 22 introduce Contrastive Log-ratio Upper Bound (CLUB), combining MI estimation with contrastive learning to approximate the MI upper bound. Their work is more suitable for MI minimization.
3

---

(a)
Unified Codstock
(b)
A.P.
Video VQ
Aucio VD
Surli Model
Weall fostur
Vector Quantiattion
Audo Fastum
Insootrrction
Cross CPC
demantc
Auds Semantic
Autio-Caacc
Nodd
Visual Backbone
Audio Backbone
AcspgIssio Madd
The overview of our proposed Uni-Cod framework, we use audio as example. (a) The main pipeline our model. (b) The process of Multi-modal Vector Quantization, which contains Multi-modal EMA (MM-EMA) and a new commitment loss, notice that the abundant discrete codes will be reset. (c) The architecture our proposed Cross-CPC.
Figure 2:
3 Cross Modal Generalization Task
Given a set of paired multi-modal data X = {x; x. xf,xx".., ...) of size N, where A, B, C, and so on represent different modalities, the Cross Modal Generalization (CMG) task aims to map these various modalities into a unified discrete space during the pre- pre-tnnin phase, enabling discrete latent codes be shared among different modalities with the same semantic. Subsequently, in downstream tasks, when only one modality (e.g., mode A) has annotated information, the model can transfer knowledge learned from A mode to other modalities (e.g., mode and C) based on the shared discrete space obtained during pre-training to achieve zero-shot generalization ability.
4 Unified Representation Learning
Different from previous works which simply extract information from paired modalities and then direct maps, we argue that the success of unified representation lies in the extraction of modality- agnostic semantic features. Thus in this paper, we achieve this from two perspectives: first, we introduce a DCID module, designed to extract fine-grained semantic information and separate it from the corresponding modality-speciffe information within each modality. Second, we compress the extracted semantic features into discrete variables through VQ-VAR ensuring that the compressed discrete variables can still contain the original semantic information through reconstruction loss. For simplicity, we take two modalities as an example to illustrate the process. Fig givess an overview illustration our network.
4.1 Baseline
fi=1 we first utilize two semantic encoders to and Φb to extract modal-agnostic features z z E Rd RTDD and use two modal-specific encoders e and y to extract remain features 2i, x from modality A and B, respectively:
Given two paired modals, {(x., (xxxx)) )
th (x
- = vr((x))) z =x((x)) = .'x")).)
(1)
where T, represent time and hidden dimension, respectively. The dimension of z z various for different modalities. Then we apply vector quantized operation map the semantic features zi,,, 2i, discrete latent codes in fine-grained level, [0, 7]:
- VQ(*"" (x.)) VQ(2.?) = et: where 1 argmin, ( eyl12, m (a, (2)
the latent codebook E R D is shared across modality A and B, where L is the the size of the discrete latent space. Finally, we combine 2.7h with z!nh together to reconstruct original features:
x D(2, 11
|g[[" (x)")
ell + 3110² (x"") sgie]lle.
m (a, (3)
reconstruction loss
VQ lasss
commitment loss

---

where 9 is 0.25 for all our experiments, sg is stop gradient. In this work, we use Exponential Moving Average (EMA) replace VQ loss, since EMA is more robust. The reconstruction loss can guarantee that the compressed latent codes e still maintain the semantic information of different modalities. In ideal conditions, z and 2, encoded from different modalities with the same semantics should be mapped to the same discrete latent code. However, without effective supervision, the existence of modality gap will result in z and z converging to separate regions of the codebook [7] Thus in this paper, we introduce the following modules to alleviate these problems.
4.2 Dual Cross-modal Information Disentangling
We introduce our DCTD module from two aspects: MI minimization between modal-agnostic semantic features and modal-speciae features in each modality (CLUB), and MI maximization between modal- agnostic semantic features across different modalities (Cross-CPC).
CLUB-based MI Minimization: Compared to the methods that try to optimize the MI lower bound, such as InfoNCE 23 and MINE [35], CLUB 22 can effectively optimize the MI upper bound, demonstrating superior advantages in information disentanglement (24) Given two variables x and y, the objective function of CLUB defined as:
I.CLUB (cLuB(x:y) y) Epxx.y) ()(x))) Ex(x))pr)) [lg ( x)],
where go is the variational approximation of ground-truth posterior of given x and can be parame- terized by network 0. We use CLUB to optimize the MI upper bound between the semantic features zz" and modal-specift features z!", E {a, b}. we modify LICLB into temporal version:
Í.CLUB Σ T
log go (2!T
11
Σ j=1t=1
log go (2) 12]) m E {a, (5)
The approximation network and the main networks are optimized alternatively during pre-training, Finally, we can reduce the correlation between semantic information and modal-specific information in each modality. Nevertheless, merely minimizing mutual information using CLUB presents challenge for the model identify relevant semantic features. Given that paired multimodal information can provide mutual guidance and serve as supervisory signals for each other, we devise the Cross CPC approach to alleviate this issue.
MI Maximization with Cross-CPC: Contrastive Predictive Coding (CPC) 23 can maximize the mutual information among the adjacent items within sequence by predicting the future samples with powerful autoregressive models, has been widely used in self-supervised learning. While for human, we can not only predict subsequent scenarios based on the current modality, but are also capable of associating with potential situations that may occur in other modalities. For instance, people can infer forthcoming auditory information based on a presently viewed video segment or read text, or envision a subsequent scene by perceiving an audio portion. Inspired by this, in this paper, we extend CPC to cross-modal constrictive prediction. Given the semantic features z", z E R prediction of K steps and random time moment t E (0, T-K], we first use two single layers unidirectional LSTM to summarize the information of all zo 2 and can obtain two context representations as ct"h LSTM(ZZ)) RD, {a, b}.
For modality A, we first select a set Z6 of N-1 random negative samples and one positive sample zi+k from modality B, then we use of to predict k-th future step 2 in modality B, and the InfoNCE loss for modality A can be optimized as:
1 K
eapl explz)+e Enjezs explz)
1
K Σ log explzi Whot) Enjez, eap eaple;
Iaab-
Diog
b2a
K
where the WE the linear projection matrix for different step k. The optimization L 5 for modality B is vice versa. Based on these, features with the same semantics across different modalities can be mutually extracted through fine-grained cross-modal prediction. Despite is difficult to align paired modalities perfectly in unrestricted scenarios, the model can still predict the possible information in the next step of the other modality at the fine-grained level through contrastive learning, due to the summarization of the historical information using an autoregressive model.
5

---

4.3 Multi-modal Exponential Moving Average
struggle to achieve fine mne-griindd cross-modal alignment unconstrained scenarios, to tackle their limitations, we propose MM-EMA which can allow different modalities to serve as teacher-student iteratively and update each other during the quantization process.
Previous works 20 16
First, we use cross-attention 42 to extract related information from the opposite modality, taking mode A an example: r cross-att 2"; 2²), where x is query, 2 is key and value. The vector r contains the semantic information derived from z exhibiting a strong correlation with zf. while also preserving the intrinsic attributes of modality B. Thus, can serve as an intermediary facilitating the alignment of modality A with modality B during EMA procedure.
Given code vector C₁ we can obtain n. semantic vectors of modality {2 1 and nę semantic
vectors of modality B { [ f=1 that are quantized to e;;
N ((()) (N(( +1 [n, a(t)
6(t)
(t)
(4)
/N!")
+2
O
200)
a(t)
B(E)
n
M(B)
b(t)
(()) 1.
of' = 70, 11 + (1 (1-7)
Σ
1.
2
2
where is batch sequence in order, N₁ and o; are accumulated vector count and volume, respectively, Besides, we also modify the commitment loss in Eq which can use the code vector c (quantized from z²) as a teacher, to guide the Encoder Φ towards not only approximating e but also converging to e with a certain ratio (50% in our setting, /3 is the same as in Eq 3
commiit = 33100 (x{) - sg 1g + 2
of (x+) sg 1
(8)
The modified commitment loss Dommmii also the same process with modality During training, the distance between different modalities in the latent space gradually decreases.
Reset inactivated code: In order to alleviate the codebook collapse problem existing in VQ process, which implies that only a subset of codes are extensively utilized while the majority of codes remain inactive, we adopt code reset strategy: for given code vector e,, it will be retained only if it has been selected at least once by both modality A and modality B in Nye consecutive batches, whereas the remaining codes will be ce-initialized. This strategy can effectively mitigate the issue of some code being perpetually uncxploitable due to poor initialization, as well as the problem of generating redundant code during the process of aligning modalities A and B (see appendix for more details).
4.4 Training and Downstream Tasks
The full objective of our pre-training framework is the combination of all above objection functions: = Lrrcon + Lecommst + Lepe + Leemem + L where L is 1.CLUB. Lemom is the objective loss proposed in [17. which can also promote the alignment among modalities. All these objectives except Lemen integrate two modalities. After training, we can obtain a unified discrete latent space.
In this paper, we investigate the unified representation of various modalities, including audio-visual, audio-text, visual-text, and even more, the tri-modal combination of audio-visual-text Then we can apply the pre-trained multi-modal encoder in various downstream tasks, i.e., cross-modal event classification, event localization, query-basee video segmentation and cross both domain & modal event localization. Note that during downstream tasks, the parameters of the pre-trained encoder are frozen.
Experiments
5.1 Datasets and Tasks
Pre-train: We use IGGsound-AVEL. 44 45 to pre-train our unified representation, and divide into several different sizes: 24K, 40K, 81K. Considering that the VGGsound-AVEL dataset only contains audio, video and event label, we design prompts for these labels and modify them into descriptive sentences, see Appendix for more details.

---

Table 1: Compared with sate-of-thee methods on two downstream tasks. We use precision to indict the performance of the models on AVE tasks, and use accuracy for AVVP tasks.
VGGsounds-AV
24K
VGGsounds-AVe 40K
EGGsounds-AVER 81K
Method
AVE
AVVP
AVE
AVVP
AVE
AVVP
V-+A A->V
V-AA-A A-+V
V-+A A-
V-A
A-+V
V-+A
A-+V
Bascline
4.4
5.9
7.6
8.4
5.5
5.4
6.9
8.7
7.1
9.3
5.6
7.2
S-Mit 43
12.7
16.9
17.2
22.8
14.4
15.9
19.0
22.3
13.4
17.0
20.9
22.8
MST 13
13.3
19.0
25.7
29.1
19.5
23.1
22.7
24.5
18.6
20.5
19.1
24.8
CODIS 16
18.5
22.0
29.4
33.7
20.8
26.4
35.1
37.9
28.5
30.2
34,0
37.8
20
17.7
21.0
29.4
32.4
19.1
24.3
36.9
39.3
27.6
31.4
33.8
38.1
CMCM[1
28.9
35.9
42.6
50.4
32.7
36.8
41.9
45.1
31.1
34.0
39.3
44.8
DCID4S-Mit
28.1
32.3
45.9
49.2
32.2
34.0
47.8
53.0
34.8
37.6
51.9
53.5
DCID+MST
31.2
35.0
50.7
52.1
34.9
37.8
54.4
59.1
33.5
35.4
DCID+TURN
29.4
35.3
53.4
56.0
29.7
36.9
55.2
58.2
31.9
36.8
DCID+CODIS
33.4
36.0
53.8
60.2
36.7
41.0
52.6
62.0
35.9
40.1
54.3
59.0
DCID+CMCM
34.1
38.8
57.6
60.8
36.4
42.9
58.7
62.8
38.8
41.4
57.5
60.5
Uni-Code
44.0
49.7
61.9
65.7
47.7
52.3
64.0
65.6
41.2
45.6
60.5
61.7
Table 2: Ablation studies of audio-visual pre-training AVE and AVVP tasks.
VGGsounds-AVE
24K
VGGsounds-Ae AVEL 40K
CLUB Cross-CPC
MM-EMA
Reset code
Lęmem
AVE
AVVP
AVE
AVVP
>A
V
A
>
V
>
A
v
>
>
34.9
35.1
30.6
54.0
37.2
40.3
52.9
59.5
4.6
5.8
10.9
24.6
5.2
7.1
12.3
24.1
29.8
34.6
30.4
32.5
35.2
36.9
32.3
34.0
34.1
38.8
57.6
60.8
42.9
58.7
62.8
V
37.8
41.2
59.1
61.5
38.9
40.3
55.4
62.1
39.7
42.6
58.2
62.1
41.3
46.0
58.7
62.8
28.2
30.9
41.4
49.2
31.5
33.8
46.0
48.3
44.0
49.7
61.9
65.7
47.7
52.3
64.0
65.6
Downstream: we evaluate the pre-trained models on several downstream tasks using different datasets: Cross-modal event classification (AVE (46)): In AVE, each video contains primary event that is present in both audio and visual information. Therefore, we can train an event classifier using a single modality (e.g.. video) and directly evaluate the classifier's performance on another modality (e.g., audio). Cross-modal event localization (AVVP [47): In AVVP, portion of the data is annotated with fine-grained event labels for both audio and visual modalities. Similar to AVE, we can perform event localization on one modality and then directly transfer the model to the other modality for testing. Cross both modal and dataset localization/classine clasiffiaat:::: In order to prove that our model can also achieve cross modal transfer different downstream datasets, classification tasks for one modality AVE to train, and directly test fine-grate ane-grained event localization ability of the model another modality in AVVP (from AVE AVVP). Also test the cross- modsl classification tasks between the visual modality in part of 101 (16 classes) and audio modality in part of VGGSound-AVEI (16 classes). Cross-modal video segmentation (AVSBench-S4 (48) In AVS, each piece of AVS data contains a piece of audio, a video clip of the object that emits the sound, text label corresponding the object, and manually labeled mask. We expand each text label into a descriptive sentence. We use one modality (e.g., audio) to train the query-based video segmentation, and directly test the segmentation ability in another modality (e.g., text). Cross-modal retrieval (VGGSound-AVEL, We select retrieval task to demonstrate that our method can also be applied visual-text generalization as well. We use audio as an inter-medium measure the generalization ability of our model across these two modalities and implement an X-to-audio retrieval task. To be detailed, the first stage, we train visual-text unified representation learning using VGGSound24K dataset, and then in the second stage, during downstream training, we let the model learn text(video)-audio retrieval, finally during inference, we directly test the generalization ability of the model on video(text)-and retrieval.
cross-modal
5.2 Baselines and Implementation Details
We use the method depicted Section 4.1 as our baseline. We also compare our methods with several state-of-the-art unified representation methods: MST [13], CODIS [16), S-MiT [43), CMCM [17).

---

Table Ablation studies of audio-visual-Eee pre- traiiiinnng on three downstream tasks.
VGGsounds- 40K
CLUB
Cross-CPC
MM-EMA
Reset code
Lenom
AVE
AVVP
AVE-+AVVI
UCFVV-VGGGa
y-AA--M
V-AA A-AA- A
V--A--V
V--AA-X
50.2
51.8
62.4
66.2
50.1
51.2
9.87
43.8
49.2
59.3
61.1
45.5
50.6
60.6
54.6
52.9
49.9
62.0
67.3
48.1
46.8
66.5
60.5
33.0
35.5
56.7
61.2
7.4
12.6
43.3
35.2
56.4
50.4
60.0
52.4
54.5
57.5
72.9
50.5
48.7
69,9
59.7
54.1
55.0
63.4
71.0
53.0
52,4
67.1
60.6
Evaluation results of the laheled modality
64.8
65.8
71.0
72.9
80.0
85.4
AVC
AVC
86
66
Abcuraa
50
60
20
20
256
400
612
B00
1024
266
400
612
800
1024
Pretrained audio-visual.
(b) Pretrained audio-visual-tet..
Figure 3: The effect of different codebook sizes on two pre-training tasks.
TURN [20]
tasks and test their performance two downstream tasks. We use the perception to evaluate the performance on AVE, VGGSound-AVEs and UCF101 datasets, and use accuracy (%) to evaluate the performance AVVP datasets, and use Fi-score (%) to evaluate the performance for AVE to AVVP generalization task, and use mloU and F-score (as same AVS (8) for AVS-S4 dataset. The implementation details are provided in Appendix.
We implement these methods
5.3 Compared with State-of-the-art Methods
We mainly compare model with these leading methods
downstream tasks, cross-modal event classification and localization, all these models are pre-trained using audio-visual modalities on three different sizes of datasets. As we can see in Table [| our methods can outperform all these previous works on three settings. The baseline results indicate that without any constraints, it is challenging for the model learn the alignment relationship between different modalities during pre- pra-training, leading to poor performance in downstream tasks. Moreover, the experiment results also demonstrate that merely employing these state-of-the-ang methods such unconstrained audio-visun alignment has considerable limitations. However, when incorporated with our proposed DCID module, these models exhibit substantial improvements downstream tasks. This further suggests that our proposed DCID can effectively decouple the semantic information of different modalities.
In addition,
observe from Table that when the scale of pre-training data increases from 24K to 40K, there is significant improvement in the performance of the model on downstream tasks. However, when the scale reaches 81K, the performance of the model does not improve much and even declines. This also indicates that excessive pre-training data may not yield desirable results, only an appropriate amount of data can enable the model to learn the correct alignment relationships.
5.4
Ablation Studies
Main Components We use the audio-visual unified representation as pre-training task and test the performance on the downstream AVE and AVVP tasks by removing each main component of our model, the results are depicted in Table 2 removing these components. In the DCID module, the impact of removing Cross-CPC is far greater than that of removing CLUB, which indicates that without the fine- graingrindd cross-modal alignment constraints provided by Cross-CPC, it is difficult for CLUB alone to extract effective semantic
We can observe that the performances decline after

---

information from the original modal features. When MM-EMA is removed. the model performance also declines significantly. demonstrating that MM-EMA can effectively utilize the Cross-Attention mechanism between different modalities to gradually map the relevant information together, helping to achieve a unified representation. The experimental results of removing the "reset code" indicate that this module can effectively remove the redundancy latent codes caused by MM-EMA.
The Effect of Codebook Size Different codebook sizes may affect the effectiveness of modality aggregation, thereby influencing the performance of downstream cross-modal generalization tasks. Here we investigate how the size of codebook influences pre- mre-trainin performance. From Fig we can see that the appropriate codebook size can achieve the best pre- pre- training performance, while too large size (which may cause signals from different modalities aggregate at different positions the codebook) and too small size (where semantically distinct signals are erroneously mapped together due to limited capacity) will both degrade the performance of pre-trained models.
Unified Representation of Three Modalities Compared with previous works that can only align two modalities together, our method can disentangle and align the semantic information of three modalities, as shown in Table 3 From the results we can see that the removal of these modules will lead to a decline in the performance of the pre-trained model on downstream tasks, similar to the results in T However, compared to the unified representation of the two modalities, the removal of these modules leads to a smaller decline in the model performance. We think the reason is that the introduction of the third modality can facilitate the other two modalities approaching each other, and the performance of the model pre-trained on three modalities is significantly higher than that of the model pre-trained on two, which also illustrate this point.
see in Table 4 our pre-trained model can effectively transfer the segmentation ability across modals. The visualization of segmentation results in Fig our model can accurately identify vocal regions in video frames for unknown modalities. Furthermore, the performance of our method is closer to the evaluation results of the AVS model on source modality. More visualization results and analysis can be found in Appendix.
Results AVS-S4 As
Results on Cruss Modal Retrieval Tasks We also test the performance of X-to-audio retrieval task under cross modal generalization setting, where X can be visual or text, the results are shown in 5 Compared with baseline model, our methods can greatly advance the audio retrieval ability in both two generalization directions.
-10
(a) baseline
(b) CMCM
(c) Uni-Code
Figure 4: Visualization of the discrete codes. The blue dot means that the number of visual modal quantization accounts for more than 80% of the total number of audio and video quantification, the red dot means that the audio station accounts for more than 80%. and the green dot means that audio and video each account for more than 20%. As we can seen in (b), the discrete codes obtained by CMCM still contain a little single modality mapped codes, while ours (c) are almost all green codes which can be mapped by both two modalities.
5.5 Qualitative Analyses
To further illustrate the effectiveness of our model during pre-training phase, we take audio and video as an example and visualize the latent discrete codes obtained from two modalities. As we can see in aa) when there is no additional supervisory signal, different modalities are difficult to map to the same discrete code, which also proves why the baseline performs poorly downstream tasks. The CMCM [17 method can effectively alleviate this problem, but there are still many codes that can only be mapped by single modality. However, our model enables information from different modalities but with the same semantics to be mapped into the same codes. (as shown in Fig 4 c)).

---

Table 4: Performance on AVS-S4 datasets (pre- trained on audio-visual-tees visual-text modalities).
Table Performance audio retrieval tasks under cross modal generalization directions.
A2T
T2A
Methods
mloU
F-score mloU F-score
V2T
T2V
Methods
Baseline
69.8
81.4
69.9
81.3
R@5
R
10
R@5
R 10
Our full model
78.0
87.1
77.7
86.7
Baseline
0.47
1.03
0.62
0.85
SST
49
(A2A)
60.3
80.1
Our full model 10.3
21.9
8.47
16.7
AVS
48
(A2A)
78.7
87.9
Text: The ambulance is driving and honking
Text: These lions are rearing
baseline
baseline
Visualization results of A2T (left) and T2A (right) of our model on AVS-S4 dataset. We compare our method with the baseline model.
Figure 5:
Conclusion and Discussion
In this paper, propose new Cross Modal Generalization task, and bring up two innotative modules, DCID and -MM,, which can achieve multi-modal unified representation. We achieve unified representation of different multi-modal combinations, including audio-visual, audio-text and audio-visual-text. Also, we conduct extensive experiments on various downstream tasks. The extensive experiments demonstrate the effectiveness of proposed methods. Furthermore, our model can help existing models to extend into other modalities, achieving more extensive applications. Limitations and Boarder Impact: In our work we only focus on the unified representation of three modalities. However, our method can inspire future works to explore the combination of more modalities. Our work do not contain potential negative social impact.
7
Acknowledgments
This work is supported by National Key R&DD Program of China under Grant No. No.2022ZD0162000. National Natural Science Foundation China under Grant No. 62222211 and No.6183666 We also gratefully acknowledge the support of MindSpore (https://wer mindspore cn), which is new deep learning computing framework.
References
[1] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015.
[2] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. arXiv preprint arXiv:1809.01694 2018.
[3] Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19108-19118, 2022.
[4] Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, and Cees GM Snock. Actor and action video segmentation from sentence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5958-595888 5966, 2018
[5] Seonguk Seo, Koon-Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with large-scale benchmark. In Computer Vision-ECCV 2020: 16th
10

---

European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16, pages 208-223: Springer, 2020.
[6] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European Conference Computer Vision (ECCV), pages 247-263, 2018.
[7] Haoming Xu, Runhao Zeng, Qingyao Wu, Mingkui Tan, and Chuang Gan. Cross-modal relation-aware networks for audio-visual event localization. In Proceedings of the 28th ACM International Conference on Multimedia, pages 3893-3901, 2020.
[8] Huijuan Xu and Kate Saenko. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. Computer Assin-EECYs 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VII 14, pages 451 451-466. Springer, 2016.
[9] Tamás Matuszka, Iván Barton, Ádám Butykai, Péter Hajas, Dávid Kiss, Domonkos Kovács, Sándor Kunsági-Mäté, Péter Lengyel, Gábor Németh, Levente Petó, et al. aimotive dataset: A multimodal dataset for robust autonomous driving with long-range perception. arXiv preprint arXiv:2211.09444 2022.
[10] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[11] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 721-725. IEEE, 2020.
[12] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self supervuper learning from raw video, audio and text. Advances in Neural Information Processing Systems, 34:24206 14222226-24221, 2021.
[13] Haoxuan You, Luowei Zhou, Bin Xiao, Noel Codella, Yu Cheng, Ruochen Xu, Shih-Fu Chang, and Lu Yuan. Learning visual representation from modality-shared contrastive language-image pre-training. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVII, pages 69-87 87. Springer, 2022.
[14] Alex Andonian, Shixing Chen, and Raffay Hamid. Robust cross-modal representation learning with progressive self-distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16430-16441, 2022.
[15] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint MxXiv:2006009166 2022.
[16] Jiali Duan. Liqun Chen, Son Tran, Jinyu Yang, Yi Xu, Belinda Zeng, and Trishul Chilimbi. Multi-modal alignment using representation codebook. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 15651-15660, 2022.
[17] Alexander H Liu, SouYoung Jin, Cheng-I Jeff Lai, Andrew Rouditchenko, Aude Oliva, and James Glass. Cross-modas discrete representation learning. arXiv preprint arXiv:2106. 05438, 2021.
[18) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8766 0866.8763..10 PMLR, 2021.
[19] Victor Weixin L Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:17612-17625 2022.
11

---

[20] Yang Zhao. Chen Zhang, Haifeng Huang, Haoyuan Li, and Zhou Zhao. Towards effective multi-modas interchanges in zero-resouree sounding object localization. Advances in Neural Information Processing Systems, 35:38089-38102. 2022.
[21] Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, et al. SpeechtS: Unified-modal encoder-decoder pre-training for spoken language 10.00205, 2021.
processing. arXiv preprini
[22] Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: contrastive log-ratio upper bound of mutual information. In International conference on machine learning, pages 1779-1788. PMLR, 2020.
[23] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 0087403778, 2018.
[24] Disong Wang, Liqun Deng, Yu Ting Yeung. Xiao Chen, Xunying Liu, and Helen Meng. Vgmive: Vector quantization and mutual information-basee unsupervised speech representation disentangiement for one-shot voice conversion. arXiv preprint arXiv:2106. 10132, 2021.
[25] Yan Xia and Zhou Zhao. Cross-modal background suppression for audio-visual event localiza- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19989-19999) 2022.
[26] Stavros Petridis, Themos Stafylakis, Pingchuan Ma, Georgios Tzimiropoulos, and Maja Pantic. Audio-visual speech recognition with a hybrid etc/attention architecture. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 513-520. IEEE, 2018
[27] Pritam Sarkar and Ali Etemad. Xkd: Cross-modal knowledge distillation with domain alignment for video representation learning. arXiv preprint arXiv:2211.13922 2022.
[28] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, pages 104-120. Springer, 2020.
[29] Teng Wang. Wenhao Jiang, Zhichao Lu, Feng Zheng. Ran Cheng, Chengguo Yin, and Ping Luo. Vlmixer: Unpaired vision-language pre-training via cross-modal cutmix. In International Conference on Machine Learning, pages 22690- PMLR, 2022.
[30] Chengyi Wang. Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng. and Xuedong Huang. Unispeech: Unified speech representation learning with labeled and unlabeled data. In International Conference on Machine Learning, pages 10937-10947. PMLR, 2021.
[31] Fabrizio Pedersoli, Dryden Wiebe, Amin Banitalebi, Yong Zhang, and K Kwang Moo Yi. Estimat- ing visual information from audio through manifold learning. arXiv preprint 2008.2200..02337, 2022.
[32] Simon Jenni, Alexander Black, and John Collomosse. Audio-visual contrastive learning with temporal self-supervision. arXiv preprint arXiv:2302.07702, 2023.
[33] Adrià Recasens, Jason Lin, João Carreira, Drew Jaegle, Luyu Wang. Jean lean-baptiste Alayrac, Pauline Luc, Antoine Miech, Lucas Smaira, Ross Hemsley, et al. Zorro: the masked multimodal transformer. arXiv preprint 2301:::303000999 2023.
[34] Chi Han, Mingxuan Wang, Heng Ji, and Lei Li. Learning shared semantic space for speech-to- text translation. arXiv preprint arXiv:2105. 100.009555 2021.
[35] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio. Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning, pages 531-540. PMLR, 2018.
[36] Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint 1907. J 13625, 2019.
12

---

[37] Lingpeng Kong, Cyprien de Masson d' Autume, Wang Ling, Lei Yu, Zihang Dai, and Dani Yogatama. A mutual information maximization perspective of language representation learning. arXiv preprini arXiv: 1910008300 2019.
[38] Malik Boudiaf, Imtiaz Ziko, Jérôme Rony, José Dolz, Pablo Piantanida, and Ismail Ben Ayed. Information maximization for few-shot learning. Advances in Neural Information Processing Systems, 11:2455-4457 2020.
[39] Wei Zhu, Haitian Zheng. Haofu Liao, Weijian Li, and Jiebo Luo. Learning bias-invariant repre- sentation by cross-sample mutual information minimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15002-15012. 2021.
[40] Solomon Kullback. Information theory and statistics. Courier Corporation, 1997.
[41] Avraham Ruderman, Mark Reid, Darío Garcfa-Garcia, and James Petterson. Tighter varia- tional representations of f-divergences via restriction to probability measures. arXiv preprint arXiv: 4664, 2012.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[43] Mathew Monfort, SouYoung Jin, Alexander Liu, David Harwath, Rogerio Feris, James Glass, and Aude Oliva. Spoken moments: Learning joint audio-visual representations from video descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14871-1488/, 2021.
[44] Jinxing Zhou, Dan Guo, and Meng Wang. Contrastive positive sample propagation along the audio-visual event line. IEEE Transactions on Puttern Anulysis and Machine Intelligence, 2022.
[45] Jinxing Zhou, Liang Zheng. Yiran Zhong, Shijie Hao, and Meng Wang. Positive sample propagation along the audio-visual event line. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8436 8444, 2021.
[46] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European Conference on Computer Vision (ECCV), pages 247-263, 2018
[47] Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly- supervised audio-visual video parsing. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 436-454. Springer, 2020.
[48] Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang. and Yiran Zhong. Audio-visual segmentation. In Com- puter Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII, pages 386-403. Springer, 2022.
[49] Brendan Duke, Abdalla Ahmed, Christian Wolf, Parham Aarabi, and Graham W Taylor. Sstvos: Sparse spatiotemporal transformers for video object segmentation. In Proceedings of the IÉEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5912-5921, 2021.
13


Reviews:
[{"review_id": "ilEBVSGXKO", "reviewer": "Reviewer_kKgn", "summary": "The paper proposes a new pretraining task called Cross Modal Generalization (CMG) for learning unified multimodal representations. The goal is to map different modalities (e.g. audio, visual, text) to a shared discrete latent space during pretraining, such that the model can generalize to unseen modalities in downstream tasks when only one modality is labeled. The paper solves two issues: (1) unified semantic features shared cross-modalities. (2) representing these semantic features using a unified codebook. The paper mainly contributes to the first aspect. ", "questions": "- In the formula (1), are the $\\Phi^{a}$ and $\\Phi^{b}$ the same in the implementation?", "limitations": "- None.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- New task is brought to the community and the task makes sense to me because in reality, only partial data is labeled and other with them is largely not.\n- DCID and MM-EMA are intuitive methods for disentangling and aligning semantic information across modalities. Using mutual information optimization and teacher-student aggregation.\n- Experiments and ablation studies are quite solid.", "weaknesses": "- I didn't find weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer kKgn:\n\nThank you very much for your acknowledgement of our methods and giving us positive feedback! We will respond to your question as follows:\n\n**In the formula (1), are the ${\\Phi}^{a}$ and ${\\Phi}^{b}$ the same in the implementation?**\n \n Sorry about the misunderstanding. For different modalities, ${\\Phi}^{a}$ and ${\\Phi}^{b}$ are not always the same in the implementation. For a given visual feature $V \\in R^{B\\times T \\times H \\times W \\times C}$, we first apply average pooling operation on it and obtain $\\bar{V} \\in R^{B\\times T \\times C}$. We follow the Spatial-Channel Attention proposed in CMRAN [1], which use the $\\bar{V}$ to calculate attention score with $V$ in spatial and channel level, then we apply self-attention in temporal level and get the final visual semantic results as $Z^{v} \\in R^{B\\times T \\times C}$. For audio and text input feature, we directly use a self-attention layer in temporal level to get the final audio or text semantic features as $R^{B\\times T \\times C}$. Thank you very much for pointing this out, we will add these implementation details in appendix. \n\nWe thank the reviewer again for your very positive feedback! If you have any further questions or comments, please let us know, we are glad to respond.\n\n[1] Cross-Modal Relation-Aware Networks for Audio-Visual Event Localization"}, {"review_id": "vs3eI4bEvJ", "reviewer": "Reviewer_jARS", "summary": "This paper first introduces a new task called Cross Modal Generalization, which aims to learn a unified discrete representation from paired multi-modal data during pre-training, and realize zero-shot generalization in other modalities in downstreams tasks. This paper proposes Dual Cross-modal Information Disentangling (DCID) module and Multi-Modal Exponential Moving Average (MM-EMA) to facilitate bidirectional supervision between modalities and align semantically equivalent information in a shared discrete latent space. Experiments on various downstream tasks validate the effectiveness of the proposed methods.", "questions": "1. The subfigure (b) and (c) in Figure 4 do not look much different. May consider a better visualization, such as the color of the points.\n2. \"sg\" in Equation (3) and (8) is not explained.", "limitations": "The authors mention limitations that they only focus on the unified representations of three modalities and future works can explore more modalities.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Introduce a new task CMG, mapping various modalities into a unified discrete space.\n2. Propose DCID and MM-EMA, extracting shared semantic information and project them into a common quantized latent space.\n3. Performance significantly outperforms previous methods.", "weaknesses": "1. This paper only perform modality transfer on one pair of modalities, A & V.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer jARS:\n\nWe appreciate your positive feedback and providing very valuable suggestions. Let us respond to your questions point by point.\n\n---\n\n**For Weaknesses: This paper only perform modality transfer on one pair of modalities, A & V.**\n\nThanks for asking! Yes, most of our pretraining and downstream experiments are conducted on paired audio-visual dataset. However, to further demonstrate the effectiveness of our methods, we also pretrain our model on **audio-text** and **audio-visual-text** combinations. And the results on **downstream referring video segmentation** tasks (see Table 4, Figure 5 and Line 301-305) illustrate that our methods can also successfully learn a unified representation of audio & text, and transfer the video segmentation ability from seen modality (e.g., text) to unseen modality (e.g., audio) on AVS-S4 dataset. \n\nIn addition, we conduct a series of experiments on **unpaired downstream datasets**: transferring the event localization ability of the model from **one modality in AVE dataset** to **unseen modality in AVVP dataset** (Line 242-245), as shown in the right part in Table 3. The results prove that even though the seen and unseen modalities come from different sources, our method can still guarantee a strong zero-shot cross-modal generalization ability with unsupervised pretraining. We will add more analysis about these in our paper. \n\nFurthermore, according to the suggestion of Reviewer WtGp, we also add a simple cross-modal retrieval experiment to demonstrate that our model can also be extended to visual-text. Please see the ''**For Weakness 3 & Questions**'' section of the reply to reviewer WtGp for more experimental details. \n\nWe hope these analysis and new experiments could address the reviewer’s concern and strengthen our paper.\n\n---\n\n**For Question 1: The subfigure (b) and (c) in Figure 4 do not look much different.**\n\nThank you very much for your constructive suggestions! As you mentioned, although our model can achieve better multi-modal alignment result than CMCM method, the difference between subfigure (b) and (c) is not obvious. We have replaced the color of both two modalities mapped codes **from purple to green**, please note that we have added these visualizations to the newly updated global PDF.\n\n---\n\n**For Question 2. \"sg\" in Equation (3) and (8) is not explained.**\n\nThank you for pointing this out. We apologize for omitting such important information. Sg in Eq (3) and (8) is the abbreviation of **Stop Gradient** that blocks gradients from flowing into its argument. We will revise this in our paper. \n\n---\n\nWe thank the reviewer again for your positive feedback. We would be very grateful if the reviewer could take time to read our responses and let us know your thoughts.\n"}, {"review_id": "1SZ5XTVGSk", "reviewer": "Reviewer_WtGp", "summary": "In this paper the authors propose a novel task called Cross Modal Generalization (CMG), where they aim to use unlabelled internet scale paired multimodal data during pre training and then use it for zero shot generalization to other modalities in downstream tasks. The authors claim that disentagling modality specific features is crucial for allowing shared representations to generalize across modalities for which they propose a Dual Cross-modal Information Disentangling (DCID) module which incorporates 2 different aspects ie MI minimization between modal-agnostic semantic features and modal-specific features in each modality (CLUB) and MI maximization between modal agnostic semantic features across different modalities (Cross-CPC). They further propose Multi-modal Exponential Moving Average (MM-EMA)  to achieve fine-grained cross-modal alignment in unconstrained scenarios.\nThe authors demonstrate the effectiveness of their approach on several downstream tasks and show significant improvements over the prior state of the art on standard benchmarks. They also present an elaborate ablation study to demonstrate the contributions of each of the components of their proposed approach and show how their proposed modules improve upon an otherwise weak baseline. They present their results on AVE, AVVP and AVS-S4 datasets and show the effectiveness of their approach on Audio-video, audio-text and audio-video-text modalities.", "questions": "- The work seems to be able to easily extended to text-visual modality as well for which there are several benchmarks and baselines available to compare against and would have made for a good comparison but that extension was never done. Curious as to why that combination wasnt considered?\n- \n", "limitations": "- No major negative impacts or limitations", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper is very well written and addresses several major challenges in the field of learning unified multimodal representations. It's claim of the necessity to disentangle modality specific features is well founded in literature and the method proposed to achieve it solid and sound.\n- Thorough ablation studies are performed to highlight the use of each of the proposed method components\n- Presentation of the different components and the overall setup is easy to follow and understand\n- I like the formulation of the Dual Cross-modal Information Disentangling module. It takes cues from MI maximization theory of disentaglement and is able to formulate it in a simple and succint manner while being able to carefully optimize and train using a widely diverse training objective. ", "weaknesses": "- The paper doesnt include comparisons to a vast body of literature of Masked Autoencoders which have recently gotten very popular for learning joint vector embeddings. For example: Contrastive Audio-Visual Masked Autoencoder by Gong et al, MAViL: Masked Audio-Video Learners by Huang et al, Masked Autoencoders Are Scalable Vision Learners by He et al, Masked Autoencoders that Listen by Huang et al etc. Similarly the CLIP style approaches and their extensions to further modalities: AVE-CLIP, AV-CLIP, Audio-CLIP etc have not been discussed and compared against. Both these lines of methods have proven to be very effective in literature and should make for good comparison cases here (or discussed as to why they arent relevant comparisons for this work). Another recent work which expands to even more modalities is ImageBind: One Embedding Space To Bind Them All by Girdhar et al and should be compared against\n- The suite of downstream tasks to be evaluated against need to be more diverse to truly evaluate if the semantic modality agnostic features have truly been captured by the representations. \n- The work seems to be able to easily extended to text-visual modality as well for which there are several benchmarks and baselines available to compare against and would have made for a good comparison but that extension was never done. Curious as to why that combination wasnt considered?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer WtGp:\n\nThank you very much for your insightful comments and your acknowledgement of our proposed Dual Cross-modal Information Disentangling module. Let us illustrate your questions point by point. \n\n---\n\n**Weakness 1. For comparison with MAE-based and CLIP-based method**\n\nWe much appreciate the reviewer for pointing out these important methods we missed. As you mentioned, the **MAE-based** and **CLIP-based** audio-visual methods can effectively align two modalities sharing the same semantics within a joint embedding space. These implicit unified representation methods can greatly advance the performance of cross-modal retrieval and multi-modal fusion tasks. However, these methods can not explicitly map different modalities together in the latent space, which may not obtain optimal performance in our cross-modal generalization downstream tasks. For example, we implement AVE-CLIP pre-trained methods on VGGSound-AVEL 40k and compare its performance with our methods on cross-modal event classification task (AVE dataset):\n\n| Model               | V->A | A->V |\n|------------------|-------|------|\n| AVE-CLIP         | 24.8  | 27.8  |\n| CMCM             | 32.7  | 36.8  |\n| Our Full Model | 47.7  | 52.3  |\n\nFrom the results we can observe that such contrastive learning between audio-visual modalities is not sufficient for perfect multi-modal unified representation.\n\nIn recent years, explicit unified representation methods such as CODIS (CVPR 22'), TURN (NeurIPS 22'), and CMCM (ACL 22') have been introduced. These methods utilize discrete latent variables as bridges to explicitly merge different modalities into quantized codes, making them more suitable for this new task. Furthermore, TURN (NeurIPS 22') also contains a self-cross-reconstruction mechanism similar to MAE, we run their code on our task, and the results in Table 1 show that our methods can greatly suppress them. We much appreciate the reviewer for pointing this out, we have added these discussions of these methods and ImageBind in our paper. \n\n---\n\n**Weakness 2. The suite of downstream tasks to be evaluated against need to be more diverse to truly evaluate if the semantic modality agnostic features have truly been captured by the representations.**\n\nHaving said that, it is a very constructive suggestion. We totally agree with the reviewer for this opinion. In our paper, we conduct following experiments to demonstrate it:\n1) We incorporate our DCID module with other state-of-the-art models, and the results in Table 1 show that with the equipment of DCID, all these model can obtain substantial improvements in downstream tasks, which can illustrate that our DCID method can effectively extract modality-agnostic features and facilitate all other compared models. \n\n2) Furthermore, as shown in Figure 5, the visualization results illustrate that although there are huge domain gaps between text and audio modalities, the model trained based on text (or audio), can still accurately localize the right visual region when the query been replaced as audio (or text) during inference. These experiments show that our pre-train model can **effectively disentangle semantic modality agnostic features** from text and audio modalities. \n\nThank you very much for your valuable suggestions. We will design more diverse experiments to demonstrate that our model can truly capture the semantic modality agnostic features.\n\n---\n\n**Weakness 3 & Question. More experiments about text-visual cross modal generalization tasks**\n\nWe appreciate the reviewer for such constructive comment. Yes, our work can be extended to text-visual modality. Here we select **retrieval task** to demonstrate that our method can also be applied to **visual-text generalization** as well. We use audio as an inter-medium to measure the generalization ability of our model across these two modalities and implement an X-to-audio retrieval task. To be detailed, in the first stage, we train **visual-text unified representation** learning using VGGSound24K dataset, and then in the second stage, during downstream training, we let the model learn text(video)-audio retrieval, finally during inference, we directly test the generalization ability of the model on video(text)-audio retrieval. To simplify, we choose cross-attention network as our downstream retrieval model.\n We test the retrieval performance on part (8k) of AudioSet dataset, the results are as follows:\n\n\nv->t (v2a retrieval for training, t2a retrieval for test)\n\n| Model         | R@5   | R@10  |\n|---------------|-------|-------|\n| Baseline      | 0.47  | 1.03  |\n| Full Model    | 10.3  | 21.9  |\n\n\nt->v (t2a retrieval for training, v2a retrieval for test)\n\n| Model         | R@5   | R@10  |\n|---------------|-------|-------|\n| Baseline      | 0.62  | 0.85  |\n| Full Model    | 8.47  | 16.7  |\n\nThe results show that compared with baseline model, our model can effectively achieve **zero-shot cross-modal audio retrieval** ability. \nDue to the rebuttal time constraints, here we only compare our full model with baseline model, however, we will further perform visual-text benchmarks on other comparison methods and add these experiments to our paper. We really appreciate your comments, which make our paper more solid.\n\n---\n\nWe thank the reviewer again for raising these insightful suggestions to make our paper better. If you have any further questions or comments, please let us know, we are glad to respond."}, {"review_id": "W2oLY1qpPm", "reviewer": "Reviewer_y1JE", "summary": "The paper proposes a model to learn a unified discrete representation from paired multimodal data during pre-training. Then in downstream tasks, the model can achieve zero-shot generalization ability in other modalities when only one modality is labeled. The two key contributions are the Dual Cross-modal Information Disentangling (DCID) module and the Multi-Modal Exponential Moving Average (MM-EMA). These methods facilitate bidirectional  supervision between modalities and align semantically equivalent information in a shared discrete latent space, enabling fine-grained unified representation of multimodal sequences. Extensive experiments on various downstream tasks show the effectiveness of their method.", "questions": "1. Sg is not defined in eq3\n2. How to select the negative set Zb is not clear. Is the size N-1? Then where does the randomness come from? In time steps?\n3. The text prompts (Sec B.3) seems to be manually designed and are very specific to the event and datasets. Are there ways to easily generate prompts across different events in different datasets?\n4. It would be better to also show some qualitative video segmentation examples of the baseline (compared) methods.", "limitations": "The authors have addressed the limitations of their work", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors performed extensive experiments to demonstrate the effectiveness of their method.\n2. Their method can be applied to more than two modalities and can achieve zero-shot generalization ability in other modalities when only one modality is labeled in downstream tasks.\n", "weaknesses": "1. There are still some parts needs to be clarified, see Questions part.\n2. It seems the new commitment loss (eq 8) is not in the ablation study. How does it compare with the original commitment loss in eq(3)?\n3. It would be better to include same modality upper bound in downstream tasks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer y1JE:\n\nThank you so much for taking time to read our paper and providing valuable comments. We are glad to respond your questions point-by-point. \n\n\n**Weakness. The comparison with original commitment loss**\n\nThank you for pointing out our missing ablations, we conduct a series of experiments with the model pretrained on VGGSound 40K to illustrate these:\n\nCross-modal event classification (AVE dataset):\n\n| Model         | V->A  | A->V  |\n|---------------|-------|-------|\n| Full Model    | 47.7  | 52.3  |\n| Eq(8)->Eq(3)  | 45.1  | 48.2  |\n\nCross-modal event localization (AVVP dataset):\n\n| Model         | V->A  | A->V  |\n|---------------|-------|-------|\n| Full Model    | 64.0  | 65.6  |\n| Eq(8)->Eq(3)  | 59.6  | 62.3  |\n\nThe results show that compared with original Eq (3), our new proposed commitment loss can achieve better multi-modal unified representation. We hope this additional experiment could strengthen our contribution.\n\n\n\n**Question 1. Sg is not defined in eq3**\n\nSorry about this mistake. Sg in Eq (3) and (8) is the abbreviation of **Stop Gradient** that blocks gradients from flowing into its argument. We will revise this in our paper. \n\n**Question 2. How to select the negative set Zb is not clear**\n\nYes, as we mentioned in Line 186, the size of negative samples is **N-1**, and these negative samples are randomly selected from other sequences within the same batch. Thank you for noticing this detail, we will add more clarification in our paper. \n\n\n**Question 3. About the design of the text prompts**\n\nYes, it is a good question. In our paper, for simplified, we manually design several different prompt templates for each event category in VGGSound-AVEL and AVS-S4 datasets. Under your suggestion, we try to use ChatGPT to automatically generate a variety of corresponding prompts according to the characteristics of each event, and the quality is satisfactory.\n\n**Question 4. It would be better to also show some qualitative video segmentation examples of the baseline methods.**\n\nThank you for pointing this out, we have added the qualitative video segmentation results of the baseline model. Please refer to the global PDF we just uploaded. We will also update the Figure 5 in our paper.\n\n\nWe thank the reviewer again for your positive feedback. If you have any further questions or comments, please do not hesitate to ask, we are glad to respond.\n"}, {"review_id": "u0ymZBDil1", "reviewer": "Reviewer_bPma", "summary": "This paper proposes to learn a unified discrete representation from paired multimodal data during pre-training. During the downstream task, it can achieve zero-shot generalization ability in other modalities when only one modal is labeled. Specifically, it develops a Dual Cross-modal Information Disentangling (DCID) module and a Multi-Modal Exponential Moving Average (MM-EMA) to achieve the goal. Experiments are conducted on four tasks.", "questions": "see above", "limitations": "See above", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to follow.\n2. The motivation is reasonable.\n3. The proposed method is technically sound.\n4. Experiments are sufficient.", "weaknesses": "1. The compared methods are out-of-date. The authors should provide more latest works for comparison.\n2. Missing some related work of unified representation learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer bPma:\n\nThank you very much for your acknowledge of our paper. We are glad to answer your questions point by point.\n\n\n**Weakness 1. The compared methods are out-of-date. The authors should provide more latest works for comparison**\n\nMulti-modal unified representation is a challenging task which has not been studied much. The methods we compared are all the latest studies in explicit unified representation, CODIS (CVPR 22'), CMCM (ACL 22'), TURN (NeurIPS 22'). To achieve cross-modal generalization, discrete variables that can explicitly aggregate different modalities together are necessary. Thus in the experiments, we only compare our model with these latest explicit unified representation methods. Meanwhile, we will add more discussion of the latest implicit representation methods in our paper. \n\n\n**Weakness 2. Missing some related work of unified representation learning**\n\nThank you very much for pointing this out. We will discuss some recent excellent implicit unified representation models in the related works. For example, as the Reviewer WtGp suggested, the MAE-based (MAViL, CAV-MAE) and CLIP-based (AVE-CLIP, AV-CLIP) multi-modal models, ImageBind, Zorro and XKD. All these works can align different modalities closer in latent embedding space. We will add the discussions of these works in introduction and relate work parts. \n\nWe thank the reviewer again for your positive feedback. If you have any further questions or comments, please let us know, we are glad to respond."}, {"review_id": "rS5tpNEhBW", "reviewer": "Reviewer_w8Lb", "summary": "This paper is focused on a novel and promising setting, which is the zero-shot generalization ability in other modalities that lacks annotations. It is meaningful in real-world scenarios even though it still requires paired multimodal pretraining. In addition, I agree with the authors that the ability of the network for unseen modalities is significant to the community, and the paradigm without fine-tuning encoders in the downstream tasks is also worth more attention.", "questions": "- According to Figure 3, it seems that text plays an important role in pretraining. It makes the contributions weaker, for which the improvements of this method can be attributed to the paired data of text, audio, and image. In other means, the dependency on paired data still limits the performance although the authors aim to reinforce the ability of unseen modality generalization.", "limitations": "The limitation of this paper is mainly about current applications on audio-visual tasks.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Useful insights and meaningful technical direction\n- Promising experimental results and detailed evaluation\n- Detailed and sufficient analyses ", "weaknesses": "- The organization of the content. The manuscript lacks more promising proposals in the introduction. Obviously, the paragraphs in L58-75 can be greatly reduced.\n- The visualization in Figure 4 contains very few instances. It's suggested to present a more comprehensive comparison.\n- This paper is more focused on audio-visual understanding and introduces text to these tasks. The core contribution of generalizing to unseen modalities is not sufficiently evaluated.\n- The ability of the network for zero-shot learning on seen modality is not evaluated. With strong pretraining based on paired data, I think that it has a strong ability for zero-shot learning. I strongly encourage the authors to add two significant experiments: i. Zero-shot classification on ImageNet ii. Zero-shot classification on AudioSet. It will significantly improve the paper's quality and impact, and I will consider improving my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer w8Lb:\n\nThank you very much for taking time to read our paper and giving such insightful comments. Please see the following for your point-by-point response. \n\n---\n\n**Weakness 1. The organization of the introduction**\n\nWe greatly appreciate your useful comments to make our paper better. We have reorganize the description of the introduction, especially the content you mentioned in Line 58-75. \n\n---\n\n**Weakness 2. The visualization in Figure 4 contains very few instances.**\n\nSorry about the confusion, actually, we provide more visualization results of our model and baseline model in different training epochs, as shown in Figure 2 in appendix material. The results can more intuitively illustrate the process of our model gradually aligning different modalities in the latent space.\n\n---\n\n**Weakness 3. This paper is more focused on audio-visual understanding and introduces text to these tasks.**\n\nSorry about the confusion, in this paper, we do not focus on audio-visual understanding task, instead, we use various audio-visual tasks to demonstrate that our pre-trained model can directly generalize from seen modality to unseen modality. For example, in **cross-modal event localization** task, after obtaining pre-trained unified representation model, we use it as Encoder to train the event classification task on seen modality (e.g., audio), then during inference, we directly test the performance of event classification ability on unseen modality (e.g., video). Also we conduct downstream experiments on AVS-S4 dataset to demonstrate that our model can achieve zero-shot cross-modal (text2audio and audio2text) segmentation ability.\nWe think such these cross-modal generalization tasks can demonstrate that our method can successfully achieve multi-modal unified representation. \n\n---\n\n**Weakness 4. The ability of the network for zero-shot learning on seen modality is not evaluated.**\n\nIn this paper, the purpose of our pre-training is to align different modalities closer in the hidden embedding space, by using a set of intermediate discretized codes. Thus, after pre-training, different modalities but share the same semantics will aggregate to the same discrete latent codes. However, these discrete variables do not contain any knowledge of downstream tasks. To achieve zero-shot cross-modality knowledge transfer, the model has to train these discrete vectors using seen modality during downstream training. In conclusion, our model can not achieve zero-shot learning on seen modality. \n\nHowever, our pre-train model can not only align different modalities, but also can converge features with similar meaning in the same modality into the same discrete variable. Thus, our model can achieve promising results in few-shot ability on seen modality in downstream tasks.\n\nWe conduct a simple few-shot experiments on UCF-101 dataset (video) and part of AudioSet (audio), the model is pre-trained on VGGSound-AVEL 40K.\n\n**Video (UCF-101 dataset):**\n\n| Model         |1-shot |2-shot |4-shot |8-shot |16-shot|32-shot|\n|---------------|-------|-------|-------|-------|-------|-------|\n| baseline      | 1.20  | 1.18  | 1.23  | 1.21  | 1.23  | 1.22  |\n| w/o CrossCPC  | 18.6  | 23.4  | 29.3  | 33.3  | 36.2  | 37.7  |\n| Full model    | 26.1  | 33.2  | 39.0  | 43.2  | 45.9  | 47.3  |\n\n**Audio (AudioSet dataset):**\n\n| Model         |1-shot |2-shot |4-shot |8-shot |16-shot|32-shot|\n|---------------|-------|-------|-------|-------|-------|-------|\n| baseline      | 1.17  | 1.23  | 1.21  | 1.23  | 1.24  | 1.17  |\n| w/o CrossCPC  | 23.6  | 31.2  | 38.0  | 42.8  | 46.5  | 49.2  |\n| Full model    | 32.4  | 44.5  | 51.8  | 56.9  | 58.0  | 60.6  |\n\n\nFrom the results we can see that our method can outperform the baseline by a large margin under all setting, which can further illustrate that our model can aggregate the similar semantic features, whether they come from the same modality or different modalities. We thank the reviewer again for pointing this out, which can further improve the quality of our work. We hope the additional experiments will address your concern. We will add these few-shot experiments and analysis in the revised paper.\n\n---\n\n**Question 1. According to Figure 3, it seems that text plays an important role in pretraining**\n\nThank you very much for your careful observation, let we explain your question in detail. As you can see in Figure 3, the introduction of text will indeed improve the unified representation of audio-visual. This is because the introduction of a third-party modality will serve as a bridge to help the alignment of the other two modalities, which has also been mentioned in other papers, such as UniVAL [1]. When introducing the third mode, our proposed Cross-CPC and MM-EMA will let any two modes interact with each other to shorten the distance, which can facilitate better alignment among these modalities. The performance gain brought by the text does not mean that our contribution is weakened.  \n\n[1] UnIVAL: Unified Model for Image, Video, Audio and Language Tasks\n\n---\n\n**Question 2. In other means, the dependency on paired data still limits the performance although the authors aim to reinforce the ability of unseen modality generalization.**\n\nSorry about the confusion. In addition, we conduct a series of experiments on unpaired downstream datasets: transferring the event localization ability of the model from **seen modality in AVE dataset** to **unseen modality in AVVP dataset** (Line 242-245), as shown in the right part in Table 3. The results prove that even though the seen and unseen modalities come from different sources, our method can still guarantee a strong zero-shot cross-modal generalization ability with unsupervised pretraining. We will add more analysis about these in our paper. \n\n---\n\nWe thank the reviewer again for your positive feedback. If you have any further questions or comments, please let us know, we are glad to respond."}]

Author rebuttal:
Dear reviewers,

We much appreciate for your acknowledgement of our work and helpful, insightful comments. Following the reviewers' suggestions, we have made a major revision of the paper and conducted a series of new experiments to address the reviewers' concerns. We have also updated two figures in the single-page PDF file as suggested by two reviewers. In the following, under each reviewer's comment, we address the concerns of the reviewers point by point.

Meta-review: This paper introduces a novel idea supported by strong empirical results. The reviewers appreciate the valuable contribution of this study. In preparation for the camera-ready version, we kindly urge the authors to incorporate the feedback provided by the reviewers, which will undoubtedly further refine the quality and impact of the work.
Decision: Accept (poster)