SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yan Xia $^{1*}$
Hai Huang $^{1*}$
Jieming Zhu
Zhou Zhao $^{1,2\dagger}$
<sup>1</sup>Zhejiang University  <sup>2</sup>Shanghai Artificial Intelligence Laboratory  <sup>3</sup>Huawei Noah's Ark Lab
xiayan.zju@gmail.com haihuangcode@outlook.com
jiemingzhu@ieee.org zhouzhao@zju.edu.cn
This paper introduces a novel task called Cross Modal Generalization (CMG), which addresses the challenge of learning a unified discrete representation from paired multimodal data during pre-training. Then in downstream tasks, the model can achieve zero-shot generalization ability in other modalities when only one modal is labeled. Existing approaches in multimodal representation learning focus more on coarse-grained alignment or rely on the assumption that information from different modalities is completely aligned, which is impractical in real-world scenarios. To overcome this limitation, we propose Uni-Code, which contains two key contributions: the Dual Cross-modal Information Disentangling (DCID) module and the Multi-Modal Exponential Moving Average (MM-EMA). These methods facilitate bidirectional supervision between modalities and align semantically equivalent information in a shared discrete latent space, enabling fine-grained unified representation of multimodal sequences. During pre-training, we investigate various modality combinations, including audio-visual, audio-text, and the tri-modal combination of audio-visual-text. Extensive experiments on various downstream tasks, i.e., cross-modal event classification, localization, cross-modal retrieval, query-based video segmentation, and cross-dataset event localization, demonstrate the effectiveness of our proposed methods. The code is available at https://github.com/haihuangcode/CMG.
Although recent years have witnessed significant achievements in many multimodal areas, e.g., multi-modality question-answering [1, 2, 3], query-based segmentation [4, 5], audio-visual event localization [6, 7], labeling these tasks always consumes extensive human resources. Moreover, the labeling cost for different modalities can vary significantly [8], leading to scenarios where only a subset of modalities is labeled, while others remain scarce [9]. For example, text-based visual segmentation is common while the label data for audio-based visual segmentation is rare. Consequently, models trained on such limited data are restricted to specific scenarios, hindering their broader applicability. Fortunately, unannotated paired multimodal data is readily accessible, such as the abundant image-caption [10] and audio-visual pairs [11] available on the Internet. Hence, in this paper, we develop a new task, Cross-Modal Generalization (CMG), for investigating how to learn unified discrete representations from these unlabeled multi-modal data pairs. Our objective is to transfer the knowledge acquired from labeled modalities to other unseen modalities in downstream tasks, enabling models to generalize effectively.
† Corresponding Author
Figure 1: The overview of our proposed CMG task, different colors in the left and middle parts mean different semantics. The left part is the illustration of bad multi-modal unified representation, where features from various modalities sharing the same semantic meaning are mapped into disparate latent codes, while the good unified representation (middle part) is totally different. The right part shows that in downstream tasks, the model will be directly transferred to unseen modalities.
Humans possess a natural ability to associate different modalities with similar semantics, facilitating cross-modal knowledge transfer based on known modalities. Inspired by this, numerous studies have explored the integration of diverse multimodal information into a unified semantic space, which can be categorized into two types: implicit representations [12, 13, 14] and explicit representations [15, 16, 17]. For implicit representations, several methods utilize a modality-agnostic encoder [12, 13] to represent different modalities, or employ contrastive learning [18, 19] to bring different modalities closer in high-dimensional semantic space. In contrast, explicit representations aim to use a unified codebook [17] or prototype [16] to represent different modalities, serving as a bridge to facilitate robust alignment across modalities. Furthermore, the use of discrete space allows for the aggregation of similar input features in a high-dimensional space, enabling complex feature representation with a reduced number of latent codes. However, these studies predominantly compress sequential features within modalities into single vectors before quantization [16, 20] or rely on the assumption that information from different modalities is completely aligned [17, 21]. As a result, these methods are typically limited to simple tasks such as cross-modal retrieval and demonstrate poor performance in more complex scenarios.
As shown in Fig 1, the visual information in unconstrained videos comprises events of cat meowing and dog barking, while the audio information comprises events of woman speaking and dog barking. Directly applying previous methods [13, 17, 16, 20] would lead to inaccurate mapping of multimodal information lacking shared semantics. To address this limitation, our paper primarily focuses on the implementation of a fine-grained, unified representation for multimodal sequences. We address this novel problem from two key aspects: 1) extracting information with identical semantics across different modalities while mitigating the influence of modality-specific details, and 2) representing these diverse modalities with shared semantics using a unified codebook. Most of the previous works primarily focus on the second aspect while neglecting the importance of the first aspect, which we argue is crucial for achieving cross-modal generalization.
Our work builds upon the advantages of explicit representations and proposes two modules to address the aforementioned challenges. To address the first aspect, we propose Dual Cross-modal Information Disentangling (DCID) module, which combines Contrastive Log-ratio Upper Bound (CLUB) [22] with our proposed Cross-modal Contrastive Predictive Coding [23] (Cross-CPC). Concretely, we employ CLUB to optimize the upper bound of mutual information, efficiently distinguishing the modal-agnostic semantic information (conveying primary content or events) from the modal-specific information (additional details unrelated to semantics, such as light and perspective in vision, or timbre and pitch in audio, etc.) across various modalities. However, without effective guidance, it is challenging for the model to identify which aspects of semantic information are useful [24]. Motivated by the complementary guidance nature of multi-modal information [25], we propose a novel Cross-CPC method, to predict the future information in other modalities based on the known sequence information of the current modality, which can effectively maximize the fine-grained mutual information between different modalities. To address the second aspects, we propose Multi-Modal Exponential Moving Average (MM-EMA), which employs a teacher-student mechanism to facilitate
cross-modal guidance during pretraining. This mechanism encourages the aggregation of quantized vectors encoded from different modalities with shared semantics into the same latent code space. Otherwise, distinct modalities would aggregate in separate regions of the discrete space, rather than being mapped together [20, 17]. To summarize, our main contributions are threefold:
- We introduce a new task, CMG, which enhances the applicability of existing models in the realm of multimodal learning, addressing the challenges arising from modality scarcity and the substantial cost of annotating certain modalities.
- We propose a novel framework named Uni-Code, which can effectively extract shared semantic information from paired multi-modal data and project them into a common quantized latent space in a fine-grained level.
- We investigate the unified representation of various modalities, including audio-visual, audio-text, visual-text, and even the challenging tri-modal combination of audio-visual-text. Extensive experiments on various downstream tasks, e.g., multimodal event classification, localization, cross modal retrieval and video segmentation, demonstrate the effectiveness of our proposed methods.
Implicit Multi-Modal Unified Representation. The past few years have witnessed remarkable achievements in the implicit multi-modal unified representation, which aims to align diverse modalities within a shared latent space [26, 27, 14] or try to learn a modality-agnostic encoder for extracting information across various modal [28, 29]. These methods investigate the unified representation for various modality combinations, i.e., Speech-Text [26, 30], Video-Audio [31, 32, 27, 33], Vision-Text [18, 14, 29]. To achieve these, different kinds of methods have been proposed. Pedersoli et al. [31] and Sarkar et al. [27] introduce a cross-modal knowledge distillation to transfer knowledge across modalities. CLIP-based methods [18, 14] use contrastive loss to learn image-text consistency from a large paired dataset, and have gained incredible zero-shot ability in various downstream tasks.
Explicit Multi-Modal Unified Representation. Recently, some works investigate how to achieve multimodal explicit unified representation by utilizing a universal codebook [15, 17, 20] or prototype [16] to explicitly express multimodal content. Duan et al. [16] apply Optimal Transport to map the feature vectors extracted from different modalities to the prototypes. Zhao et al. [20] use self-cross-reconstruction to enhance the mutual information between different modals. However, these coarse-grained alignment methods [16, 20] are only suitable for simple tasks such as retrieval, and are unable to accomplish fine-grained comprehension in downstream tasks. In recent years, [34, 21] align the text and speech temporally with a unified discrete space. Liu et al. [17] use the same scheme to align short videos and speech/text. However, a key assumption underlying their success is that the modalities they chose have a one-to-one alignment (i.e. text-to-speech), while in more general case, it is hard to guarantee the semantic information in the two paired modalities is completely consistent (i.e. unconstrained video and audio). In this paper, we mainly investigate how to map a paired multimodal sequence into a common discrete semantic space, where the information in the multimodal sequence is unconstrained and may not be perfectly aligned.
Mutual Information Estimation. Mutual information (MI) estimation aims to measure the dependence between two random variables. Recent works primarily concentrate on how to combine MI estimation with deep neural networks [35, 22, 23], including MI maximization and minimization. MI maximization aims to learn representations that capture meaningful and useful information about the input data, leading to improved performance in various downstream tasks [36, 37, 38]. In order to maximize mutual information in a sequence, Contrastive Predictive Coding (CPC) [23] employs an autoregressive model and contrastive estimation to capture long-term relations while maintaining local features within a sequence. MI minimization tries to reduce the dependency between two random variables while preserving the relevant information, has been successfully applied in disentangled representation learning [39, 24]. Belghaz et al. [35] propose a Mutual Information Neural Estimator (MINE), which builds a neural network to estimate mutual information based on dual representations of the KL-divergence [40, 41]. Chen et al. [22] introduce Contrastive Log-ratio Upper Bound (CLUB), combining MI estimation with contrastive learning to approximate the MI upper bound. Their work is more suitable for MI minimization.
Figure 2: The overview of our proposed Uni-Code framework, we use audio-visual as an example. (a) The main pipeline of our model. (b) The process of Multi-modal Vector Quantization, which contains Multi-modal EMA (MM-EMA) and a new commitment loss, notice that the abundant discrete codes will be reset. (c) The architecture of our proposed Cross-CPC.
3 Cross Modal Generalization Task
Given a set of paired multi-modal data  $\mathbb{X} = \{(\mathbf{x}_i^A, \mathbf{x}_i^B, \mathbf{x}_i^C, \ldots)\}_{i=1}^N$  of size  $N$ , where A, B, C, and so on represent different modalities, the Cross Modal Generalization (CMG) task aims to map these various modalities into a unified discrete space during the pre-training phase, enabling discrete latent codes to be shared among different modalities with the same semantic. Subsequently, in downstream tasks, when only one modality (e.g., mode A) has annotated information, the model can transfer knowledge learned from A mode to other modalities (e.g., mode B and C) based on the shared discrete space obtained during pre-training to achieve zero-shot generalization ability.
4 Unified Representation Learning
Different from previous works which simply extract information from paired modalities and then direct maps, we argue that the success of unified representation lies in the extraction of modality-agnostic semantic features. Thus in this paper, we achieve this from two perspectives: first, we introduce a DCID module, designed to extract fine-grained semantic information and separate it from the corresponding modality-specific information within each modality. Second, we compress the extracted semantic features into discrete variables through VQ-VAE, ensuring that the compressed discrete variables can still contain the original semantic information through reconstruction loss. For simplicity, we take two modalities as an example to illustrate the process. Fig 2 gives an overview illustration of our network.
4.1 Baseline
Given two paired modals,  $\{(\mathbf{x}_i^a,\mathbf{x}_i^b)\}_{i = 1}^N$ , we first utilize two semantic encoders  $\Phi^a$  and  $\Phi^b$  to extract modal-agnostic features  $\mathbf{z}_i^a$ ,  $\mathbf{z}_i^b\in R^{T\times D}$ , and use two modal-specific encoders  $\Psi^a$  and  $\Psi^b$  to extract remain features  $\overline{\mathbf{z}}_i^a$ ,  $\overline{\mathbf{z}}_i^b$  from modality A and B, respectively:
$$
\mathbf {z} _ {i} ^ {a} = \Phi^ {a} \left(\mathbf {x} _ {i} ^ {a}\right) \quad \bar {\mathbf {z}} _ {i} ^ {a} = \Psi^ {a} \left(\mathbf {x} _ {i} ^ {a}\right) \quad \mathbf {z} _ {i} ^ {b} = \Phi^ {b} \left(\mathbf {x} _ {i} ^ {b}\right) \quad \bar {\mathbf {z}} _ {i} ^ {b} = \Psi^ {b} \left(\mathbf {x} _ {i} ^ {b}\right), \tag {1}
$$
where  $T$ ,  $D$  represent time and hidden dimension, respectively. The dimension of  $\bar{\mathbf{z}}_i^a$ ,  $\bar{\mathbf{z}}_i^b$  various for different modalities. Then we apply vector quantized operation to map the semantic features  $\mathbf{z}_{i,t}^{a}$ ,  $\mathbf{z}_{i,t}^{b}$  to discrete latent codes in fine-grained level,  $t\in [0,T]$ :
$$
\hat {\mathbf {z}} _ {i, t} ^ {m} = V Q \left(\Phi^ {m} \left(\mathbf {x} _ {i, t} ^ {m}\right)\right) = V Q \left(\mathbf {z} _ {i, t} ^ {m}\right) = e _ {l}, \quad \text {w h e r e} \quad l = \operatorname {a r g m i n} _ {j} \| \Phi (x) - e _ {j} \| _ {2}, \quad m \in \{a, b \}, \tag {2}
$$
the latent codebook  $\mathbf{e} \in R^{L \times D}$  is shared across modality A and B, where  $L$  is the size of the discrete latent space. Finally, we combine  $\hat{\mathbf{z}}_i^m$  with  $\overline{\mathbf{z}}_i^m$  together to reconstruct original features:
$$
L = \underbrace {\left\| \mathbf {x} _ {i} ^ {m} - D \left(\hat {\mathbf {z}} _ {i} ^ {m} ; \bar {\mathbf {z}} _ {i} ^ {m}\right) \right\| _ {2} ^ {2}} _ {\text {r e c o n s t r u c t i o n l o s s}} + \underbrace {\left\| \operatorname {s g} \left[ \phi^ {m} \left(\mathbf {x} _ {i} ^ {m}\right) \right] - \mathbf {e} \right\| _ {2} ^ {2}} _ {\text {V Q l o s s}} + \underbrace {\beta \left\| \phi^ {m} \left(\mathbf {x} _ {i} ^ {m}\right) - \operatorname {s g} [ \mathbf {e} ] \right\| _ {2} ^ {2}} _ {\text {c o m m i t m e n t l o s s}}, \quad m \in \{a, b \}. \tag {3}
$$
where  $\beta$  is 0.25 for all our experiments, sg is stop gradient. In this work, we use Exponential Moving Average (EMA) to replace VQ loss, since EMA is more robust. The reconstruction loss can guarantee that the compressed latent codes  $e_l$  still maintain the semantic information of different modalities. In ideal conditions,  $\mathbf{z}_i^a$  and  $\mathbf{z}_i^b$  encoded from different modalities with the same semantics should be mapped to the same discrete latent code. However, without effective supervision, the existence of modality gap will result in  $\mathbf{z}_i^a$  and  $\mathbf{z}_i^b$  converging to separate regions of the codebook [20, 17]. Thus in this paper, we introduce the following modules to alleviate these problems.
4.2 Dual Cross-modal Information Disentangling
We introduce our DCID module from two aspects: MI minimization between modal-agnostic semantic features and modal-specific features in each modality (CLUB), and MI maximization between modal-agnostic semantic features across different modalities (Cross-CPC).
CLUB-based MI Minimization: Compared to the methods that try to optimize the MI lower bound, such as InfoNCE [23] and MINE [35], CLUB [22] can effectively optimize the MI upper bound, demonstrating superior advantages in information disentanglement [24]. Given two variables  $\mathbf{x}$  and  $\mathbf{y}$ , the objective function of CLUB is defined as:
$$
I _ {\mathrm {v C L U B}} (\mathbf {x}; \mathbf {y}) := \mathbb {E} _ {p (\mathbf {x}, \mathbf {y})} [ \log q _ {\theta} (\mathbf {y} | \mathbf {x}) ] - \mathbb {E} _ {p (\mathbf {x})} \mathbb {E} _ {p (\mathbf {y})} [ \log q _ {\theta} (\mathbf {y} | \mathbf {x}) ], \tag {4}
$$
where  $q_{\theta}$  is the variational approximation of ground-truth posterior of  $\mathbf{y}$  given  $\mathbf{x}$  and can be parameterized by a network  $\theta$ . We use CLUB to optimize the MI upper bound between the semantic features  $\mathbf{z}_i^m$  and modal-specific features  $\bar{\mathbf{z}}_i^m$ ,  $m \in \{a, b\}$ , we modify  $I_{\mathrm{vCLUB}}$  into temporal version:
$$
\hat {I} _ {\mathrm {v C L U B}} = \frac {1}{N} \sum_ {i = 1} ^ {N} \left[ \frac {1}{T} \sum_ {t = 1} ^ {T} \log q _ {\theta} \left(\bar {\mathbf {z}} _ {i} ^ {m} \mid \mathbf {z} _ {i} ^ {m}\right) - \frac {1}{N} \frac {1}{T} \sum_ {j = 1} ^ {N} \sum_ {t = 1} ^ {T} \log q _ {\theta} \left(\bar {\mathbf {z}} _ {j} ^ {m} \mid \mathbf {z} _ {i} ^ {m}\right) \right], \quad m \in \{a, b \} \tag {5}
$$
The approximation network and the main networks are optimized alternatively during pre-training. Finally, we can reduce the correlation between semantic information and modal-specific information in each modality. Nevertheless, merely minimizing mutual information using CLUB presents a challenge for the model to identify relevant semantic features. Given that paired multimodal information can provide mutual guidance and serve as supervisory signals for each other, we devise the Cross-CPC approach to alleviate this issue.
MI Maximization with Cross-CPC: Contrastive Predictive Coding (CPC) [23] can maximize the mutual information among the adjacent items within a sequence by predicting the future samples with powerful autoregressive models, has been widely used in self-supervised learning. While for human, we can not only predict subsequent scenarios based on the current modality, but are also capable of associating with potential situations that may occur in other modalities. For instance, people can infer forthcoming auditory information based on a presently viewed video segment or read text, or envision a subsequent scene by perceiving an audio portion. Inspired by this, in this paper, we extend CPC to cross-modal constrictive prediction. Given the semantic features  $\mathbf{z}^a$ ,  $\mathbf{z}^b \in R^{T \times D}$ , a prediction of K steps and a random time moment  $t \in (0, T - K]$ , we first use two single layers unidirectional LSTM to summarize the information of all  $\mathbf{z}_{\leq t}^a$ ,  $\mathbf{z}_{\leq t}^b$  and can obtain two context representations as  $\mathbf{c}_t^m = \mathrm{LSTM}(\mathbf{z}_{\leq t}^m) \in R^D$ ,  $m \in \{a, b\}$ .
For modality A, we first select a set  $Z_{b}$  of N-1 random negative samples and one positive sample  $\mathbf{z}_{t+k}^{b}$  from modality B, then we use  $\mathbf{c}_{t}^{a}$  to predict k-th future step  $\mathbf{z}_{t+k}^{b}$  in modality B, and the InfoNCE loss for modality A can be optimized as:
$$
L _ {c p c} ^ {a 2 b} = - \frac {1}{K} \sum_ {k = 1} ^ {K} \log \left[ \frac {\exp \left(\mathbf {z} _ {t + k} ^ {b} W _ {k} ^ {a} \mathbf {c} _ {t} ^ {a}\right)}{\sum_ {\mathbf {z} _ {j} \in Z _ {b}} \exp \left(\mathbf {z} _ {j} ^ {b} W _ {k} ^ {a} \mathbf {c} _ {t} ^ {a}\right)} \right]; \quad L _ {c p c} ^ {b 2 a} = - \frac {1}{K} \sum_ {k = 1} ^ {K} \log \left[ \frac {\exp \left(\mathbf {z} _ {t + k} ^ {a} W _ {k} ^ {b} \mathbf {c} _ {t} ^ {b}\right)}{\sum_ {\mathbf {z} _ {j} \in Z _ {a}} \exp \left(\mathbf {z} _ {j} ^ {a} W _ {k} ^ {b} \mathbf {c} _ {t} ^ {b}\right)} \right], \tag {6}
$$
where the  $W_{k}^{a}$  is the linear projection matrix for different step k. The optimization  $L_{cpc}^{b2a}$  for modality B is vice versa. Based on these, features with the same semantics across different modalities can be mutually extracted through fine-grained cross-modal prediction. Despite it is difficult to align paired modalities perfectly in unrestricted scenarios, the model can still predict the possible information in the next step of the other modality at the fine-grained level through contrastive learning, due to the summarization of the historical information using an autoregressive model.
4.3 Multi-modal Exponential Moving Average
Previous works [20, 16, 17] struggle to achieve fine-grained cross-modal alignment in unconstrained scenarios, to tackle their limitations, we propose MM-EMA, which can allow different modalities to serve as teacher-student iteratively and update each other during the quantization process.
First, we use cross-attention [42] to extract related information from the opposite modality, taking mode A as an example:  $\mathbf{r}_i^b =$  cross-att  $(\mathbf{z}_i^a;\mathbf{z}_i^b;\mathbf{z}_i^b)$ , where  $\mathbf{z}_i^a$  is query,  $\mathbf{z}_i^b$  is key and value. The vector  $\mathbf{r}_i^b$  contains the semantic information derived from  $\mathbf{z}_i^b$ , exhibiting a strong correlation with  $\mathbf{z}_i^a$ , while also preserving the intrinsic attributes of modality B. Thus, it can serve as an intermediary facilitating the alignment of modality A with modality B during EMA procedure.
Given a code vector  $\mathbf{e}_i$ , we can obtain  $n_i^a$  semantic vectors of modality A  $\{\mathbf{z}_{i,j}^a\}_{j = 1}^{n_i^a}$  and  $n_i^b$  semantic vectors of modality B  $\{\mathbf{z}_{i,j}^b\}_{j = 1}^{n_i^b}$  that are quantized to  $\mathbf{e}_i$ :
$$
N _ {i} ^ {(t)} = \gamma N _ {i} ^ {(t - 1)} + (1 - \gamma) \left[ n _ {i} ^ {a (t)} + n _ {i} ^ {b (t)} \right] \quad \mathbf {e} _ {i} ^ {(t)} = \mathbf {o} _ {i} ^ {(t)} / N _ {i} ^ {(t)} \tag {7}
$$
$$
\mathbf {o} _ {i} ^ {(t)} = \gamma \mathbf {o} _ {i} ^ {(t - 1)} + (1 - \gamma) \Big [ \sum_ {j = 1} ^ {n _ {i} ^ {a (t)}} \frac {\mathbf {z} _ {i , j} ^ {a (t)} + \mathbf {r} _ {i , j} ^ {b (t)}}{2} + \sum_ {j = 1} ^ {n _ {i} ^ {b (t)}} \frac {\mathbf {z} _ {i , j} ^ {b (t)} + \mathbf {r} _ {i , j} ^ {a (t)}}{2} \Big ],
$$
where  $t$  is batch sequence in order,  $N_{i}$  and  $o_{i}$  are accumulated vector count and volume, respectively. Besides, we also modify the commitment loss in Eq 3, which can use the code vector  $\mathbf{e}_i^b$  (quantized from  $\mathbf{z}_i^b$ ) as a teacher, to guide the Encoder  $\phi^a$  towards not only approximating  $\mathbf{e}_i^a$ , but also converging to  $\mathbf{e}_i^b$  with a certain ratio (50% in our setting,  $\beta$  is the same as in Eq 3).
$$
L _ {\text {c o m m i t}} ^ {a} = \beta \| \phi^ {a} \left(\mathbf {x} _ {i} ^ {a}\right) - \operatorname {s g} \left[ \mathbf {e} _ {i} ^ {a} \right] \| _ {2} ^ {2} + \frac {\beta}{2} \| \phi^ {a} \left(\mathbf {x} _ {i} ^ {a}\right) - \operatorname {s g} \left[ \mathbf {e} _ {i} ^ {b} \right] \| _ {2} ^ {2} \tag {8}
$$