SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jie Xu<sup>1</sup> Shuo Chen<sup>2</sup> Yazhou Ren<sup>1</sup> Xiaoshuang Shi<sup>1</sup>
Heng Tao Shen $^{1}$  Gang Niu $^{2}$  Xiaofeng Zhu $^{1,*}$
<sup>1</sup>University of Electronic Science and Technology of China, China
$^{2}$ RIKEN Center for Advanced Intelligence Project, Japan
Recently, numerous studies have demonstrated the effectiveness of contrastive learning (CL), which learns feature representations by pulling in positive samples while pushing away negative samples. Many successes of CL lie in that there exists semantic consistency between data augmentations of the same instance. In multi-view scenarios, however, CL might cause representation degeneration when the collected multiple views inherently have inconsistent semantic information or their representations subsequently do not capture sufficient discriminative information. To address this issue, we propose a novel framework called SEM: Self-weighted Multi-view contrastive learning with reconstruction regularization. Specifically, SEM is a general framework where we propose to first measure the discrepancy between pairwise representations and then minimize the corresponding self-weighted contrastive loss, and thus making SEM adaptively strengthen the useful pairwise views and also weaken the unreliable pairwise views. Meanwhile, we impose a self-supervised reconstruction term to regularize the hidden features of encoders, to assist CL in accessing sufficient discriminative information of data. Experiments on public multi-view datasets verified that SEM can mitigate representation degeneration in existing CL methods and help them achieve significant performance improvements. Ablation studies also demonstrated the effectiveness of SEM with different options of weighting strategies and reconstruction terms.
Contrastive learning (CL) explicitly enlarges the feature representation similarity between semantic-relevant samples, and it is adept at capturing high-level semantics while discarding irrelevant information. This learning paradigm has facilitated many research and application fields, such as visual representation [1, 2], text understanding [3, 4], and cross-modal agreement [5, 6, 7]. Samples with consistent semantics are typically constructed as positive sample pairs for CL loss (e.g., InfoNCE [8]), which motivates multi-view learning scenarios [9, 10] where researchers focus on exploring common semantics among multi-view data. However, this kind of data usually is with heterogeneous views and thus cannot be directly processed by previous CL methods with two shared network branches.
To handle this situation, many multi-view contrastive learning (MCL) methods [11, 12, 13, 14] have been proposed, which treats multiple views as positive sample pairs and achieves important progresses in exploring multi-view common semantics (see Sec. 2 for details). Nevertheless, we find that CL might cause representation degeneration that the representations of high-quality views tend to degenerate. This may make the MCL methods perform worse than the optimal single view (see Sec. 3.1 and Sec. 4.1), and thus heavily limiting the usability of MCL in practical scenarios. Although
*Corresponding Author (seanzhuxf@gmail.com). Code link: https://github.com/SubmissionsIn/SEM.
several CL work [15, 16] proposed different CL losses aiming at increasing robustness to noise and made important advances on vision and graph data, our experiments discover that these CL losses are still fragile in multi-view scenarios as multi-view data are with more diversity than single-view data. Different from changing CL loss, recent MCL methods [14, 17] focused on changing model structures and successfully improved the effectiveness of clustering the learned representations. Nevertheless, representation degeneration still exists in many cases and it requires further solutions.
We find that there could be two reasons leading to representation degeneration in MCL. I) The quality difference among multiple views. The success of CL is based on the priori condition that the constructed positive sample pair has semantic consistency, which generally holds in previous CL applications [1, 5, 8]. Unfortunately, for multi-view learning, the collected views usually have quality difference and the semantic of positive sample pairs might be inconsistent due to view diversity. Consequently, CL causes the representation degeneration of high-quality views due to the existence of low-quality views. II) Losing discriminative information during data processing. Multi-view data typically involve heterogeneous data forms [9, 18], e.g., different dimensions, modalities, and sparsity. For achieving MCL, the model needs to transform heterogeneous multi-view data into the same form with different encoders. However, data transformation could lose discriminative information as this process has no supervised signals for maintaining information. As a result, CL might miss multiple views' common semantics and focus on semantic-irrelevant information due to inductive bias.
To this end, we propose SElf-weighted Multi-view contrastive learning with reconstruction regularization (SEM) as shown in Figure 1 that takes the  $m,n,o$  -th views in  $V$  views as an example (where  $\mathcal{W}^{m,n}$  denotes the pairwise weight,  $\mathcal{L}_{CL}^{m,n}$  is the contrastive loss, and  $\mathbf{Z}^m$  is the learned representations). Specifically, SEM minimizes self-weighted contrastive losses  $\mathcal{W}^{m,n}\mathcal{L}_{CL}^{m,n}$  and  $\mathcal{W}^{n,o}\mathcal{L}_{CL}^{n,o}$  after measuring the discrepancy between pairwise views' representations, i.e.,  $(\mathbf{Z}^m,\mathbf{Z}^n)$  and  $(\mathbf{Z}^n,\mathbf{Z}^o)$ , respectively. This makes SEM adaptively strengthen CL between the useful pairwise views and also weaken CL between the unreliable pairwise views. Meanwhile, SEM takes self-supervised reconstruction objectives as regularization terms  $(\mathcal{R}^m,\mathcal{R}^n$  ,and  $\mathcal{R}^o$  on the hidden features  $(\mathbf{H}^m,\mathbf{H}^n$  ,and  $\mathbf{H}^o$  ) of encoders for individual views, respectively. This reconstruction regularization assists CL in accessing sufficient discriminative information hidden in raw input data  $(\mathbf{X}^m$ $\mathbf{X}^n$  ,and  $\mathbf{X}^o$  ), which could be implemented by exist
Figure 1: The framework of SEM. It leverages different networks to extract information of different views and conducts the proposed self-weighted multi-view contrastive learning with reconstruction regularization.
ing information encoder-decoder models, e.g., AE [19], DAE [20], and MAE [21]. In SEM, the representations and pairwise weights are alternatively updated to mutually enhance one another.
In summary, our contributions are: I) We propose a novel general framework SEM that leverages self-weighting and information reconstruction to address representation degeneration in MCL. II) We provide three options with different advantages to implement the weighting strategy of SEM including class mutual information, JS divergence, and maximum mean discrepancy. III) Theoretical and experimental analysis verified the effectiveness of SEM. It helps many CL methods (e.g., InfoNCE [8], RINCE [15], and PSCL [16]) achieve significant performance improvements in multi-view scenarios.
Contrastive learning (CL) As a popular self-supervised learning paradigm, CL focuses on learning semantically informative representations for downstream tasks [22, 23, 24, 25]. The most widely used loss function is InfoNCE [8] which pulls in the representations between positive sample pairs while pushing away that between negative sample pairs. Some work have attempted to explain the reasons for the success of applying InfoNCE, e.g., from perspectives of mutual information [8, 26], task-dependent view [27], or deep metric learning [28, 29]. Furthermore, [30, 31] pointed out to conduct CL with reconstruction regularization to achieve robust representations for downstream tasks. RINCE [15] (a short name of Robust InfoNCE) is a variant of InfoNCE contrastive loss that considers noise in false positive sample pairs. The recent work [16] investigates CL without
(a) ACC vs. Views
Figure 2: (a) Clustering accuracy of individual views on Caltech dataset. (b) Contrastive loss and representation similarity between view 1 and view 4. (c) Clustering accuracy of view 1 and view 4 during contrastive learning.
(b) Loss and similarity vs. Epoch
(c) ACC vs. Epoch
conditional independence assumption on positive sample pairs and proposes a population spectral contrastive loss (we call it PSCL for short). Despite important progresses have been made, in this work, we discover that these CL losses are still fragile in multi-view scenarios where data qualities are hard to be guaranteed, and even the reconstruction regularized CL is not enough.
Multi-view contrastive learning (MCL) Different from many CL methods that usually generate two inputs by data augmentation [32], MCL aims to handle multi-view data widely exiting in real-world applications. Multi-view data often contain more than two views/modalities and they naturally form multiple inputs [33, 34, 35]. Since the semantic consistency among multiple views is not guaranteed, it is challenging to capture the useful information in multi-view data, while considering the side effects of harmful information. Therefore, MCL attracts increasing attention in recent years [36, 37, 38]. For example, CMC [11] empirically shows that MCL performed with more scene views obtains the better representations with semantic information. DCP [39] leverages the maximization of mutual information to conduct consistency learning across different views and aims to achieve a provable sufficient and minimal representation. MFLVC [14] observes the conflict between consistency and reconstruction objectives in encoder-decoder frameworks and proposes to learn multi-level features for multiple views. DSIMVC [17] establishes a theoretical framework to reduce the risk of clustering performance degradation from semantic inconsistent views. Although satisfactory results are achieved in many cases, the representation degeneration caused by CL is still not well considered and addressed. In this paper, we point out that the representation degeneration could seriously limit the application of CL in multi-view scenarios, and propose the discrepancy-based self-weighted MCL to address it.
Notations This paper leverages bold uppercase characters and bold lowercase characters to denote matrices and vectors, respectively. Operator  $\| \cdot \| _2$  denotes vector  $\ell_2$ -norm and operator  $\| \cdot \| _F$  is matrix  $F$ -norm.  $\{\mathbf{x}_i^v\in \mathbf{X}^v\}_{i = 1,2,\dots ,N}^{v = 1,2,\dots ,V}$  denotes the multi-view dataset with  $N$  samples in  $V$  views.
This section first illustrates the phenomenon of representation degeneration in multi-view contrastive learning. To address this issue, we then establish a general framework of SEM: Self-weighed Multiview contrastive learning with reconstruction regularization. To implement the SEM framework, we further provide different options of weighing strategy, contrastive loss, and reconstruction term.
3.1 Motivation: Representation Degeneration in Multi-View Contrastive Learning
Researchers proposed many contrastive learning approaches and also achieved plenty of progress in multi-view learning. However, multi-view contrastive learning might result in the representation degeneration of high-quality views (i.e., those views contain rich semantic information) due to the diversity of multi-view data. Specifically, we illustrate it in Figure 2 that takes a popular multi-view dataset Caltech [40] (6 views) as an example. We leverage unsupervised linear clustering accuracy obtained by K-Means [41] to evaluate the representation quality of containing class-level semantics.
Firstly, we leverage self-supervised autoencoders (the setting is shown in Appendix B) to pretrain the representations of each view's data. In Figure 2(a), one can find that different views inherently have different levels of discriminative information and exhibit different qualities, where the worst (view 1)
and the best (view 4) have a large gap. Then, we adopt InfoNCE loss to perform contrastive learning between view 1 and view 4 in Figure 2(b), and record the clustering accuracy of their representations in Figure 2(c). We can observe that InfoNCE loss is well-minimized, which makes the representation similarity (evaluated by cosine) between view 1 and view 4 converge to 1.0. The performance on view 1 gradually increases. Nevertheless, the cost is that the representations of view 4 degenerate, on which the useful discriminative information reduces and thus the performance gradually decreases.
In multi-view learning, quality difference among multiple views is a common phenomenon. However, the representation degeneration in multi-view contrastive learning might make the representations of some high-quality views tend to be mediocre and thus miss their useful discriminative information.
3.2 Self-Weighted Multi-View Contrastive Learning with Reconstruction Regularization
To mitigate representation degeneration in multi-view contrastive learning, we propose a simple but effective framework called SEM: Self-weighted Multi-view contrastive learning with reconstruction regularization as shown in Figure 1. Specifically, given view-specific data  $\mathbf{X}^v\in \mathbb{R}^{N\times d_v}$ , we let  $\mathbf{Z}^v\in \mathbb{R}^{N\times z}$  denote the corresponding new representations learned by a view-specific encoder. Between  $\mathbf{X}^v$  and  $\mathbf{Z}^v$ , we record a precursor state of representations as  $\mathbf{H}^v\in \mathbb{R}^{N\times h_v}$  (termed as hidden features), and the encoder is partitioned into two parts (the front and back parts are stacked and denoted as  $f^v$  and  $g^v$  sequentially). For the  $v$ -th view, we let  $\Psi^v$  and  $\Phi^v$  denote the network parameters of  $f^v$  and  $g^v$ , respectively, and then the view-specific model can be formulated as follows:
$$
\mathbf {Z} ^ {v} = g ^ {v} \left(\mathbf {H} ^ {v}; \Phi^ {v}\right) = g ^ {v} \left(f ^ {v} \left(\mathbf {X} ^ {v}; \Psi^ {v}\right); \Phi^ {v}\right). \tag {1}
$$
In SEM, we leverage  $\mathcal{L}_{CL}^{m,n}(\mathbf{Z}^m,\mathbf{Z}^n)$  to denote a contrastive loss $^2$ , and let  $\lambda >0$  denote a trade-off coefficient on regularization terms. Then, SEM is trained by minimizing the following objective:
$$
\sum_ {m, n} \mathcal {W} ^ {m, n} \mathcal {L} _ {C L} ^ {m, n} \left(\mathbf {Z} ^ {m}, \mathbf {Z} ^ {n}\right) + \lambda \sum_ {v} \mathcal {R} ^ {v} \left(\mathbf {X} ^ {v}, \mathbf {H} ^ {v}\right), \tag {2}
$$
where  $\mathcal{W}^{m,n}$  is the pairwise weight between the  $m$ -th and the  $n$ -th views, and  $\mathcal{R}^v (\mathbf{X}^v,\mathbf{H}^v)$  denotes the reconstruction regularization on  $\mathbf{H}^v$ . We define  $\mathcal{D}(\mathbf{Z}^m,\mathbf{Z}^n)$  as the discrepancy between  $\mathbf{Z}^m$  and  $\mathbf{Z}^n$  and denote  $\mathcal{F}$  as a negative correlation function. Then, in SEM, the pairwise weight is updated by
$$
\mathcal {W} ^ {m, n} = \mathcal {F} \left(\mathcal {D} \left(\mathbf {Z} ^ {m}, \mathbf {Z} ^ {n}\right)\right). \tag {3}
$$
Self-weighting In unsupervised settings, it is hard to know which representations within  $\{\mathbf{Z}^v\}_{v=1}^V$  contain useful semantic information and which are with more noise. To mitigate the representation degeneration caused by contrastive learning, SEM needs to be adaptive to quality difference among multiple views. Therefore, different from using equal-sum manner [11, 14, 17] (e.g.,  $\sum_{m,n} \mathcal{L}_{CL}^{m,n}$ ), we propose to use the pairwise weighted multi-view contrastive loss, i.e.,  $\sum_{m,n} \mathcal{W}^{m,n} \mathcal{L}_{CL}^{m,n}$ . Here,  $\mathcal{W}^{m,n}$  leverages the discrepancy to achieve the adaptive self-weighting. Concretely, if two views are useful pairwise views and both with informative semantics, contrastive learning between them is adaptively strengthened; if two views are unreliable pairwise views (for example, one or two of them are with less informative semantics), contrastive learning between them is adaptively weakened.
Reconstruction regularization In Eq. (2),  $\mathcal{R}^v (\mathbf{X}^v,\mathbf{H}^v)$  acts as a self-supervised objective to transfer as much discriminative information as possible from  $\mathbf{X}^v$  to  $\mathbf{H}^v$ . When we record  $\mathbf{H}^v$  as the hidden features in encoder networks, the information transfer path can be described as  $\mathbf{X}^v\rightarrow \mathbf{H}^v\rightarrow \mathbf{Z}^v,v\in$ $\{1,2,\ldots ,V\}$ . However, information losing might occur in the processing of  $\mathbf{X}^v\to \mathbf{H}^v$  such that discriminative information from some views' data is lost, and thus making contrastive learning among  $\{\mathbf{Z}^v\}_{v = 1}^V$  focus on harmful noise instead of common semantics across multiple views. To this end, on hidden features  $\mathbf{H}^v$ , our SEM leverages  $\mathbf{X}^v$  to build the reconstruction regularization  $\mathcal{R}^v (\mathbf{X}^v,\mathbf{H}^v)$  to assist contrastive learning in accessing sufficient discriminative information from raw data.
$^{2}\mathcal{L}_{CL}^{m,n}(\mathbf{Z}^{m},\mathbf{Z}^{n})$  can be easily replaced by previous contrastive losses, e.g., InfoNCE [8], RINCE [15], and PSCL [16]. Let  $\mathcal{P}$  denote the set of positive sample pairs and  $\mathcal{N}$  is the set of negative sample pairs in the  $m,n$ -th views,  $q$  and  $\alpha$  are hyper-parameters of RINCE, then the three contrastive losses could be formulated as follows:
$\mathcal{L}_{InfoNCE}^{m,n} = -\mathbb{E}_{s^+ \in \mathcal{P}}\left[s^+ - \log \left(e^{s^+} + \sum_{s^- \in \mathcal{N}} e^{s^-}\right)\right]$ ,
$\mathcal{L}_{RINCE}^{m,n} = -\mathbb{E}_{s^+ \in \mathcal{P}}\left[\frac{1}{q}\cdot e^{q\cdot s^+} - \frac{1}{q}\cdot \left(\alpha \cdot \left(e^{s^+} + \sum_{s^- \in \mathcal{N}}e^{s^-}\right)\right)^q\right],$
$\mathcal{L}_{PSCL}^{m,n} = -\mathbb{E}_{s^+ \in \mathcal{P}}\left[2 \cdot s^+\right] + \mathbb{E}_{s^- \in \mathcal{N}}\left[(s^-)^2\right]$ ,
where  $s^+ (s^-)$  denotes the cosine distance between the representations of positive (negative) sample pair.
3.3 Different Options for Implementing the SEM Framework
The crucial components of our proposed SEM as Eq. (2) include the weighting strategy  $\mathcal{W}^{m,n}$ , contrastive loss  $\mathcal{L}_{CL}^{m,n}$ , and regularization term  $\mathcal{R}^v$ . Next, we concentrate on the implementations of  $\mathcal{W}^{m,n}$  (including JSD, MMD, and CMI) and briefly introduce the implementations of  $\mathcal{L}_{CL}^{m,n}$  and  $\mathcal{R}^v$ .
Discrepancy measurements of weighting strategy When implementing  $\mathcal{W}^{m,n} = \mathcal{F}(\mathcal{D}(\mathbf{Z}^m,\mathbf{Z}^n))$  in Eq. (3), many methods can measure the discrepancy  $\mathcal{D}(\mathbf{Z}^m,\mathbf{Z}^n)$ . Firstly, we can transfer representations to a probability distribution and leverage Jensen-Shannon divergence (JSD) to compute the discrepancy  $\mathcal{D}_{JSD}(\mathbf{Z}^m,\mathbf{Z}^n)$ . The advantages of JSD are its symmetry and simplicity, but it might be inapplicable when two distributions are non-overlapping. Furthermore, we can leverage maximum mean discrepancy (MMD) as the second method to obtain the discrepancy  $\mathcal{D}_{MMD}(\mathbf{Z}^m,\mathbf{Z}^n)$ . MMD can effectively measure non-overlapping two distributions, but it has higher complexity than  $\mathrm{JSD}^3$ .
Actually, both JSD and MMD leverage all information of representations  $\mathbf{Z}^m$  and  $\mathbf{Z}^n$ . However, the semantic-irrelevant information or random noise might also be embedded in  $\mathbf{Z}^m$  and  $\mathbf{Z}^n$ . Moreover, what we expect to obtain is the mutual relation of their most representative semantic information. To this end, we propose Class Mutual Information (CMI) as the third method to obtain the discrepancy  $\mathcal{D}_{CMI}(\mathbf{Z}^m,\mathbf{Z}^n)$ . To be specific, since it is difficult to accurately estimate the mutual information (denoted as  $I$ ) for multi-dimensional continuous variables  $I(\mathbf{Z}^m;\mathbf{Z}^n)$ , we denote  $\mathbf{y}^m$  and  $\mathbf{y}^n$  as 1-dimensional discrete vectors and change estimating  $I(\mathbf{Z}^m;\mathbf{Z}^n)$  to computing  $I(\mathbf{y}^m;\mathbf{y}^n)$  such that:
$$
\mathcal {W} ^ {m, n} \approx \mathcal {F} \left(1 / I \left(\mathbf {Z} ^ {m}; \mathbf {Z} ^ {n}\right)\right) \approx \mathcal {F} \left(1 / I \left(\mathbf {y} ^ {m}; \mathbf {y} ^ {n}\right), s. t. \underset {\mathbf {y} ^ {m}, \mathbf {y} ^ {n}} {\operatorname {a r g m a x}} I \left(\mathbf {Z} ^ {m}; \mathbf {y} ^ {m}\right) + I \left(\mathbf {Z} ^ {n}; \mathbf {y} ^ {n}\right). \right. \tag {4}
$$
Intuitively, discrete class information in  $\mathbf{Z}^v$  ( $v \in \{m, n\}$ ) is 1-dimensional as well as the most representative information. Hence, we can optimize K-Means objective to extract the class information:
$$
\mathbf {Y} ^ {v *} = \underset {\mathbf {Y} ^ {v}, \mathbf {C} ^ {v}} {\operatorname {a r g m a x}} \| \mathbf {Z} ^ {v} - \mathbf {Y} ^ {v} \mathbf {C} ^ {v} \| _ {F} ^ {2}, \text {s . t .} \mathbf {Y} ^ {v} \left(\mathbf {Y} ^ {v}\right) ^ {T} = \mathbf {I} _ {N}, \mathbf {Y} ^ {v} \in \{0, 1 \} ^ {N \times K}, \tag {5}
$$
where  $\mathbf{C}^v\in \mathbb{R}^{K\times z}$  denotes the  $K$  cluster centers of  $\mathbf{Z}^v$ .  $\mathbf{Y}^{v*}\in \{0,1\}^{N\times K}$  is the indicator matrix that can be further transformed to 1-dimensional discrete vector  $\mathbf{y}^v$  by defining  $y_{i}^{v}:= \operatorname{argmax}_{j}y_{ij}^{v*}$  where  $y_{i}^{v}\in \mathbf{y}^{v},y_{ij}^{v*}\in \mathbf{Y}^{v*}$ . In this way, the class information in  $\mathbf{Z}^m$  and  $\mathbf{Z}^n$  can be compressed into  $\mathbf{y}^m$  and  $\mathbf{y}^n$ , respectively. Then, the class mutual information  $I(\mathbf{y}^m;\mathbf{y}^n)$  is normalized and the discrepancy measurement  $\mathcal{D}_{CMI}(\mathbf{Z}^m,\mathbf{Z}^n)$  between pairwise views is defined as follows:
$$
\mathcal {D} _ {C M I} \left(\mathbf {Z} ^ {m}, \mathbf {Z} ^ {n}\right) = \frac {H \left(\mathbf {y} ^ {m}\right) + H \left(\mathbf {y} ^ {n}\right)}{2 \cdot I \left(\mathbf {y} ^ {m} ; \mathbf {y} ^ {n}\right)}, \tag {6}
$$
where  $H(\mathbf{y}^m) = -\sum_{i=1}^{N} p(y_i^m) \log p(y_i^m)$  is the cross-entropy of  $\mathbf{y}^m$ . This design of CMI has at least two advantages: 1) It is conducive to maintaining the representative class information while filtering out noise information; 2) Calculation is easy and owns better physical meaning.
Finally, it is also flexible to implement the negative correlation function  $\mathcal{F}$ . Considering  $\mathcal{W}^{m,n} \geq 0$ , we base on the three different discrepancies and simply give the following weighting strategies:
$$
\mathcal {W} _ {C M I} ^ {m, n} = \mathcal {F} _ {C M I} \left(\mathcal {D} _ {C M I} \left(\mathbf {Z} ^ {m}, \mathbf {Z} ^ {n}\right)\right) = e ^ {1 / \mathcal {D} _ {C M I} \left(\mathbf {Z} ^ {m}, \mathbf {Z} ^ {n}\right)} - 1,
$$
$$
\mathcal {W} _ {J S D} ^ {m, n} = \mathcal {F} _ {J S D} \left(\mathcal {D} _ {J S D} \left(\mathbf {Z} ^ {m}, \mathbf {Z} ^ {n}\right)\right) = e ^ {1 - \mathcal {D} _ {J S D} \left(\mathbf {Z} ^ {m}, \mathbf {Z} ^ {n}\right)} - 1, \tag {7}
$$
$$
\mathcal {W} _ {M M D} ^ {m, n} = \mathcal {F} _ {M M D} (\mathcal {D} _ {M M D} (\mathbf {Z} ^ {m}, \mathbf {Z} ^ {n})) = e ^ {- \mathcal {D} _ {M M D} (\mathbf {Z} ^ {m}, \mathbf {Z} ^ {n})}.
$$
Compatibility for contrastive learning When implementing the contrastive loss  $\mathcal{L}_{CL}^{m,n}$ , it should be pointed out that multi-view contrastive learning usually has to handle more than two views (i.e.,
<sup>3</sup>We write  $\hat{\mathbf{z}}_i^m\in \hat{\mathbf{Z}}^m = Softmax(\mathbf{Z}^m)$ .  $k(\mathbf{z}_i,\mathbf{z}_j)$  denotes the inner product of  $\phi (\mathbf{z}_i)$  and  $\phi (\mathbf{z}_j)$ , where  $\phi (\cdot)$  denotes the mapping (e.g., by Gaussian kernel) to project representations into Reproducing Kernel Hilbert Space (RKHS). Then,  $\mathcal{D}_{JSD}(\mathbf{Z}^m,\mathbf{Z}^n)$  and  $\mathcal{D}_{MMD}(\mathbf{Z}^m,\mathbf{Z}^n)$  can be formulated as follows: