SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ping Guo‡,§
Baosong Yang†
Xiangpeng Wei\*,*
Dayiheng Liu†
Yue Hu‡,§,*
Fei Huang†
Jun Xie†
$^{\ddagger}$ Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China
$^{\S}$ School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
†Machine Intelligence Technology Lab, Alibaba DAMO Academy, Hangzhou, China
{guoping, huyue}@ie.ac.cn, pemywei@gmail.com
Expressing universal semantics common to all languages is helpful in understanding the meanings of complex and culture-specific sentences. The research theme underlying this scenario focuses on learning universal representations across languages with the usage of massive parallel corpora. However, due to the sparsity and scarcity of parallel data, there is still a big challenge in learning authentic "universals" for any two languages. In this paper, we propose EMMA-X: an EM-like Multilingual pre-training Algorithm, to learn (X)Cross-lingual universals with the aid of excessive multilingual non-parallel data. EMMA-X unifies the cross-lingual representation learning task and an extra semantic relation prediction task within an EM framework. Both the extra semantic classifier and the cross-lingual sentence encoder approximate the semantic relation of two sentences, and supervise each other until convergence. To evaluate EMMA-X, we conduct experiments on XRETE, a newly introduced benchmark containing 12 widely studied cross-lingual tasks that fully depend on sentence-level representations. Results reveal that EMMA-X achieves state-of-the-art performance. Further geometric analysis of the built representation space with three requirements demonstrates the superiority of EMMA-X over advanced models<sup>2</sup>.
Research on how to express universal semantics for natural languages (metaphorically as "alphabet of human thoughts" by Leibniz and von Leibniz [1996]) has lasted a long time. Usually, these universal meanings underlying all human natural languages are referred to as irreducible semantic cores [Wierzbicka, 1999]. These common cores across languages can serve as a bridge, to help better understand the exact meanings of complex sentences in different languages.
In the context of computational linguistics, various works [Huang et al., 2019, Conneau et al., 2020, Chi et al., 2021, Wei et al., 2021, Lee et al., 2022, Li et al., 2023, Chen et al., 2023] have led to great improvements on learning cross-lingual universal representations with the usage of parallel corpora, and verify that multilingual universality contributes a major performance on cross-lingual understanding. However, due to the sparsity and scarcity of parallel data, these advanced techniques face a big challenge in learning real universality among all languages. For instance, among the widely-available top 100 languages that theoretically can build 4950 language pairs, only about 200 language pairs have considerable parallel data [Aharoni et al., 2019, Bapna et al.,
$^{2}$ Codes and datasets of the XRETE benchmark: https://github.com/guopingjie/EMMA-X
2022]. Recently, Large Language Models (LLMs) (e.g., PaLM [Chowdhery et al., 2022], OPT [Zhang et al., 2022b], BLOOMZ [Workshop et al., 2023], ChatGPT, etc.) have reached a milestone in the field of Natural Language Processing, for their promising capability at understanding and following complex natural language instructions in different languages. By modeling a wide variety of sentence samples in discrete sentence space, LLMs can capture some universal linguistic phenomena to gain cross-lingual transferability. This is consistent with our goal of building a universal basement that supports all languages. The difference lies in that we achieve it through learning universal continuous representations across different languages.
Concretely, we propose EMMA-X to tackle the above challenge from a continuous perspective. EMMA-X can learn cross-lingual universal sentence representations with excessive non-parallel multilingual data by unifying two highly dependent tasks in an EM [Moon, 1996] framework: semantic relation classification and cross-lingual universal representation learning. For the former, we introduce a Gaussian Mixture Model [Everitt and Hand, 1981] classifier (GMM classifier) to deal with the key challenge of forming positive sentence pairs for non-parallel multilingual corpora, by annotating the semantic relationship of sentence pairs in any two arbitrary languages on the fly. For the latter, we employ a cross-lingual encoder to learn universal sentence representations via contrastive learning, where positive pairs are chosen by GMM classifier. Further, we construct training signals according to the output of the cross-lingual encoder, to inversely supervise GMM classifier. From the perspective of EM algorithm, in E-step, both modules try to approximate the semantic relationship given a sentence pair sampled from two arbitrary languages. One module is supervised by the approximation of the other to build its own expectation. In M-step, two modules update their parameters by maximizing expectations, respectively. We give a theoretical justification about how these two tasks can be interpreted from an EM perspective (Section 4).
To incentivize the research of universal sentence representation learning, we form a Cross-lingual RRepresentation Transfer Evaluation (XRETE) benchmark, which includes 12 cross-lingual tasks covering more than 50 languages. XRETE fully depends on sentence-level representations. Experimental results demonstrate that EMMA-X significantly outperforms pre-trained language models [Conneau et al., 2020, Chi et al., 2021] by  $32\%$  at most on XRETE. We also perform an evaluation of ChatGPT on XRETE to explore its multilingual performance. Detailed analysis also shows that EMMA-X can mitigate the representation discrepancy between head and massive long-tail languages. We further conduct geometric analysis directly on representation space from three perspectives: Invariance [Abend and Rappoport, 2017], Canonical Form [Teller, 2000] and Isotropy [Mu and Viswanath, 2018], which provides a further understanding of the cross-lingual transferability of these models.
Cross-lingual representation learning aims at mapping sentences from different languages into a unified continuous space, where synonyms across different languages are pulled closer. Given a sentence  $\mathbf{x}$ , the representation is formulated as
$$
\gamma^ {(\mathbf {x})} = f \left[ g \left(\mathcal {M} (\mathbf {x}; \Theta_ {\mathcal {M}})\right) \right], \tag {1}
$$
where  $\mathcal{M}(\cdot ;\Theta_{\mathcal{M}})$  denotes the encoder network with a set of trainable parameters  $\Theta_{\mathcal{M}}$  , which is typically implemented as a transformer encoder architecture [Vaswani et al., 2017, Devlin et al., 2019, Lee et al., 2022, Feng et al., 2022].  $f(\cdot)$  is L-2 normalization and  $g(\cdot)$  is the aggregate function. We take the final hidden states of " [CLS] " token as the aggregate sentence representation.
To learn reasonable representations that can express universal semantics across different languages, various well-designed techniques have been applied to  $\gamma^{(\mathbf{x})}$ . A predominant one is to build contrastive learning (CTL) [Saunshi et al., 2019] objective with parallel corpora. The basic idea is to maximize the similarity between representations (i.e.,  $\gamma^{(\mathbf{x})}$  and  $\gamma^{(\mathbf{y})}$ ) of two semantically-equivalent sentences  $(\mathbf{x},\mathbf{y})$ , while keep randomly sampled irrelevant ones  $\gamma^{(\mathbf{y}^{\prime})}$  away. Formally, assume  $\mathcal{B}$  to be a batch of multilingual parallel bitexts, the contrastive loss under InfoNCE [Oord et al., 2018] formulation is
$$
\mathcal {L} _ {\mathbf {C T L}} = - \log \frac {e ^ {s \left(\gamma^ {(\mathbf {x})} , \gamma^ {(\mathbf {y})}\right)}}{e ^ {s \left(\gamma^ {(\mathbf {x})} , \gamma^ {(\mathbf {y})}\right)} + \sum_ {\mathbf {y} ^ {\prime} \in \mathcal {B} , \mathbf {y} ^ {\prime} \neq \mathbf {y}} e ^ {s \left(\gamma^ {(\mathbf {x})} , \gamma^ {(\mathbf {y} ^ {\prime})}\right)}}, \tag {2}
$$
where  $s(\cdot)$  is implemented as the cosine similarity  $s(\gamma^{(\mathbf{x})},\gamma^{(\mathbf{y})}) = \frac{\gamma^{(\mathbf{x})\top}\gamma^{(\mathbf{y})}}{\|\gamma^{(\mathbf{x})}\| \cdot \|\gamma^{(\mathbf{y})}\|}$ ,  $\mathbf{y}$  and  $\mathbf{y}'$  are typically called positive and negative samples.
Figure 1: Illustration of EMMA-X, involving two modules (GMM classifier and Cross-lingual Encoder) that are reciprocated from each other and are updated alternatively.  $\mathbf{x}$  means the current instance,  $\{\mathbf{y}_1,\mathbf{y}_2,\mathbf{y}_3,\ldots \}$  are samples in various languages for comparison.  $\gamma^{(\mathbf{x})}$  is the continuous representation given a discrete sentence  $\mathbf{x}$ .  $c_{\mathcal{M}}^*$  and  $c_{\mathcal{G}}^*$  formulate the semantic ranks approximated according to Eq. 10 and Eq. 9, to supervise the GMM classifier and cross-lingual encoder, respectively.
We propose EMMA-X that fully exploits massive monolingual data to learn cross-lingual universal representations. As illustrated in Figure 1, EMMA-X consists of two modules: 1) A GMM classifier  $\mathcal{G}(\cdot ;\Theta_{\mathcal{G}})$  to approximate the semantic relation of non-parallel sentences. 2) A cross-lingual encoder  $\mathcal{M}(\cdot ;\Theta_{\mathcal{M}})$  to convert multilingual sentences into universal representations. For optimization, EMMA-X unifies these two modules in an EM framework with dual supervision. In this section, we begin with a definition of the semantic relation rank (§3.1). Then, we introduce model initialization (§3.2) and the proposed training paradigm (§3.3), followed by a dual supervision strategy (§3.4). For a clearer presentation, an Algorithm of EMMA-X is shown in Algorithm 1.
3.1 Rank of Semantic Relations
Mainstream methods model semantic relations with a strict binary separation: positives and negatives. However, the boundary between positives and negatives is blurry, and many samples cannot be clearly classified as either positives or negatives. So it cannot maximize the potential of models to perceive more subtle semantic changes. Also, a binary separation will lead to far more negative samples than positive ones (imbalanced data). To more accurately capture the semantic relation between two sentences and alleviate imbalanced problem, we subdivide the relation into  $N$  semantic ranks, where the semantic similarity of each rank decreases as  $N$  increases, e.g.,  $c = 1$  denotes two sentences are paraphrases of each other, while  $c = N$  implies sentences are irrelevant. In practice, we set  $N$  to 4.
3.2 Model Initialization
In EMMA-X, the GMM classifier  $\mathcal{G}(\cdot ;\Theta_{\mathcal{G}})$  and cross-lingual encoder  $\mathcal{M}(\cdot ;\Theta_{\mathcal{M}})$  are initialized by training with massive parallel corpora, respectively.
Initialization of Cross-lingual Encoder. It is initialized with XLM-R [Conneau et al., 2020] and then continuously trained with InfoNCE [Oord et al., 2018] loss by Eq.2. Following HICTL [Wei et al., 2021] and INFOXLM [Chi et al., 2021], we treat the parallel sentence pairs as the query sentence  $\mathbf{x}$  and its positive counterpart  $\mathbf{y}$ , while treating the randomly sampled sentence as a negative one  $\mathbf{y}'$ .
Initialization of GMM Classifier. A reasonable solution to warm up the GMM classifier is to use the available cross-lingual parallel corpora as training signals. Suppose  $\mathbf{x}$  and  $\mathbf{y}$  are parallel sentences,
and  $\mathbf{y}'$  is an outlier. We set the semantic ranks for  $(\gamma^{(\mathbf{x})}, \gamma^{(\mathbf{y})})$  and  $(\gamma^{(\mathbf{x})}, \gamma^{(\mathbf{y}')})$  as  $c = 1$  and  $c = N$ , respectively, according to the definition described in §3.1. To obtain the fine-grained semantic ranks, we design a linear interpolation strategy similar to Wei et al. [2022] and mixup [Zhang et al., 2018], which provides virtual training examples for each semantic rank. Formally,
$$
\gamma^ {(\tilde {\mathcal {Y}})} = (1 - \lambda) \cdot \gamma^ {(\mathcal {Y})} + \lambda \cdot \gamma^ {(\mathcal {Y} ^ {\prime})}, \tag {3}
$$
where  $\lambda \in [0,1]$  is sampled from a uniform distribution. We compute  $r = \lceil (1 - \lambda)\cdot (c = 1) + \lambda \cdot (c = N)\rceil$  as the soft semantic rank for  $(\gamma^{(\mathbf{x})},\gamma^{\langle \tilde{y}\rangle})$ , where  $\lceil \cdot \rceil$  means the least integer greater than or equal to the input. The virtual training examples are grouped together with the real parallel corpora to optimize the GMM classifier:
$$
\mathcal {L} _ {\mathbf {M L E}} = - \log P _ {\mathcal {G}} (c = 1 | \mathbf {x}, \mathbf {y}; \Theta_ {\mathcal {G}}) - \log P _ {\mathcal {G}} (c = N | \mathbf {x}, \mathbf {y} ^ {\prime}; \Theta_ {\mathcal {G}}) - \sum_ {\tilde {\mathbf {y}}} \log P _ {\mathcal {G}} (c = r | \mathbf {x}, \tilde {\mathbf {y}}; \Theta_ {\mathcal {G}}). \tag {4}
$$
The posterior probability  $P_{\mathcal{G}}(\cdot)$  is formulated as
$$
P _ {\mathcal {G}} (c = r | \mathbf {x}, \mathbf {y}; \Theta_ {\mathcal {G}}) = \frac {\pi_ {r} \cdot \mathcal {N} _ {r} \left(\gamma^ {(\mathbf {x})} - \gamma^ {(\mathbf {y})} \mid \mu_ {r} , \sigma_ {r}\right)}{\sum_ {j = 1} ^ {N} \pi_ {j} \cdot \mathcal {N} _ {j} \left(\gamma^ {(\mathbf {x})} - \gamma^ {(\mathbf {y})} \mid \mu_ {j} , \sigma_ {j}\right)}, \tag {5}
$$
where the Gaussian form  $\mathcal{N}_r$  is assigned with a prior probability  $\pi_r$ , mean  $\mu_r$  and standard deviation  $\sigma_r$  that are all parameterized by trainable variables, thus  $\Theta_{\mathcal{G}} = \{(\pi_r,\mu_r,\sigma_r)\mid r\in [1,N]\}$ . See Appendix G for the specific calculation of Gaussian form  $\mathcal{N}_r$ .
3.3 The EM Iteration Framework
After initialization, EMMA-X further trains the GMM classifier  $\mathcal{G}(\cdot ;\Theta_{\mathcal{G}})$  and cross-lingual encoder  $\mathcal{M}(\cdot ;\Theta_{\mathcal{M}})$  with only multilingual non-parallel data with an EM framework.
E-Step. For optimization, we represent a training batch of multilingual non-parallel sentences as  $\mathcal{X} = \{\mathbf{x}_1,\mathbf{x}_2,\dots ,\mathbf{x}_I\}$  accompanied by a queue of random sentences as  $\mathcal{Y} = \{\mathbf{y}_1,\mathbf{y}_2,\dots ,\mathbf{y}_K\}$  for instance comparison. Formally, the expectation for GMM classifier is:
$$
\mathcal {L} _ {\mathbf {M L E}} (\mathcal {X}, \mathcal {Y}; \Theta_ {\mathcal {G}}) = - \mathbb {E} _ {\mathbf {x} _ {i} \sim \mathcal {X}} \mathbb {E} _ {\mathbf {y} _ {k} \sim \mathcal {Y}} \left[ \log P _ {\mathcal {G}} \left(c = c _ {\mathcal {M}} ^ {*} \mid \mathbf {x} _ {i}, \mathbf {y} _ {k}; \Theta_ {\mathcal {G}}\right) \right], \tag {6}
$$
where  $c_{\mathcal{M}}^* \in [1, N]$  represents an approximated semantic rank for the combination of anchor  $\mathbf{x}_i$  and another random sentence  $\mathbf{y}_k$ , based on the cosine similarity among representations (i.e.,  $\gamma^{(\mathbf{x}_i)}$  and  $\gamma^{(\mathbf{y}_k)}$ ) produced by the cross-lingual encoder (i.e.,  $\mathcal{M}(\cdot; \Theta_{\mathcal{M}})$ ). Please refer to §3.4 for details.
Correspondingly, the expectation for the cross-lingual encoder can be calculated with contrastive learning, where the positive samples are established by the maximum a posteriori approximation (argmax prediction)  $c_{\mathcal{G}}^{*}$  given by the GMM classifier. Specifically, we apply ranking InfoNCE [Hoffmann et al., 2022] as the training objective, which recursively takes parallel sentence pairs in each rank (e.g.,  $c_{\mathcal{G}}^{*}$ ) as positives and ranks that are larger than  $c_{\mathcal{G}}^{*}$  as negatives. Formally,
$$
\begin{array}{l} \mathcal {L} _ {\mathbf {C T L}} (\mathcal {X}, \mathcal {Y}; \Theta_ {\mathcal {M}}) = - \mathbb {E} _ {\mathbf {x} _ {i} \sim \mathcal {X}} \left[ \log \frac {\sum_ {\mathbf {y} _ {k} \sim \mathcal {Y} _ {c _ {\mathcal {G}} ^ {*} = 1}} e ^ {s [ \gamma^ {(\mathbf {x} _ {i})} , \gamma^ {(\mathbf {y} _ {k})} ]}}{\sum_ {\mathbf {y} _ {t} \sim \mathcal {Y} _ {c _ {\mathcal {G}} ^ {*} \in [ 1 , N ]}} e ^ {s [ \gamma^ {(\mathbf {x} _ {i})} , \gamma^ {(\mathbf {y} _ {t})} ]}} + \log \frac {\sum_ {\mathbf {y} _ {k} \sim \mathcal {Y} _ {c _ {\mathcal {G}} ^ {*} = 2}} e ^ {s [ \gamma^ {(\mathbf {x} _ {i})} , \gamma^ {(\mathbf {y} _ {k})} ]}}{\sum_ {\mathbf {y} _ {t} \sim \mathcal {Y} _ {c _ {\mathcal {G}} ^ {*} \in [ 2 , N ]}} e ^ {s [ \gamma^ {(\mathbf {x} _ {i})} , \gamma^ {(\mathbf {y} _ {t})} ]}} \right] \\ + \dots + \log \frac {\sum_ {\mathbf {y} _ {k} \sim \mathcal {Y} _ {\mathcal {G}} ^ {*} = N - 1} e ^ {s [ \gamma (\mathbf {x} _ {i}) , \gamma (\mathbf {y} _ {k}) ]}}{\sum_ {\mathbf {y} _ {t} \sim \mathcal {Y} _ {\mathcal {G}} ^ {*} \in [ N - 1 , N ]} e ^ {s [ \gamma (\mathbf {x} _ {i}) , \gamma (\mathbf {y} _ {t}) ]}} \Bigg ], \tag {7} \\ \end{array}
$$
where  $c_{\mathcal{G}}^{*} \in [1, N]$  represents a semantic rank approximated by the posteriori of GMM classifier (§3.4). For simplicity, we omit the temperature term in Eq. 7, and please see Appendix G for details.
M-Step. We use gradient descent algorithm to update the parameters of each module by optimizing its expectation. At each time step  $t$ , where  $\eta$  and  $\eta'$  are learning rates for each expectation, formally,
$$
\begin{array}{l} \Theta_ {\mathcal {G}} ^ {t + 1} \leftarrow \Theta_ {\mathcal {G}} ^ {t} - \eta \times \nabla_ {\Theta_ {\mathcal {G}}} \mathcal {L} _ {\mathbf {M L E}} (\mathcal {X}, \mathcal {Y}; \Theta_ {\mathcal {G}}), \tag {8} \\ \Theta_ {\mathcal {M}} ^ {t + 1} \gets \Theta_ {\mathcal {M}} ^ {t} - \eta^ {\prime} \times \nabla_ {\Theta_ {\mathcal {M}}} \mathcal {L} _ {\mathbf {C T L}} (\mathcal {X}, \mathcal {Y}; \Theta_ {\mathcal {M}}). \\ \end{array}
$$
Algorithm 1: EMMA-X Training Algorithm
Input: multilingual parallel and non-parallel corpora   
Output:  $\mathcal{M}(:,\Theta_{\mathcal{M}})$  and  $\mathcal{G}(:,\Theta_{\mathcal{G}})$    
Phase 1; /\*Warm-up two modules with multilingual parallel corpora \*/   
while not convergence do Sample a batch of multilingual bitexts B; for  $(\mathbf{x},\mathbf{y})\in \mathcal{B}$  do Sample  $\mathbf{y}'\in \mathcal{B},\mathbf{y}'\neq \mathbf{y}$  , Build virtual training examples  $\tilde{\mathbf{y}}$  with Eq. 3; Compute  $\mathcal{L}_{\mathrm{CTL}}$  with Eq. 2 to update  $\Theta_{\mathcal{M}}$  and  $\mathcal{L}_{\mathrm{MLE}}$  with Eq. 4 to update  $\Theta_{\mathcal{G}}$  end   
end   
Phase 2; /\*EMMA-X training with EM framework using only non-parallel corpora \*/ while not convergence do E-Step Sample a batch of multilingual anchors  $\mathcal{X}$  , Queue a batch of random sentences  $\mathcal{V}$  . for  $(\mathbf{x}_i,\mathbf{y}_k)\in \mathcal{X}\times \mathcal{Y}$  do Approximate semantic rank  $c_{\mathcal{G}}^{*}$  with Eq. 9 and  $c_{\mathcal{M}}^{*}$  with Eq. 10 ; end Compute  $\mathcal{L}_{\mathrm{MLE}}(\mathcal{X},\mathcal{Y};\Theta_{\mathcal{G}})$  according to Eq. 6 and  $\mathcal{L}_{\mathrm{CTL}}(\mathcal{X},\mathcal{Y};\Theta_{\mathcal{M}})$  according to Eq. 7; M-Step Update  $\Theta_{\mathcal{M}}$  and  $\Theta_{\mathcal{G}}$  according to Eq. 8;
3.4 Dual Supervision
The approximated semantic ranks  $c_{\mathcal{G}}^{*}$  and  $c_{\mathcal{M}}^{*}$  are critical in EMMA-X training algorithm. To preserve their quality, we propose dual supervision: predictions from one module are fed to the other to calculate the expectation. In this section, we explain in detail how we approximate the semantic ranks for GMM classifier and cross-lingual encoder, respectively.
Approximate Semantic Rank with GMM classifier. The way to obtain semantic rank with semantic classifier is straightforward. The semantic rank corresponding to the highest probability among multiple Gaussian distributions is chosen as the prediction, which is further used to supervise the cross-lingual encoder  $\mathcal{M}(\cdot ;\Theta_{\mathcal{M}})$ , as illustrated in Eq. 7. Formally,
$$
c _ {\mathcal {G}} ^ {*} = \underset {r} {\operatorname {a r g m a x}} P _ {\mathcal {G}} (c = r | \mathbf {x} _ {i}, \mathbf {y} _ {k}; \Theta_ {\mathcal {G}}). \tag {9}
$$
Approximate Semantic Rank with Cross-lingual Encoder. One common way to calculate sentence relation is to measure the similarity between two real-valued representations. Assuming  $s_r$  (a scalar initialized as  $\frac{r}{N}$ ) can reflect the general similarity score in semantic rank  $c = r$ . Given a random sentence pair  $(\mathbf{x}_i, \mathbf{y}_k)$ , if its similarity score is close to  $s_r$ , the sentence pair is likely to belong to rank  $c = r$ . Cross-lingual encoder  $\mathcal{M}(\cdot; \Theta_{\mathcal{M}})$  determines the semantic relation for each pair according to
$$
c _ {\mathcal {M}} ^ {*} = \underset {r} {\operatorname {a r g m i n}} | s \left(\gamma^ {\left(\mathbf {x} _ {i}\right)}, \gamma^ {\left(\mathbf {y} _ {k}\right)}\right) - s _ {r} |, \tag {10}
$$
where  $|\cdot |$  refers to absolute value. Symmetrically,  $c_{\mathcal{M}}^{*}$  is used to supervise GMM classifier (Eq. 6).
During the training process, the general similarity score for each semantic rank may vary. Thus, we propose a moving-average strategy to adaptively adjust the value of  $s_r$  to simulate this change. Specifically, at time step  $t$ ,  $s_r$  is updated by cosine similarity of all the sentence pairs, which are currently categorized into the rank  $c = r$  according to the cross-lingual encoder in Eq. 9.
$$
s _ {r} ^ {t} \leftarrow \epsilon \cdot s _ {r} ^ {t - 1} + (1 - \epsilon) \cdot s \left(\gamma^ {\left(\mathbf {x} _ {i}\right)}, \gamma^ {\left(\mathbf {y} _ {k}\right)}\right), \quad \text {i f} \mathbf {y} _ {k} \in \mathcal {Y} _ {c _ {\mathcal {G}} ^ {*} = r}. \tag {11}
$$
Here  $\epsilon \in [0,1]$  is a momentum coefficient to make  $s_n$  evolve smoothly during training.
In this section, we provide theoretical justification for EMMA-X and demonstrate the mutual influence between two modules with rigorous interpretation from an EM algorithm perspective. We show that
under dual supervision, minimizing the positive terms in Eq. 7 intrinsically maximizes the objective of a classical clustering algorithm. For simplicity, we assume that each semantic rank has the same number of sentence pairs  $n$  and represents model parameters with  $\Theta$ . In EMMA-X, we model the semantic relation of sentence pair  $(\mathbf{x}_i, \mathbf{y}_k)$  through a joint distribution  $P(\mathbf{x}_i, \mathbf{y}_k)$  with the semantic rank  $c$  as a latent variable. Let  $Q(c)$  be a prior distribution over the possible values of semantic ranks. That is  $\sum_r Q(c = r) = 1$ ,  $Q(c) \geq 0$ . The training goal is to maximize the following likelihood:
$$
\begin{array}{l} \underset {\Theta} {\operatorname {a r g m a x}} \sum_ {\mathbf {x} _ {i} \in \mathcal {X}} \sum_ {\mathbf {y} _ {k} \in \mathcal {Y}} \log P (\mathbf {x} _ {i}, \mathbf {y} _ {k} | \Theta) = \underset {\Theta} {\operatorname {a r g m a x}} \sum_ {\mathbf {x} _ {i} \in \mathcal {X}} \sum_ {\mathbf {y} _ {k} \in \mathcal {Y}} \log \sum_ {r = 1} ^ {N} P (\mathbf {x} _ {i}, \mathbf {y} _ {k}, c = r | \Theta) \tag {12} \\ \geq \operatorname *{argmax}_{\Theta}\sum_{\mathbf{x}_{i}\in \mathcal{X}}\sum_{\mathbf{y}_{k}\in \mathcal{Y}}\sum_{r = 1}^{N}Q(c = r)\log \frac{P(\mathbf{x}_{i},\mathbf{y}_{k},c = r|\Theta)}{Q(c = r)}. \\ \end{array}
$$
E-Step. To make the inequality hold with equality, we have:
$$
Q (c = r) = \frac {P \left(\mathbf {x} _ {i} , \mathbf {y} _ {k} , c = r \mid \Theta\right)}{\sum_ {j = 1} ^ {N} P \left(\mathbf {x} _ {i} , \mathbf {y} _ {k} , c = j \mid \Theta\right)} = P (c = r \mid \mathbf {x} _ {i}, \mathbf {y} _ {k}, \Theta), \tag {13}
$$
which is the posterior probability and is approximated by the prediction from GMM classifier. Since each sentence pair  $(\mathbf{x}_i,\mathbf{y}_k)$  belongs to only one semantic rank, we approximate  $Q(c = r) = \mathbb{I}(c_{\mathcal{G}}^{*} = r)$ , which is a one-hot distribution.
M-Step. We try to maximize the likelihood in Eq. 12 under the semantic rank  $c_{\mathcal{G}}^{*}$ :