SlowGuess/NAACL_2022_OCR huggingface dataset:
MinhQuang Pham†‡*, Josep Crego†, François Yvon‡
†SYSTRAN / 5 rue Feydeau, 75002 Paris, France
firstname_lastname@systrangroup.com
‡Université Paris-Saclay, CNRS, LIMSI, 91405 Orsay, France
firstname.lastname@limsi.fr
Multidomain and multilingual machine translation often rely on parameter sharing strategies, where large portions of the network are meant to capture the commonalities of the tasks at hand, while smaller parts are reserved to model the peculiarities of a language or a domain. In adapter-based approaches, these strategies are hardcoded in the network architecture, independent of the similarities between tasks. In this work, we propose a new method to better take advantage of these similarities, using a latent-variable model. We also develop new techniques to train this model end-to-end and report experimental results showing that the learned patterns are both meaningful and yield improved translation performance without any increase of the model size.
Multidomain and multilingual machine translation aim to develop one single model to perform translation for multiple domains and multiple language pairs, respectively. These paradigms are motivated by the compactness of the resulting translation system (Chu and Dabre, 2018; Dabre et al., 2020), the hypothetical positive knowledge transfer between similar domains (Pham et al., 2021) or between languages in the same family (Tan et al., 2019). However, having all the tasks use exactly the same model parameters can cause negative interference between unrelated tasks (Conneau et al., 2020; Wang et al., 2020b). Hence, the recent development of approaches relying on a partial sharing of the parameters, e.g. using adapter layers as studied in (Houlsby et al., 2019; Bapna and Firat, 2019; Pham et al., 2020; Philip et al., 2020). If these techniques have proven effective for building strong
belines, they fail to fully take advantage of the similarities that exist between domains and tasks, as documented eg. in (Pham et al., 2021). This is because the partition of the parameter space between generic or task-specific subparts, and their allocation to each task, is hard-coded in the network, irrespective of the actual commonalities and differences in the data space.
In this work, we study and develop a new method, multi-task group dropout, aimed to take into account the similarity between tasks in a more effective way, by learning the network organization from the data. To this end, we introduce a set of latent variables in the model, to account for the unseen association between tasks and regions of the representation space and show how training can still be performed end-to-end using a variational surrogate of the log-likelihood loss function. Our experiments with multilingual and multidomain machine translation confirm that this method can automatically detect similarities in the data, meaning that related tasks use the same subparts of the network. Our results also show that this method is comparable to using adapter layers in a number of empirical comparisons; however, contrarily to adapters, these performance are obtained without any increase of the model size. Our contributions are primarily methodological and can be summarized as follows:
1. We introduce a novel, sound mathematical formulation to the problem of jointly learning task-dependent sub-networks and the parameters of the underlying models using variational probabilistic modeling techniques;
2. We present algorithms to train this model end-to-end with very little extra parameters;
3. We report, using an extensive set of experiments, gains for multidomain MT and very low-resourced languages in multilingual MT;
4. We study how this method can actually exploit the similarities between tasks to learn
* Now Research Scientist at Zoom Video Communications
1We will refer to these two situations as 'multi-task MT' and refer to individual domains and languages as 'tasks'.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2469 - 2481
Figure 1: Latent group dropout. The set of nodes in each layer is divided into equal-sized groups. For each task, we only keep a fixed number of active groups of nodes and nullify all the other nodes.
interpretable sub-networks.
2 Multi-task group dropout
2.1 Network architecture, groups and layers
Many architectures for multitask learning are based on a matching of subset of model parameters with tasks. Given the task and the input instance, only a subpart of the network will be involved in the computation of the output value, based on a predefined association between subnetworks and tasks. The adapter architecture of (Bapna and First, 2019) illustrates this strategy, where a task-dependent set of layers is activated for each task.
In our approach, we also require to know the task  $d \in [0 \dots n_d - 1]$  for each training and test instance. The structure of our Transformer networks (Vaswani et al., 2017) is however based on the notion of groups of nodes in the computation graph. At the input of each Transformer layer  $l \in [1 \dots L]$ , we partition all input state vectors into  $n_p$  groups of nodes, and zero-out a task-dependent subset of these groups. The assignment of tasks to groups will be learned from the data, under the constraint that each task only activates exactly  $k$  groups of active nodes, while the all the other values are nullified, akin to a dropout process (see Figure 1). Formally, a group dropout mask  $m_l^d$  is a  $n_p$ -dimensional binary vector containing exactly  $k$  ones: group  $p \in [0, \dots, n_p - 1]$  is retained for task  $d$  if  $m_l^d(p) = 1$  and is dropped if  $m_l^d(p) = 0$ . We denote  $\Delta_k^{n_p} = \{m \in \{0, 1\}^{n_p}$  such that  $|m|_{L_1} = k\}$  the set of all admissible masks, with  $|m|_{L_1}$  the  $L_1$  norm of vector  $m$ ;  $\# \Delta_k^{n_p}$  is the cardinal of  $\Delta^{n_p}$ .
Given  $m_l^d$ , the mask  $r_l^d$  for task  $d$  in layer  $l$  is
then derived as:
$$
r _ {l} ^ {d} (i) = m _ {l} ^ {d} (p) \text {i f} p \times \frac {d _ {k}}{n _ {p}} \leqslant i <   (p + 1) \times \frac {d _ {k}}{n _ {p}},
$$
where  $d_{k}$  is the dimension of the hidden state. The propagation of information within the network then depends on the current task value as follows:
$$
\begin{array}{l} \forall l \in [ 0, \dots , L - 1 ]: \tilde {h} ^ {l} = h ^ {l} \odot r _ {l} ^ {d}, \\ h ^ {l + 1} = \operatorname {L A Y E R} ^ {l + 1} (\tilde {h} ^ {l}), \\ \end{array}
$$
where  $\mathrm{LAYER}^l()$  represents all the computations in Transformer layer  $l$ ,  $\odot$  is element-wise product. It is applied at all positions of each layer in the encoder and in the decoder.
2.2 Training with latent dropout masks
Assuming standard notation for our translation model  $P(y|x,d;\theta)$  where  $x, y$  and  $\theta$  respectively refer to the input, output, and parameter vector, the latent variables  $m_{l}^{d}, l \in [0,\dots,L], d \in [0,\dots,n_{d}-1]$  are introduced as follows. We chose the prior distribution for  $m_{l}^{d}$  as the uniform distribution over  $\Delta_{k}^{n_{p}}$ :  $P(m_{l}^{d}|x,d;\theta) = \mathrm{Unif}(\Delta_{k}^{n_{p}})$ ; variables for each layer are independent and collectively referred to as  $m^{d}$ . For any (variational) distribution  $Q(m^{1}\dots m^{n_{d}};\Phi)$  with parameters  $\Phi = \{\phi_{l}^{1},\dots,\phi_{L}^{n_{d}}\}$ , it is well-known that the log-likelihood is lower-bounded by the so-called ELBO function (hereafter denoted  $\ell$ ), made of a summation of two terms: the distortion  $D$  and the rate  $R$  defined as follows:
$$
\begin{array}{l} \log P (y | x, d; \theta) \geq \ell (x, y, d; \theta , \Phi) \\ \ell (x, y, d; \theta , \Phi) = D (x, y, d; \theta , \Phi) - R (x, y, d; \theta , \Phi) \tag {1} \\ \end{array}
$$
$$
D (x, y, d; \theta , \phi) = \mathbb {E} _ {m ^ {d} \sim Q (m ^ {d} | d, \Phi)} \log P (y | m ^ {d}, x, d; \theta)
$$
$$
R (x, y, d; \theta , \phi) = \mathrm {K L} (Q (m ^ {d} | d, \Phi) | | P (m ^ {d} | x, d; \theta)),
$$
where KL is the Kullback-Leibler divergence. We use  $-\ell (x,y,d;\theta ,\Phi)$  as our surrogate training loss, as a tractable approximation of the likelihood, and try to minimize this function in  $\theta$  and  $\Phi$
The variational distribution  $Q$  of  $m^d$  is defined independently on a layerwise basis; this means that each layer only involves a subset  $\Phi_l^d$  of variational parameters.  $Q$  is computed as follows:
$$
\begin{array}{l} \operatorname {I n d} ^ {d} = \left\{i _ {1}, \dots , i _ {k} \right\} \sim \operatorname {S R S} \left(\operatorname {s o f t m a x} \left(\Phi_ {l} ^ {d}\right), k\right) \\ m _ {l} ^ {d} (i) = \mathbb {I} (i \in \operatorname {I n d} ^ {d}), \\ \end{array}
$$
where  $\mathrm{SRS}(\pi ,k)$  denotes the process of sampling  $k$  times without replacement from the distribution  $\pi$ , and  $\mathbb{I}$  is the indicator function. This modeling choice for the latent vector  $m_l^d$  is motivated by the Gumbel Top-K trick of Kool et al. (2019) that we
use below. Given our choices for the prior and the variational distributions, the two terms in Eq. (1) can be computed as:
$$
\begin{array}{l} D (\dots) = \mathbb {E} _ {m ^ {d} \sim Q (m ^ {d} | d; \Phi)} \log P (y | m ^ {d}, x, d, \theta) \\ = \mathbb {E} _ {g ^ {d} \sim \mathrm {i . i . d}} G (0, 1) \left[ \log P \left(y | \tilde {m} ^ {d}, x, d, \theta ,\right) \right] \\ \end{array}
$$
where the generation process  $G(0,1)$  is a product of independent Gumbel distributions, yielding:
$$
\forall d, g ^ {d} = \left[ g _ {1} ^ {d}, \dots , g _ {L} ^ {d} \right], \text {w i t h} g _ {l} ^ {d} \in \mathbb {R} ^ {n _ {p}}
$$
$$
\forall p, g _ {l} ^ {d} (p) \stackrel {\mathrm {i . i . d}} {\sim} \operatorname {G u m b e l} (0, 1)
$$
$$
\begin{array}{l} \operatorname {I n d} ^ {d} = \left\{i _ {1}, \dots , i _ {k} \right\} = \operatorname {T o p - k} \left\{g _ {l} ^ {d} (0) + \Phi_ {l} ^ {d} (0), \dots , \right. \\ \left. g _ {l} ^ {d} \left(n _ {p - 1}\right) + \Phi_ {l} ^ {d} \left(n _ {p - 1}\right) \right\} \tag {2} \\ \end{array}
$$
$$
\tilde {m} _ {l} ^ {d} (p) = \mathbb {I} (p \in I n d ^ {d}).
$$
For the second term, the derivation is the following:
$$
\begin{array}{l} R = \operatorname {K L} (Q (m ^ {d} | d, \Phi) | | P (m ^ {d} | x, d; \theta)), \\ = - \sum_ {l = 1} ^ {L} \left(\mathbb {H} \left[ Q \left(m _ {l} ^ {d} | d, \Phi\right) \right] - \log \left(\# \Delta_ {k} ^ {n _ {p}}\right)\right) \\ = - \sum_ {l = 1} ^ {L} \left(\mathbb {H} \left[ Q \left(i _ {1}, \dots , i _ {k} \mid d, \Phi\right) \right] - \log \left(\# \Delta_ {k} ^ {n _ {p}}\right)\right) \\ \leqslant - \sum_ {l = 1} ^ {L} \left(\mathbb {H} \left[ Q \left(i _ {1} | d, \Phi_ {l} ^ {d}\right) \right] - \log \left(\# \Delta_ {k} ^ {n _ {p}}\right)\right). \tag {3} \\ \end{array}
$$
We prove inequality (3) in Appendix B. This inequality shows that an upperbound of  $R$  is  $\sum_{l=1}^{L} (\log(\# \Delta_k^{n_p}) - \mathbb{H}(\text{softmax}(\Phi_l^d)))$  since  $i_1 | \Phi_l^d \sim \text{softmax}(\Phi_l^d)$ . During training, we thus maximize a sum over layers of the entropy  $\mathbb{H}(\text{softmax}(\Phi_l^d))$  which performs a regularization over the parameters  $\Phi^d$  of the variational distribution.
Thanks to the Gumbel Top-K trick, we can move the parameters  $\Phi$  into the objective function and get rid of policy gradients, which have been reported to be very unstable (Kingma and Welling, 2014). However, the operator Top-k, which serves to define  $\tilde{m}_l^d$  in Equation (2), is not differentiable. Therefore, we approximate this function by the Soft-Top-K function defined as follows:
$$
\hat {m} _ {l} ^ {d} (\tau) = \operatorname {a r g m i n} - \left(g _ {l} ^ {d} + \Phi_ {l} ^ {d}\right) ^ {T}. m - \tau H _ {b} (m)
$$
$$
\begin{array}{l} 0 \leqslant m _ {i} \leqslant 1 \\ \forall 0 \leqslant i \leqslant n _ {d} - 1 \\ 1 ^ {T}. m = k \\ \end{array}
$$
$$
H _ {b} (m) = - \sum_ {i} m _ {i} \log (m _ {i}) + (1 - m _ {i}) \log (1 - m _ {i}).
$$
In Appendix A, we prove that  $\lim_{\tau \to 0} \hat{m}_l^d(\tau) = \tilde{m}_l^d$ . Furthermore, we also provide the computation of  $\hat{m}_l^d(\tau)$  and prove that  $\hat{m}_l^d(\tau)$  is a differentiable
function w.r.t  $\Phi_l^d$  and that its gradients can be computed using the implicit differentiation theorem. During training, we approximate  $\tilde{m}_l^d$  by  $\hat{m}_l^d (\tau)$  by gradually decaying the hyper-parameter  $\pmb{\tau}$  to 0.2. The gradient of  $D$  w.r.t  $\Phi_l^d$  is computed using the chain rule as follows:
$$
\frac {\partial D}{\partial \Phi_ {l} ^ {d}} = \frac {\partial D}{\partial \hat {m} _ {l} ^ {d} (\tau)} \times \frac {\partial \hat {m} _ {l} ^ {d} (\tau)}{\partial \Phi_ {l} ^ {d}}
$$
The gradient  $\frac{\partial D}{\partial\hat{m}_l^d(\tau)}$  is computed via autograd algorithm while  $\frac{\partial\hat{m}_l^d(\tau)}{\partial\Phi_l^d}$  is computed via implicit differentiation, as explained in Appendix A.
We jointly train the Transformer parameters  $\theta$  and the parameters of the variational distribution  $\Phi$  using the following multi-task loss.
$$
\mathcal {L} (\theta , \Phi) = \sum_ {d = 1} ^ {n _ {d}} \mathbb {E} _ {x \sim \mathscr {D} _ {s} ^ {d}, y \sim M T ^ {d} (x)} \left[ - \ell (x, y, d; \theta , \Phi) \right]
$$
in which  $\mathcal{D}_s^d$  is distribution of task  $d$  over the input space  $\Omega_s^d$ ;  $\mathrm{MT}^d:\Omega_s^d\to \Omega_t^d$  is the translation function for task  $d$ , which our multi-task model needs to learn;  $-\ell (x,y,d,\theta ,\Phi)$  is the ELBO loss, defined in Equation 1.
Finally, during inference, we define the dropout mask for layer  $l$  and task  $d$  as follows:
$$
\begin{array}{l} I n d _ {l} ^ {d} = \operatorname {T o p - k} \left(\Phi_ {l} ^ {d}\right) \\ m _ {l} ^ {d} = \mathbb {I} (i \in I n d _ {l} ^ {d}) \\ \end{array}
$$
meaning that we simply pick the  $k$  most likely parameter groups for the task at hand, and define the state dropout mask accordingly.
3 Experimental settings
3.1 System design and configuration
3.1.1 Multidomain translation systems
Our systems for the multidomain experiments are designed as follows:
- Transformer: The embedding dimension for both encoder and decoder is set as 512, and the feedforward dimension is 2048; the multi-head attention mechanism contains 8 heads; 6 layers in the encoder; 6 layers in the decoder.
- Adapter-based Transformer: the intermediate feedforward dimension is set to 2048, as in Pham et al. (2021).
- Transformer using Latent multi-task group dropout (LaMGD Transformers): There is no change in the architecture. We group the 512 nodes in each layer into 16 groups of 32 consecutive nodes. For each domain, only 12 out of the 16 groups are selected. The number
of parameters of the variational distribution is  $L \times k \times L \times n_d$ , which is negligible in comparison to the size of the Transformer model.
- Transformer using heuristic multi-task group dropout (HMGD Transformer): we share 320 nodes for every task, and reserve 32 nodes for each task (totalling  $320 + 32 * 6 = 512$  nodes).
We set the dropout probability to 0.1. We train the multidomain Transformer model for 200k iterations with a batch size of 12k tokens using 4 V100 GPUs. The convergence of the standard Transformer is before 200K as its validation curve became flat near the 200K-th iteration. The LaMGD Transformer converged after 300k iterations with the same batch size. The convergence of LaMGD is controlled by its validation curve. Finally, we plug adapters to the multidomain Transformer model and finetune them for 25k iterations using the same batch size as the baseline.
3.1.2 Multilingual translation systems
The systems used in our multilingual experiments are implemented as follows:
- Multilingual Transformer: the embedding dimension for both encoder and decoder is set as 512, and the feedforward dimension is 1024, each multi-head attentions contains 8 heads as in (Wang et al., 2020a).
- Adapter based Transformer: the intermediate feedforward dimension is set as 128. We follow here the parameter setting of (Gong et al., 2021a).
- LaMGD Transformer: There is no change in the architecture. We group 512 nodes in each layer into 16 groups of 32 consecutive nodes. For each language, we select 12 groups.
We set the dropout probability to 0.3. We train the multilingual Transformer model for 40k iterations with a batch size of 9600 tokens on 16 V100 GPUs as in Gong et al. (2021a). We train LaMGD Transformer for 50k iterations with the same batch size. The convergence of the models are controlled via their validation curves. Finally, we finetune the language-specific Adapters for 5k iterations.
All the translation systems are implemented with OpenNMT-tf $^2$  (Klein et al., 2017).
3.1.3 Hyper-parameters
We choose  $n_d = 16$  so that the size of the dropout group is neither too small nor too large. The second important hyper-parameter in LaMGD is the number
of selected groups in each layer,  $k$ , which we set to 12 in every experiments. By retaining 12/16 groups, we share on average  $75\%$  active groups between two domains or languages. This design ensures that the percentage of sharing is in the same ballpark as what we obtain with adapter modules. In our future work, we intend to analyze how these choices affect the final performance of the model.
The temperature parameter  $\tau$  for the Soft-Top-K operator is gradually decreased from 0.5 to 0.2 according to the following policy:
$$
\tau = \min  \{0. 2, 0. 5 * \exp^ {- r * s t e p} \},
$$
in which  $r = 0.0001$ . While Gong et al. (2021b,a) fixed  $\tau$  to be 0.2, we select an anneal policy for  $\tau$  proposed by previous studies (Jang et al., 2017). Finally, we set the weight of the entropy term to 0.0001 in the training loss in every experiments.
3.1.4 Latent variables initialization
We initialize the distribution of the latent variables uniformly. More precisely, we set  $\Phi_l^d$ , which generates the probability of the masks via the softmax activation function, to  $0^{n_d}$ .
3.2 Datasets and metrics
3.2.1 Multidomain translation
We use the same data as in the recent work of Pham et al. (2021) on multidomain translation. The datasets  $^{3}$  for the multidomain translation experiments are detailed in Table 1. For each domain, the size of the dev set and the test set is  $1\mathrm{K}$ .
3.2.2 Multilingual translation
We evaluate our model on both one-to-many (O2M) and many-to-one (M2O) translation tasks borrowing the multilingual translation datasets from past studies. More precisely, we used:
- TED8-Related. Following the setting of Wang et al. (2020a), we use a subset of translations from Qi et al. (2018) between English and eight related languages.
- TED8-Diverse. The dataset consists of parallel sentences between English and eight diverse languages as in Wang et al. (2020a).
The languages used in the multilingual experiments are as follows (see statistics in Table 2):
- Diverse set: bos (Bosnian), Bulgarian (bul), French (fra), ell (Greek), hin (Hindi), Korean (kor) mkd (Macedonian), mar (Marathi);
3See
https://github.com/qmpham/
experiments/tree/main/tacl20
2https://github.com/OpenNMT/OpenNMT-tf
MED	LAW	BANK	IT	TALK	REL
# lines	2609 (0.68)	501 (0.13)	190 (0.05)	270 (0.07)	160 (0.04)	130 (0.03)
# tokens	133 / 154	17.1 / 19.6	6.3 / 7.3	3.6 / 4.6	3.6 / 4.0	3.2 / 3.4
# types	771 / 720	52.7 / 63.1	92.3 / 94.7	75.8 / 91.4	61.5 / 73.3	22.4 / 10.5
#uniq	700 / 640	20.2 / 23.7	42.9 / 40.1	44.7 / 55.7	20.7 / 25.6	7.1 / 2.1
Table 1: Corpora statistics: number of parallel lines  $(\times 10^{3})$  and proportion in the basic domain mixture (which does not include the NEWS domain), number of tokens in English and French  $(\times 10^{6})$ , number of types in English and French  $(\times 10^{3})$ , number of types that only appear in a given domain  $(\times 10^{3})$ .
- Related set: Azerbaijiani (aze), Belarusian (bel), Czech (ces), Galician (glg), Portuguese (por), Russian (rus), Slovak (slk), Turkish (tur).
For all experiments, we report the BLEU score of Papineni et al. (2002) computed with SacreBleu (Post, 2018). Statistical significance is computed with compare-mt $^4$  (Neubig et al., 2019). We report significant differences at the level of  $p = 0.05$ .
4 Results and analyses
4.1 Multidomain translation
For these experiments, our main results are in Table 3, where we observe that the LaMGD Transformer achieves a significant improvement (+2.78) over the generic Transformer system with zero extra parameters. Moreover, LaMGD Transformer achieves performance that are equivalent on average to that of the Adapter sytems, which is finetuned and contains approximately 25M additional parameters per domain. Variational mask learned from data by LaMGD also outperforms heuristic dropout mask HMGD by 0.5 in average.
4.1.1 Fuzzy domain separation
For this experiment, we reuse proposal of Pham et al. (2021), who measure the efficiency of a multidomain NMT system exploiting the proximity between domains. It uses the same data as in the previous experiment; however, the domain LAW is now randomly split into two pseudo-domains LAW1 and LAW2 of equal size. A truly multidomain system should be able to automatically detect the proximity between LAW1 and LAW2, and there should be no significant difference between the performance of a system trained with the six original domains (including LAW) or with the seven domains (including LAW by LAW1 and LAW2). Pham et al. (2021) reported a large gap between the two settings when using residual adapters. We replicated this setting
and report the results obtained with the LaMGD Transformer system in Table 4.
The results in Table 4 show a performance decrease for the adapter-based system when training with two pseudo-domains  $\mathsf{LAW}_1$  and  $\mathsf{LAW}_2$ . In contrast, the LaMGD model obtains very stable results. In Section 4.3, we show that our algorithm in fact computes the same sub-network for  $\mathsf{LAW}_1$  and  $\mathsf{LAW}_2$ , that allows a full sharing of information between these two pseudo-domains.
4.2 Multilingual translation
Results for the multilingual experiments are in Table 5. The LaMGD Transformer achieves an improvement of 0.42, 0.33, 0.32 in average over the multilingual Transformer in the O2M-related, M2O-related, M2O-diverse conditions, respectively. Significant gains are observed for languages BEL, GLG (both direction), HIN and BOS (O2M direction) which are very low-resource languages in our sets. However, LaMGD Transformer is outperformed by the multilingual Transformer and language Adapters for the O2M-diverse condition.
4.3 Similarity between dropping masks
This section compares the sub-networks learnt for each domain or language pair by computing the average similarity between the corresponding dropout masks concatenated for all the layers of the underlying model. For the multidomain experiment, we analyze the case of pseud-domain separation reported in Section 4.1.1 in Figure 2a. We see that the sub-networks for  $\mathsf{L}\mathsf{A}\mathsf{W}_1$  and  $\mathsf{L}\mathsf{A}\mathsf{W}_2$  are identical, yielding a full sharing between the corresponding training sets. Furthermore, we observe a large distance between REL and the other domains, which is expected given that REL is quite distinct from the other domains. REL only share around  $75\%$  its active groups with other domains, as would be obtained by chance in our setting (see Section 3.1.3). In Figure 4, we visualize the domains using their dropping masks concatenated and mapped to a 2d
4https://github.com/neulab/compare-mt
LANG	Related	Diverse
TRAIN	DEV	TEST	LANG	TRAIN	DEV	TEST
Azerbaijani	5.94k	671	903	Bosnian	5.64k	474	463
Belarusian	4.51k	248	664	Marathi	9.84k	767	1090
Galician	10.0k	682	1007	Hindi	18.79k	854	1243
Slovak	61.5k	2271	2445	Macedonian	25.33k	640	438
Turkish	182k	4045	5029	Greek	134k	3344	4433
Russian	208k	4814	5483	Bulgarian	174k	4082	5060
Portuguese	185k	4035	4855	French	192k	4320	4866
Czech	103k	3462	3831	Korean	205k	4441	5637
Table 2: Data Statistics of TED8 Datasets
Model / Domain	MED	LAW	BANK	TALK	IT	REL	AVG
Transformer [65m]	40.3	59.5	49.8	36.4	49.0	80.0	52.5
HMGD Transformer [+0m]	40.4	60.4	51.9	38.7	50.8	86.80	54.8
Adapter [+151m]	39.5	61.0	53.1	37.5	49.6	91.0	55.3
LaMGD Transformer [+0m]	40.3	60.4	52.4	39.0	52.4	87.5	55.3