Macedon: Minimizing Representation Coding Rate Reduction for Cross-Lingual Natural Language Understanding
Authors: Haoyu Wang
Yaqing Wang
Huaxiu Yao
Jing Gao
Venue: EMNLP 2023 Findings (2023)
Keywords: cross-lingual
rate reduction

Abstract: Cross-lingual natural language understanding(NLU) is one of the fundamental tasks of NLP. The goal is to learn a model which can generalize well on both high-resource and low-resource language data. Recent pre-trained multilingual language models, e.g., multilingual BERT, XLM, have shown impressive performance on cross-lingual NLU tasks. However, such promising results request the use of sufficient training data, which is a difficult condition to satisfy for low-resource language.
When the data is limited in those low resource languages, the accuracy of existing models will drop. In light of this challenge, we investigate the important task of how to train the cross-lingual model with abundant high-source language data and limited low-resource language data. Existing methods typically learn language-agnostic representation via adversarial training and mutual information estimation. Existing approaches may suffer When data is very limited (e.g., low-resource language) because it is challenging to estimate data distribution accurately. To tackle this issue, we propose a conceptually innovative approach to remove language-associated information via \textbf{m}inimizing represent\textbf{a}tion \textbf{c}oding rate r\textbf{ed}ucti\textbf{on}(Macedon). Specifically, Macedon avoids using extra codes to encode language-related information, which is measured by the rate-distortion function. To validate the effectiveness of Macedon, we conduct extensive experiments on three tasks, including paraphrase identification, natural language inference, and query advertisement matching. The
experiment results show that the proposed Macedon outperforms state-of-the-art cross-lingual NLU approaches.

Macedon:
Minimizing Representation Coding Rate Reduction for
Cross-Lingual Natural Language Understanding
Haoyu Wang', Yaqing Wang", Huaxiu Yao", and Jing Gao
Purruee University, West Lafayeute, IN, USA 'Googee Research, New York, NY, USA *UNC-Chapel Hill, Chapel Hill, NC, USA i jingngga)),,ingggaaaa))o)))0) lyagingaang@google. com, muuxiiuccs.unc.duu edu
edu,
Abstract
million first language speakers counted in 2022?. Therefore, most languages (i.e., low-resource lan- guages) have limited data when considered in nat- ural language understanding (NLU) tasks (Joshi et al., 2020), which covers variety of sub-tasks dealing with machine reading comprehension such as sentiment classification, named entity recogni- tion, and part-of-speech tagging. As result, mod- els trained on such limited data may have poor performance. An immediate solution one can think of to collect more labeled data for low-resource languages. However, could be infcasible due to the prohibitively intensive human annotation pro- cess that is both time-consuming and costly. Thus, cross-lingual NLU has gained increasing interests in recent years (Nooralahzadeh et al., 2020; Wang et al., 2021b; Mao al., 2021; Wu and Dredze, 2019), in which the model is trained with abundant high-resource language data and limited annotated low-resource language data to make the model gen- eralize well for all languages.
Cross posss-iinuul natural language understand- ing (NLU) is one of the fundamental tasks of NLP. The goal is to learn a model which can generalize well on both high-resource and low-resource language data. Recent pre-trained multilingual language models, e.g., multilin- gual BERT, XLM, have shown impressive per- formance cross-lingual NLU tasks. How- ever, such promising results request the use of sufficient training data, which diffi- cult condition to satisfy for low-resource lan- guage. When the data is limited in those low resource languages, the accuracy of existing models will drop. In light of this challenge, we investigate the important task of how to train the cross-lingual model with abundant high-source language data and limited low- resource language data. Existing methods typ- ically learn agnouage-agnostic representation via adversarial training and mutual information estimation. Existing approaches may suffer When data is very limited (c.g., low-resource language) because it challenging to estimate data distribution accurately. To tackle this is- sue, we propose conceptually innovative ap- proach to remove aaaguage-associatee informa- tion via minimizing representation coding rate reduction (Macedon). Specifically, Macedon avoids using extra codes encode language- related information, which is measured by the rate-distortion function. To validate the effec- tiveness of Macedon, we conduct extensive ex- periments on three tasks, including paraphrase identification, natural language inference, and query advertisement matching. The experiment results show that the proposed Macedon out- performs state-of-the-art cross-lingual NL.U ap- proaches.
Recently, pre-trained multilingual language mod- els (MLMs), such multilingual BERT (Devlin et al., 2018) and XLM (Lample and Conneau, 2019), have shown promising results in transfer learning across different languages in multiple downstream tasks (Hu et al., 2020; Liang et al., 2020). In MLMs, different languages are able to share one vocabulary table, enabling the model to learn information that is shared among those languages. Typical MLMs, however, require paral- lel corpora in order to fine-tune downstream tasks, which can be difficult to achieve when there exist low-resource languages. Additionally, it may be difficult to align different languages and yield satis- fying transfer performance when there are huge dif- ferences between low-resource and high-resource languages in morphology, syntax, or semantics (Ah- mad et al., 2019; Hu et al., 2020).
1 Introduction
Globally, there are about 7.106 living languages spoken'. but only 91 languages have least 10
Ptt /tetps://en..e httpn////ee..wiiediiiaa... org/wiki /t/iti/ittff...... crg.uugges_bbnuumbrrro www.bberrooo_atiiieeeeeeeeeeeee wtt__nntttveeeeeeeeee nataeeesspeakers
hhttps: //www. wtwnologue.com

---

Prior studies have shown that aligning differ- ent language representations is effective for cross- lingual transfer. The key idea is to enforce sim- ilar representations for sentences that share simi- lar semantics across languages. Existing methods fall into two categories. The first line of methods has to be based on parallel corpora. The method aligns different languages in the latent space in accordance with parallel sentences (Yang et al., 2021; Gritta and lacobacci, 2021; Cao al., 2020; Pan et al., 2020; Dou and Neubig, 2021). For lan- guages with limited resources, however, parallel corpora are extremely difficult to obtain. Therefore, another line of approaches leverages adversarial training and employs Generative Adversarial Net- works (GAN) (Goodfellow et al., 2014) to learn language-indepentn independent representations (Chen et al., 2021, 2018b,a; Keung et al., 2019; Lee and Lee. 2019; Wang et al., 2021a; Zou et al., 2018). It is worthwhile to note that adversarial training-based methods rely on an accurate estimation of data dis- tributions, which may be infeasible with limited labeled examples in low-resource languages.
as follows:
We propose a novel method for cross-lings Prossllingual NLU tasks. To the best of our knowledge, this is the first attempt to leverage coding rate function in cross-lingual NLU tasks.
We provide a new perspective of modeling cross- lingual representations. We leverage data com- pression to mix different language representa- tions in the latent space. It is more effective way to align different languages compared with adversarial training under limited data.
We conduct extensive experiments on three pub- lic datasets. Results show that the proposed Macedon outperforms the state-of-the-ar base- lines. Furthermore, experimental results show that the proposed Macedon is able reduce the distance between different language representa- tion distributions.
2
Related Work
2.1 Cross-lingual NLU
In this paper, we solve the cross-lingual NLU tasks from another perspective. The goal is still to align different languages. However, different from adversarial training, we do not need align the distributions of different language representa- tions strictly. We propose to relax the condition and simply force different language representations to overlap in the representation space, To achieve this goal, we propose Macedon which minimizes representation coding rate reduction. Macedon is built upon the coding rate function, which mea- sures the minimal average number of bits to encode set of vectors (Ma et al., 2007; Yu et al., 2020). Specifically, we measure two kinds of coding rates: the coding rate of all language representations and the average coding rate of each language repre- sentation. We encourage the two coding rates to be almost the same value. Intuitively, the learned representation does not use extra codes to encrypt language information, and can be considered as language irrelevant approximately. In addition, the coding rate function uses empirical covariance ma- trix to compute rate-distortion function directly, which likely to be singular and highly biased. Macedon, we employ a simple but effective method to modify the empirical covariance matrix to make it positive definite and reduce its bias.
Cross-lingual NLU tasks has been studied from two major perspectives: 1) multilingual word em- bedding (Doval et al., 2019; Ormazabal et al., 2019; Ruder et al., 2019; Shi et al., 2015; Søgaard et al., 2019; Wada and Iwata, 2018; Xu et al., 2018; Cao et al., 2020; Dou and Neubig, 2021), and 2) pre- trained multilingual language models, such as mul- tilingual BERT (Devlin et al., 2018), and XLM- RoBERTa (Lample and Conneau, 2019). Some studics (Devlin et al., 2018; Wu and Dredze, 2019) have shown pre- pre-teri multilingual language mod- els (MLM) have much better performance than traditional word embedding methods for multiple NLU tasks. Therefore, recent works (Hu et al., 2020; Liang et al., 2020) usually fine-tune MLMs for downstream NLU tasks. Specifically, works based on MLMs can be roughly categorised as data augmentation-bastt and language alignment- based methods.
For the data augmentation-based methods, they are to solve the key challenge of cross-lingual NLU: lack of low-resource language data. For exam- plc, Code-switch (Qin et al., 2020) proposes to randomly replace the phrase of high-resource lan- guages corresponding low-resource language phrase; Dong et al. (2021) augments the data via reordering, Ahmad et al. (2021) proposes the aug- mentation method based on syntax; and Bari et al.
The contributions of the paper are summarized

---

Cross entropy
Class
Langage
Class
Cargus
CE(D,))
Class 3
[=1
Multilingual BERT
Langage:2
Corpas
langage
X3
Coding Rate Reduction R(X,c) (x.e))
langage
language
Langage: Corpus
Figure I: The framework of proposed Macedon.
(2020) generates augmented data directly from MLMs based on the vicinity distribution of high- resource and low-resource language data samples. They are effective to improve cross-lingual NLU performance. However, they usually need to draw additional resources such bilingual dictionary. Another category is to align different languages. Some works attempt to align the contextual word embedding of different languages, such as learning projection transformations (Aldarmaki and Diab, 2019; Wu et al., 2019) and forcing model into hav- ing similar predictions for parallel sentences (Yang al., 2021; Gritta and lacobacci, 2021; Cao al.. 2020; Pan al., 2020; Dou and Neubig, 2021). And there are also a lot works exploring adver- sarial training to align different languages (Chen et al., 2021, 2018b,at 8b,a; Keung et al., 2019; Lee and Lee, 2019; Wang et al., 2021a: Zou et al., 2018). Their motivation is to learn language-agnostic rep- resentation to make model focus more under- stand text semantics and generalize better low- resource languages. However, this line of work usually needs parallel corpus or large amount low- resource language data, which are expensive to obtain and may be not always available.
deep neural networks. Recently, Yu et al. (2020) shows maximizing coding rate reduction can be considered as an alternative to traditional objective function such cross entropy. Li et al. (2022) and Chowdhury and Chaturvedi (2022) also apply the maximizing coding rate reduction method to mani- fold clustering and fairness representation respec- tively. Different from previous work, our proposed Macedon is to minimize coding rate reduction function to learn language-agnostie representation.
Preliminaries
3.1 Rate Distortion for Finite Data
Rate distortion is a concept for lossy compression used information theory. Given learned represen- tation X = [I],,.., "it,....,,,, Im E Bdxm and a precision 2, where d is the representation dimension and m is number of data samples, the rate distortion R(X, e) is to measure the minimal number of bi- nary bits to encode X averagely with the precision constraint E|] x [2] < e Yu et al. (2020) and Ma al. (2007) show that if æ; is sampled from a subspace-like distribution, the precise esti- mation (tight bound) of R(x,c)
R(X, log det + d
2.2 Coding Rate-distortion Function for Representation Learning
XXx), (I) me²
Coding rate-distortion (Berger, 2003; Ma et al., 2007) is a concept about data compression from information theory. Recently, it is introduced into representation learning. Chan et (2021) is the first work using coding dittodistottion function as an objective function for designing interpretable
where 1 1XXT is the empirical covariance matrix of X. Intuitively, the R(X,c) proportional the logarithm of the volume of region spanned by X. If X is very concentrated, then the volume of region they span will small, i.. the value R(X,e) e) will be small, and vice versa.

---

3.2 Rate Distortion for Data with Mixed Distribution
the representation space, which can be considered as language irrelevant approximately. The objec- tive function of the proposed Macedon includes two terms: 1) cross entropy loss to classify data accurately, and 2) coding rate reduction loss, which is to make different language representation mixed in the latent space. We introduce the proposed Macedon in the following section in detail.
Generally, the representation X may lie in multi- ple subspaces, e.g. multi-class, which means the representation corresponds mixed distribution, i.e. X Xi T...UUU....... U where X; corresponds the representation in the class We use a set of diagonal matrices II (Ty C Rmxm) j=1 to indicate the probability that sample belongs to X, by its diagonal entry , a]. The I satisfies that [i, i] and Ma et al. (2007) indicate that the rate distortion function for data with such mixed distribution is:
4.3 Coding Rate-Distortion Maximization
To enable the learned representation be transfer- able among different languages. we hope the model can focus on understanding the language-agnooe semantics of text following (Chen et al., 2021, 2018b; 8b; Keung et al., 2019; Wang et al., 202 Zou et al., 2018). In the other word, the sentence representation should preserve semantic informa- tion while removing language-related information. Therefore, propose the following two criteria for the representation learning:
and 2, I I. Yu et al. (2020)
( III
tr (Tl) 2m
d
log de
I
x1I
tr(
Here, we just consider that each example belongs to one category.
The learned representation should contain enough information for the NLU task. It should be informative for the target label y/-
4 Minimizing Representation Coding Rate Reduction
4.1 Problem Formulation
The distribution of representations from different languages should be similar. It to make it more transferable among different languages.
The cross-lingual natural language understand- ing (NLU) aims learn a model ft,; - Y on the training set D = {(t,, y;)]=> which consists of large amount of high-resource language (e.g.. English) data and limited low-resource language data. Here t; is the text of training example i, y/ is the corresponding label, and is the number of training examples. In Macedon, we aim learn model fo() that can generalize well to both high- resource and low-resource languages.
For the first criterion, it corresponds to a stan- dard classification problem and we minimize the cross entropy between the prediction and label. To satisfy the second criterion, we need make the representation of different languages mixed and overlap in the representation space. Usually, adver- sarial training such as Generative Adversarial Net- work (GAN) can be used to learn feature-invariane representation, which has shown their effectiveness on domain adaptation (Tzeng et al., 2017). How- ever, when there are very limited target domain data, it difficult to estimate target domain distri- bution. Therefore, adversarial training and mutual information estimation may be not effective in the cross lingual setting, which shown in the Sec- tion 5.3. In our paper, we propose an alternative method to solve this problem based on rate distor- tion. The key motivation is that the average bit number used to encode all language representation almost equals to the average bit number used encode each language representation respectively. In other words, it nearly does not need to use extra bits to encode (remember) language. To achieve this goal, we control the absolute value of the rate
In the next sections, we use X to denote the rep- resentation of the whole training data, and use X; to denote the representation of the i-th language in the training set. We also denote n, as the number of training data for the i-th language. In the fol- lowing, we first overview the entire framework of Macedon and introduce the detail of rate reduction function and the whole loss function in Section 4.3.
4.2 Overview
The goal of Macedon is to learn an effective cross- lingual model which can generalize well all lan- guages, including high-resource languages and lim- ited low-resource languages. The framework of Macedon is shown in Figure 1. is to align differ- ent languages via encouraging them to overlap in

---

reduction ((X, ((X,c)_X((X e(II) to be smaller than small value 8. Therefore, we loss function can be represented
5 Experiment
In this section, we evaluate the proposed Mace- donwith the goal of answering the following ques- tions:
min
CE((
70
RQ1
How does Macedon perform compared to state-of-the-art baselines?
s.t. ( (
c[II] 1 6,
(2)
RQ2 What is the relationship between the proposed Macedon and adversarial training?
where CE represents the cross entropy loss.
Be-
cause R(X, e) e 2 0 (Yu al., 2020),
RQ3 Is the proposed Macedon effective to align different language representations?
we can rewrite the loss function
RQ4 How does the performance change with re- spect to different coefficients of the proposed Macedon?
CE y1)
s.t. s.t.R(X, R X
cIII= = 8.
(3)
5.1
Datasets and Experiment Settings
5.1.1
Datasets
However, in cross- lingull setting, the empirical covariance matrix in R(X,c, and ( c(IT) may result in large bias because there are very limited samples. Because the representation of multilin- gual BERT is larger than number of low-resource language training data, their empirical covariance matrix may be even not positive definite. To re- duce the bias under such limited data condition, modify the empirical covariance matrix X X
To evaluate the performance of the proposed Mace- don comprchensively, we conduct experiments on three public cross-lingual benchmark datasets, in- cluding PAWS-X (Hu et al., 2020), QADSM (Liang et al., 2020), and XNLI (Hu et al., 2020). The three datasets correspond paraphrase identifica- tion, query advertisement matching, and natural language inference. For the three datasets, we ran- domly sample 100 few-shot data for each language other than English. In XNLI, we add an extra exper- iment using 2.5k data instances as few-shot data for low-resource languages, namely Macedon 2.5k, to compare with the results in (Nooralahzadeh et al., 2020) fairly. We tune hyper-parameters and evalu- the developed models on the pre-split validation and testing set. The statistics datasets are summa- rized in Table 2. PAWS-X X is public release but the Licence information is not presented in the source website QADSM and XNLI are available under Creative Commons Attribution 4.0 International and Attibattion-NonCommeeee 4.0 International Licenses respectively. We follow the licence of datasets for research use.
(1-a)XX- + tx(XX
(4)
and it has two great properties: 1) is a posi- tive definite matrix shown in Proposition 1, and 2) R(X,e) (XX 2 0 shown in Proposi- tion 2, where R(X,c) c) and (x III are modi- fied rate-distortion functions respectively.
Proposition 1. The matrix (1 x)xx a (r(XXT) I is positive definite.
The proof shown in the appendix.
Proposition 2. A(X,e) (x <(II 2 0. The equality holds when (1 a) x,xX +a X a)
5.1.2 Baselines
X ne + a
X x I.
...=(1-
We adopt the following state-of-the-art methods as baselines:
The proof shown in the appendix. Therefore, we can rewrite the objective function an unconstrained optimization problem:
mbert (Devlin et al., 2018) is the multi-linguu version of BERT (Devlin al., 2018), which uses the same architecture of BERT and is trained on Wikipedia corpora with 104 widely used lan- guages. It is one of the most effective methods
CE 9.) ( (x,
htttps: thw.:///gttttu..te wwwm///ooleeeeeeeeoooooo weweseearccddattssseeeeee paws/trree/msster/pawss
where y is a positive hyper-parameter.

---

Dutaset
Method
bg
de
Fr
hi
th
2b
ja
ko
AVG
mbeet"
94.0
87.4
87.0
77.0
73.0
69.6
82.0
ibberteTD
94,1
86.4
88.3
87.1
79.7
75.8
74.9
83.8
Syn. ?
94.0
87.8
85.9
89.1
80.7
75.8
76.3
84.3
X-MAMI.
87.6
87.3
80.3
762
76.5
miber udy
94,1
86.5
89.2
88,0
79.9
75.4
76.5
84.2
Muceloi-NCMM
93.8
87.0
89.3
85.5
81.3
71.1
71.00
84.9
Macedon
93.9
87.0
89.2
88.8
81.3
78.3
77.8
85.2
mbbiteWDD
68.0
60.9
62.8
63.9
Syn."
68.4
60.8
64.0
64.4
QADSM
X-MAML
68.2
63.7
64.4
65.4
ibeet-adv
67.7
59.1
64.6
63.8
Macedon-NoCM
67.3
63.8
63.0
64.7
Macodon
68.6
64.0
64,9
65.8
Inbert"
80.8
64.3
68.0
73.5
73.4
58.9
49.7
54.1
60.9
67.8
InberteWID
81.0
66.5
"00.4
72.4
68.0
74.6
14.2
64.1
10.0
51.3
58.3
63.3
61.4
314
71.7
67.9
Syn. 9
81.6
65.4
09.3
70.7
66.5
74.1
73.2
60.5
68.8
62.4
58.7
69.9
69.3
X-MAMI.
82.7
68.4
72.8
74.1
70.7
76.5
76.0
65.8
72.1
59.9
62.5
65.7
64.6
73.9
74.9
70.7
XNLI
inber-aad
57.2
Maveson-NoCM
81.1
66.2
70.6
72.7
68.5
74.7
73.9
63.1
70.3
51.9
56.8
64,0
60.6
71.5
71.6
67.8
Macedon
82.1
66.6
70.6
73.3
68.6
75.1
74.6
64.3
70.8
52.2
58.1
63.6
60.9
72.7
71.6
68.3
Macedon 2.5k
82,9
69.2
74.6
75.0
70.7
76.7
77.3
67.2
734
59.9
63.5
67.8
65.1
74.5
75.7
71.6
Table Performance comparison on the three datasets. "AVG" means the average accuracy of all languages. The highest scores per category are in bold. Results of are taken from (Liang al., 2020), and results of are taken from (Hu et al., 2020). Results of are taken from (Nooralahzadeh et al., 2020) and (Ahmad et al., 2021) or obtain based their official code release.
languages
Task
iTrinnelt
Deejel!
[Testtet
ing with generative adversarial networks (Good- fellow et al., 2014).
PAWS-X
50,004
14,000
14,000
QADSM
QADSM
100,900
30,000
10,000
XNLI
15
NLI
425,202
2,490
5,010
Macedon-NoCM, variant of Macedon, does covariance matrix modification.
Table 2: Statistics of datasets.
5.1.3 Evaluation and Implementation Details
for cross-lingual natural language understanding tasks.
We use Accuracy evaluate paraphrase identifi- cation, query advertisement matching. and natural language inference, following previous work (Hu et al., 2020; Liang et al., 2020). In implementation details are shown in the appendix.
mbert-wTDD (Hu al., 2020) fine-tune tne-tunes mbert with both English data and few-shot data, which can be considered as a multi-task training and is strong baseline.
5.2 Performance Comparison
Syn. (Ahmad et al., 2021), one of the state-of-the- art approaches for cross-lingual NLU tasks, takes advantage of the universal dependency tree struc- ture in mbert to benefit cross-lingual tasks. We use the official release of Syn. implementation 4
In this section, we report the results of baselines and the proposed Macedon in Table I. First, according to Table posed Macedon outperforms all the state-of-the- methods on three datasets with respect to all language average accuracy. Compared to Syn.. is a data augmentation method. augments source language data to simulate target domain data closely as possible. However, if the qual- ity of the data is not high enough, it is difficult to align high-resouree language and low-resource languages very well. However, the proposed Mace- don does rely on the augmented data. Com- pared to X-MAMI, which is different from Syn., it one of the state-of-th the-the-n art methods based on meta learning. However, based on Nooralahzadeh et al.
find that the pro-
X-MAML (Nooralahzadeh et al., 2020) proposes cross-lingual meta learning architecture based MAML (Finn et al., 2017) to learn good initialization which can be adapted for other lan- guages easily. It has shown its effectiveness on PT and NLU tasks. We adopt their official release of X-MAML implementation 5.
mbert+ady (Chen al., 2018b) learns language invariant representation based on adversarial train-
thw.bggtttttttteee// hhttpp::/ggthuu...m///ssaahhh hwt.://ssiaaaaaad//yytttaaaa 'https: wtwp:ehwwww..oococeeee/////eeeee

---

0.53
0.
Bet
25
83
123
140
100
182
(a)
(h)
Figure 2: Figure 2(a): -l.-divereence between different low-resource languages and English. Figure 2(b): Accuracy of different languages with respect to the KI--divergence between them and English.
(2020), X-MAML still needs hundreds or thou- sands low-resource language data to align them. When there are extremely limited low-resource lan- guage data, the proposed Macedon can be more effective to align different language data.
question. We show the accuracy of Macedon and mbert+adv with different number of few-shot data in Figure 3.
According to Figure 3, we find that both Mace- don and mbert+ady can have better performance as the number of few-shot data increases. With the increment of few-shot data, both Macedon and mbert-ady can achieve richer useful supervision to align different languages, they can have better performance. On the other hand, when the trans- lated data is limited, such as 10, 100, Macedon significantly outperforms than mbert+adv. The pos- sible reason is that the discriminator mbert+adv suffers from overfitting and can not distinguish dif- ferent languages, so it can not provide effective supervision signal to update the classifier. How- ever, when the amount of few-shot data increases to 1000, mbert+adv has better performance than that of Macedon. When mbert-adv can achieve enough target domain data, it can align languages better than Macedon, because Macedon just aligns the second-order moment (covariance matrix). prac- tical use, if there is only limited low-resource data, we suggest using Macedon.
Second, fine-tuning mbert with both English data and limited target language data may only pro- vide very little benefit for accuracy. Compared to mbert, mbert+wTD only achieves improvement PAWS-X and XNLI dataset. On QADSM dataset, mbert+ even performs worse than that mbert.
Third, covariance modification effective to improve the performance Macedon. Com- pared to Macedon-NoCM, Macedon has bet- ter performance on the three datasets. Because low-resource language data is limited, which is much smaller than the representation embedding, the estimated empirical covariance matrix is ill- conditioned. However, the covariance modification makes the empirical covariance matrix positive def- inite and provides more accurate estimation under limited data. Therefore, the proposed Macedon shows better performance than MaCedon-NoCM.
5.4 Effectiveness of Aligning Different Languages
0.80
In this section, show the empirical KL- divergence (Pérez-Cruz, 2008) between target lan- guages and English representation on PAWS-X dataset Figure 2 to answer the third research question.
2.545
0.86
3.558
Based on Figure 2(a), we find that the proposed Macedon is much more effective to align low- resouree languages and high-resource languages than mbert. The KL-divergence between all low- resource languages and English of Macedon is significantly smaller than that of mbert. shows that the distributions of low-resource languages and English representation is very close, which can
Figure 3: Accuracy of Macedon and adversarial train- ing with different number of translated data
5.3 Macedon and Adversarial Training
this section, we study the relationship between the proposed Macedon and mbert+ady (adversar- ial training method) to answer the second research

---

100
53
40
13
80
100
40
40
(a) mbert
(b) Macedon
SC
mbert
(d) Macedon
Figure 4: Representation visualization via t-SNE on PAWS-X Figure 4(a) is the visualization of mbert on English and German: Figure 4(b) is visualization of Macedon on English and German; Figure 4(c) the visualization of mbert English and Korean; Figure 4(d) is the visualization of Macedon English and Korean.
tial reason is that if the language has shorter lin- guistic distance to English, it will share more char- acteristics with English and easier to align.
5.5 Sensitivity w.r... Hyper- parameter In this section, we show the accuracy of Macedon with different values of coefficients on PAWS-X dataset in Figure 5 to answer the fourth research question. According to the Figure 5, we can find that for the averaged accuracy, it has similar values when changing the coefficient from le 4 to le 1. However, the accuracy will drop lot when the coefficient is 1cO. When the coefficient becomes 1e0, the loss function focuses too much on the maximizing coding rate-distortion while ignoring the classification loss. Therefore, it will lead to the decline in model accuracy.
0.6
Figure 5: Accuracy with different values of coefficients.
provide explanation that Macedon shows higher accuracy than mbert.
The accuracy of different low-resource lan- guages is almost inversely proportional to the KL- divergence according to Figure 2(b). For both the Macedon and mbert, if the KL-divergence be- tween one low-resource language and English is small, the low-resource language may have high accuracy, vice versa. Because English has rich data, model usually can predict accurately on English dataset. Therefore, the low-resource language representation close to English representation, it can borrow more information from English to make accurate predictions. We also find one more interesting phenomenon. Usually, if a low-resource language is closer to English with respect to lin- guistic distance, it may be aligned to English bet- ter (Nooralahzadeh al., 2020: Zhao al., 2020). For example, Spanish, French, and German have shorter linguistic distance to English than Chinese, Korean, and Japanese, and Spanish, French, and German have smaller KL-divergence. One poten-
5.6 Case Study
In this section, tion of Macedon and mbert respectively via SNE (Van der Maaten and Hinton, 2008) on PAWS- X dataset Figure 4 to show how different lan- guage representation distributes. Figure 4, there are two language pairs: 1) English and German in Figure 4(b), and 2) English and Korean in Fig- ure 4(d). Figure 4 shows that the representation of mbert on English and German is not aligned well. Most of data with different languages can be classi- fied easily. And this phenomenon is also shown on English and Korean data for mbert. However, for the proposed Macedon, its representation is mixed well among different languages, which intuitively
visualize the representa-

---

shows it can align different languages effectively.
Any opinions, findings, and conclusions or recom- mendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
6 Conclusion
In this paper, we study a practical scenario, i.e., learning model with abundant high-resource lan- guage data and limited low-resource language data, for cross- -ingual NLU tasks. We follow the gen- eral idea of aligning different languages, but differ- ent from widely used adversarial training models, we propose a novel approach named Macedon, which minimizes representation coding rate reduc- tion among different languages. The motivation is make model not to use extra codes to encode language information. The advantage compared to adversarial training methods is that the proposed Macedon does not rely on a large amount of low- resource language data because it does not need to estimate data distribution accurately. We conduct extensive experiments on three public datasets in- cluding paraphrase identification, natural language inference, and query advertisement matching. Ex- periment results show that the proposed Macedon outperforms the state-of-the-ar baselines.
References
Wasi Uddin Ahmad, Haoran Li, Kai-Wei Chang, and Yashar Mehdad. 2021. Syntax-augmenter multilin- gual bert for cross-lingua transfer. arXiv preprint arXiv:2106.02134.
Wasi Uddin Ahmad, Zhisong Zhang, Xuezhe Ma, Kai- Wei Chang, and Nanyun Peng. 2019. Cross-lingual dependency parsing with unlabeled auxiliary lan- guages. arXiv preprint 1XXi:1999000263
Hanan Aldarmaki and Mona Diab. 2019. Context- aware cross-lingual mapping. arXiv preprint arXiv:1903.03243
M Saiful Bari, Tasnim Mohiuddin, and Shafiq Joty. 2020. Uxla: A robust unsupervised data augmen- tation framework for zero-resource cross cross-lingual nlp. arXiv preprint arXiv:2004. 230.44000....240..
Toby Berger. 2003. dist-riitoo theory. Wiley Eney- clopedia of Telecommunications..
Limitations
Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Multi- lingual alignment of contextual word representations. arXiv prepriat arXiv:2002.035//
The proposed Macedon may be computational in- tensive in some cases. The proposed method needs to compute two terms f((X c) and e In the two terms, the complexity of determi- nant function might be high. If the matrix d-dimensional square matrix, the complexity is (d*), where w 2 2. Therefore, when repre- sentation dimensionality is high, such as XLM- ROBERTA-XI_ (Goyal et al., 2021), or the number of languages is very large (over hundreds or thou- sands of languages), the computation complexity of the proposed method may be high. How to re- duce its computation complexity is our future work in the next step.
Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi. John Wright, and Yi Ma. 2021. Redunet: A white- box deep network from the principle of maximizing rate reduction.
Weile Chen, Huiqiang Jiang, Qianhui Wu, Börje F Karlsson, and Yi Guan. 2021. Advpicker: Effec- tively leveraging unlabeled data via adversarial dis- criminator for cross- cross-lingual ner. arXiv preprint 216.::20060023000
Xilun Chen, Ahmed Hassan Awadallah, Hany Hassan, Wei Wang, and Claire Cardie. 201 Sa. Multi-source cross-lingual model transfer: Learning what to share. arXiv preprint 111:::1810
Ethics Statement
Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinherger. 2018b. Adversarial deep averaging networks for cross- lingual sentiment clas- sification. Transactions of the Association for Com- putational Linguistics, 6:557-570.
Although our framework makes significant ad- vancements in the field of cross-lingual natural lan- guage understanding, it also raises concerns about the societal impacts of automation, including job loss for those who work in annotation services and other industries that rely on human labor.
Somnath Basu Roy Chowdhury and Snigdha Chaturvedi. 2022. Learning fair representations via rate-distortion maximization. arXiv preprint arXiv:2202.00035.
A Acknowledgement
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint
This work is supported in part by the US National Science Foundation under grant NSF IIS- 108.

---

Xin Luna Dong, Yaxin Zhu, Zuohui Fu, Dongkuan Xu, and Gerard de Melo. 2021. Data augmentation with adversarial training for cross-lingual nli. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 5158-5167.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi. Ming Gong, Linjun Shou, Daxin Jiang. Guihong Cao, al. 2020. Xglue: A new benchmark datasetfor cross-lingual pre-training, un- derstanding and generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLI)), pages 6008-6018.
Zi-Y Yi Dou and Graham Neubig. 2021. Word alignment by fine- tuning embeddings on parallel corpora. arXiv preprint arXiv:2101 101.08231.
Yi Ma, Harm Derksen, Wei Hong, and John Wright. 2007. Segmentation of multivariate mixed data via lossy data coding and compression. IEEE transac- tions pattern analysis and machine intelligence, 156-51554-.
Yerai Doval, Jose Hamacho-Colladee Luis Espinosa- Anke, and Steven Schockaert. 2019. On the ro- bustness of unsupervised and semi-superviied cross- lingual word embedding learning. arXiv preprint 198.:799...0772..
Zhuoyuan Mao, Prakhar Gupta, Chenhui Chu, Mar- tin Jaggi, and Sadao Kurohashi. 2021. Lightweight cross-lingual sentence representation learning. arXiv preprint arXiv:2105. 13356.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. agdel-agmostic meta-learning for fast adaptation of deep networks. In International conference on chine learning, pages 1112-1135 PMLR.
Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva, and Isabelle Augenstein. 2020. Zero-shot cross-lingual transfer with meta learning. arXiv preprint arXi::2003.02739
lan Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems, 27.
Aitor Ormazabal, Mikel Arteixe, Gorka Labaka, Aitor Soroa, and Eneko Agirre. 2019. Analyzing the limi- tations of cross-lingual word embedding mappings. arXiv preprial 1:XXi::906.05400
Lin Pan, Chung-Wei Hang, Haode Qi, Abhishek Shah, Saloni Potdar, and Mo Yu. 2020. Multilingual bert post-pretraining alignment. arXiv preprint arXiv:2010.12547.
Naman Goyal, Jingfei Du Myle Ott, Giri Ananthara- man, and Alexis Conneau. 2021. Larger-scale trans- formers for multilingual masked language modeling. arXiv preprint
Fernando Pérez-Cruz. 2008. Kullback-leibler diver- gence estimation of continuous distributions. In 2008 IEEE international symposium on information theory, pages 1666-1670. IEEE
Milan Gritta and Ignacio lacobacci. 2021. Xeroalign: Zero-shot cross-linge lingual transformer alignment. arXiv preprint arx:: 200e210500.22 102.02272.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: massively multilingual multi-task benchmark for evaluating cross-lingual generalisa- tion. In International Conference Machine Learn- ing, pages 4411-4421. PMLR.
Libo Qin, Minheng Ni, Yue Zhang, and Wanxiang Che. 2020. Cosda-ml: Multi Multi-lingual code-switching data augmentation for zero-shot cross-lingual nlp. arXiv preprint arXiv:2006.0644
Sebastian Ruder, Tvan Vulié, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. Journal of Artificill Intelligence Research, 65:569- 631.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the nlp world. arXiv preprint arXiv:2004.09099
Tianze Shi, Zhiyuan Liu, Yang Liu, and Maosong Sun. 2015. Learning cross-lingual word embeddings via matrix co-factorization. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Confer- ence on Natural Language Processing (Volume 2: Short Papers), pages 567-572.
Phillip Keung, Yichao Lu, and Vikas Bhardwaj. 2019, Adversarial learning with contextual embeddings for zero-resource cross-lingual classification and ner. arXiv preprint arXiv:/909. 1309.00753.
Guillaume Lample and Alexis Conneau. 2019. Cross- lingual language model pretraining, arXiv preprint
Anders Søgaard, Ivan Vulić, Sehastian Ruder, and Man- aal Faruqui. 2019. Cross-lingual word embeddings. Synthesis Lectures on Human Language Technolo- gies, 12(2).1-332
Chia-Hsuan Lee and Hung-Y Lee. 2019. Cross-lingual transfer learning for question answering. arXiv preprint arXiv:1907.066
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. 2017. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176.
Zengyi Li. Yubei Chen, Yann LeCun, and Friedrich T Sommer. 2022. Neural manifold clustering and em- bedding. arXiv preprint 2201

---

Laurens Van der Maaten and Geoffrey Hinton, 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
A Proof of Proposition 1
Proof. For any non-zero vector v, we have
Takashi Wada and Tomoharu Iwata. 2018. Unsu- pervised cross-lingual word embedding by multi- lingual neural language models. arXiv preprint arXiv:1809.02300
(1- XX" tr txxxx)
v'v
= =(1-a)(X* v)² (x" v) + tr tr(XX
1¹ V¹u. v.
d
Runchuan Wang, Zhao Zhang, Fuzhen Zhuang, De- hong Gao, Yi Wei, and Qing He. 2021a. Adversarial domain adaptation for cross-lingual information re- trieval with multilingual bert. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 3498-3502.
(X we have (1-a)0² X X U
Because (x'v)) (1 X 2) 2 0. and x
0,
(x(xxx
0 0.
Therefore, (1- x1 X +a t xx definite.
is positive
Ziyun Wang, Xuan Liu, Peiji Yang, Shixing Liu, and Zhisheng Wang. 2021b. Cross- Cross-lingual text classifica- tion with heterogeneous graph neural network. arXiv preprint arXiv:2105.1122
B Proof of Proposition 2
Proof. Because log det is strictly concave, for By > 0, 2==1 = 1 and Zj E 84 we have
Shijie Wu, Alexis Conneau, Haoran Li, Luke Zettle- moyer, and Veselin Stoyanov. 2019. Emerging cross-lingual structure in pretrained language models. arXiv preprint
& 3g log (Z)); (5)
log det
Bg 2j)
y=1
j=1
Shijie Wu and Mark Dredze. 2019. Beto, bentz, be- cas: The surprising cross-lingual effectiveness of bert. arXiv preprint arXiv: 7904.9909001 09077.
and the equality holds iff Zi = = Zz. Here, we take 3, tr xx 1). Therefore, we have
?i and Zj I + nye ((1 - a X +
Ruochen Xu, Yiming Yang, Naoki Otani, and Yuexin Wu. 2018. Unsupervised cross-lingual trans- fer of word embedding spaces. arXiv preprint arXiv:1809.0366
Σ 3,
j=1
Ziqing Yang. Wentao Ma. Yiming Cui, Jiani Yc, Wanx- iang Che, and Shijin Wang. 2021. Bilingual align- ment pre-training for zero-shot cross-lingual transfer. arXiv preprint 21X.022006.01732.
k =I + d ((1 aa) Σ X,X!
nc
j=l
tr x,,,,, 1).
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaob- ing Song, and Yi Ma. 2020. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Advances in Neural Information Processing Systems, 33:9422-9434.
to d
Because
ΣØ= X;X?) = XX" we have
Wei Zhao, Steffen Eger, Johannes Bjerva, and Is- abelle Augenstein. 2020. Inducing language- agnostic multilingual representations. arXiv preprint arXiv:2008.09112.
R(X,c) c) log det
Σ By Z)),
(6)
j=1
(x, Σ Bg log det Zz).
(7)
Bowei Zou, Zengzhuang Xu, Yu Hong, and Guodong Zhou. 2018 Adversarial feature adaptation for cross- lingual relation classification. In Proceedings of the 27th International Conference Computational Lin- guistics, pages 437-448.
Therefore, the proposition holds.
C Implementation Details
For batch size and learning rate, we tune them on validation set with grid scarch over {8, 16, 32) and (le 5, 2e 5, 5) respectively. For n, we tune on a validation with grid search over [le - -,..., 1c0). For all experiments, we run 3 times and report the average results. We run our experiments on NVIDIA RTX A6000.


Reviews:
[{"review_id": "4xcStJtTYR", "reviewer": "Reviewer_AJrg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">The proposed model is not novelty enough, it simply applies the coding rate-distortion function to the representation of the last transformer layer. The authors should try to find which layer of the transformer should be applied to the coding rate-distortion function that can make the method more interesting.\n\nWe believe there may be some misunderstandings regarding our methodology. In our paper, we aim to meet two key criteria for representations:\n\n    1. The learned representation should encompass sufficient information for the NLU task.\n    2. The distribution of representations across different languages should exhibit similarity.\n\nTo fulfill the second criterion, we have extended and adjusted the structure of the coding rate-distortion function. Specifically, we advocate utilizing the final layer representation. This choice is essential to ensure that the representation employed for prediction adheres to the specified criterion. Consequently, we abstained from applying the proposed rate-distortion coding function to intermediary layers.\n\n>Lack of experiment. Authors mainly compare their model with mBERT and its variants without comparing it with current SOTA, e.g., XLM-R.\n\nThanks for pointing it out! The proposed Macedon is a versatile framework with applicability to various backbones. In our paper, we have chosen to demonstrate the effectiveness of our framework using one of the most representative multilingual pre-trained language models, mBERT. Additionally, we have incorporated XLM-R-base as another backbone and presented the results in Table A and Table B for PAWS-X and XMLI, respectively. Notably, our results reveal that Macedon consistently outperforms the baselines when paired with the XLM-R backbone.\n\nTable A: Performance comparison on the PAWS-X dataset with XLM-R-base as the backbone. The highest scores per category are in bold.\n|          |en   | de   | es   | fr   | ja   | ko   | zh   | AVG  |\n|-----------|------|------|------|------|------|------|------|------|\n|XLM-R     | 94.6 | 87.5 | 89.0 | 89.4 | 75.5 | 74.6 | 80.4 | 84.4 |\n|XLM-R+wTD | 94.1 |87.1 | 89.3 | 88.7 | 77.4 | 76.5 | 80.4 | 84.8|\n|XLM-R-adv | 94.5 | 87.1 | **89.5**| 89.1 | 77.2 | 77.3 | 80.7 | 85.1 |\n|Macedon   | **94.8** | **88.7** | 89.3 | **90.1** | **77.9** | **78.0** | **81.3** | **85.7** |\n\nTable B: Performance comparison on the XNMI dataset with XLM-R-base as the backbone. The highest scores per category are in bold.\n|           | en   | ar   | bg   | de   | el   | es   | fr   | hi   |\n|-----------|------|------|------|------|------|------|------|------|\n| XLM-R     | 84.8 | 71.7 | 77.9 | 77.5 | 76.2 | 79.1 | 78.2 | 70.1 |\n| XLM-R+wTD | 85.0 | 72.1 | 78.4 | 77.2 | 76.4 | 79.2 | 79.1 | 71.7 |\n| XLM-R-adv | 84.8 | 73.2 | 78.5 | 77.6 | 76.5 | 78.6 | 78.5 | 71.8 |\n| Macedon   | **85.3** | **73.5** | **79.0** | **78.8** | **77.6** | **80.9** | **79.6** | **72.0** |\n|           | **ru**  | **sw**   | **th**   | **tr**  | **ur**   | **vi**   | **zh**   |**AVG**  |\n| XLM-R     | 75.8 | 64.9 | 72.5 | 73.1 | 66.5 | 75.0 | 73.4 | 74.4 |\n| XLM-R+wTD | 76.3 | 66.5 | 73.9 | **74.1** | 68.0 | 75.4 | 75.2 | 75.2 |\n| XLM-R-adv | 76.6 | 65.6 | 73.3 | 73.8 | 68.3 | 75.7 | 74.8 | 75.2 |\n| Macedon   | **77.0** | **66.6** | **74.5** | 73.9 | **68.8** | **76.6** | **76.1** | **76.0** |\n\n>There are some typos in the paper, e.g., \"high-source language data\" in line 17 should be \"high-resource language data\", \"When\" -> \"when\" in line 22. Section 5.3 (RQ2) causes misunderstanding. Should it be the comparison with adversarial training?\n\nThank you for your valuable suggestions! We will incorporate these changes in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Minimizing coding rate reduction between languages is an interesting idea for aligning representations across languages. It does not require parallel data or accurate distribution estimates.", "reasons_to_reject": "- The proposed model is not novelty enough, it simply applies the coding rate-distortion function to the representation of the last transformer layer. The authors should try to find which layer of the transformer should be applied to the coding rate-distortion function that can make the method more interesting. \n- Lack of experiment. Authors mainly compare their model with mBERT and its variants without comparing it with current SOTA, e.g., XLM-R.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a new method called Macedon for cross-lingual natural language understanding. Besides the vanilla cross-entropy loss, the paper introduces a regularization term that minimizes representation coding rate reduction between languages. The idea is to learn representations that do not use extra codes to encode language-specific information. Experiments on paraphrase, inference, and query matching tasks show Macedon outperforms baselines like mBERT.", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- There are some typos in the paper, e.g., \"high-source language data\" in line 17 should be \"high-resource language data\", \"When\" -> \"when\" in line 22.\n- Section 5.3 (RQ2) causes misunderstanding. Should it be the comparison with adversarial training?"}}, {"review_id": "jYP9U9413U", "reviewer": "Reviewer_uCfL", "summary": "", "questions": "* The figures in general can be improved by increasing the font size.\n* Why not compare the proposed method to XLM-R in addition to mBERT? I understand the method won't scale along with the model size, but at least XLM-R should fit, and it is worth to be added in the baseline given its popularity and superior performance to mBERT.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">I understand the scope remains in language understanding, but would be nice to see how it works in more general setting like XTREME multilingual benchmark, or at least common multilingual tasks such as NER or QA. Also the experiment relies on mBERT as the baseline, which is less common in practice compared to XLM-R. Nevertheless, I am leaning toward accepting the paper even with lack of these aspects.\n\nWe sincerely appreciate your constructive suggestions. While we concur that evaluating our model in a more general setting such as XTREME would be beneficial, resource constraints have led us to select three representative datasets from both XTREME and XGLUE. Our intention is to include evaluations on XTREME and XGLUE in the revised version of our paper.\nAdditionally, we have introduced an experiment utilizing XLM-R-base as the backbone, the results of which are presented in Table A and Table B for PAWS-X and XMLI, respectively. As demonstrated by the data in these tables, the proposed Macedon consistently outperforms the baseline models.\n\nTable A: Performance comparison on the PAWS-X dataset with XLM-R-base as the backbone. The highest scores per category are in bold.\n|          |en   | de   | es   | fr   | ja   | ko   | zh   | AVG  |\n|-----------|------|------|------|------|------|------|------|------|\n|XLM-R     | 94.6 | 87.5 | 89.0 | 89.4 | 75.5 | 74.6 | 80.4 | 84.4 |\n|XLM-R+wTD | 94.1 |87.1 | 89.3 | 88.7 | 77.4 | 76.5 | 80.4 | 84.8|\n|XLM-R-adv | 94.5 | 87.1 | **89.5**| 89.1 | 77.2 | 77.3 | 80.7 | 85.1 |\n|Macedon   | **94.8** | **88.7** | 89.3 | **90.1** | **77.9** | **78.0** | **81.3** | **85.7** |\n\nTable B: Performance comparison on the XNMI dataset with XLM-R-base as the backbone. The highest scores per category are in bold.\n|           | en   | ar   | bg   | de   | el   | es   | fr   | hi   |\n|-----------|------|------|------|------|------|------|------|------|\n| XLM-R     | 84.8 | 71.7 | 77.9 | 77.5 | 76.2 | 79.1 | 78.2 | 70.1 |\n| XLM-R+wTD | 85.0 | 72.1 | 78.4 | 77.2 | 76.4 | 79.2 | 79.1 | 71.7 |\n| XLM-R-adv | 84.8 | 73.2 | 78.5 | 77.6 | 76.5 | 78.6 | 78.5 | 71.8 |\n| Macedon   | **85.3** | **73.5** | **79.0** | **78.8** | **77.6** | **80.9** | **79.6** | **72.0** |\n|           | **ru**  | **sw**   | **th**   | **tr**  | **ur**   | **vi**   | **zh**   |**AVG**  |\n| XLM-R     | 75.8 | 64.9 | 72.5 | 73.1 | 66.5 | 75.0 | 73.4 | 74.4 |\n| XLM-R+wTD | 76.3 | 66.5 | 73.9 | **74.1** | 68.0 | 75.4 | 75.2 | 75.2 |\n| XLM-R-adv | 76.6 | 65.6 | 73.3 | 73.8 | 68.3 | 75.7 | 74.8 | 75.2 |\n| Macedon   | **77.0** | **66.6** | **74.5** | 73.9 | **68.8** | **76.6** | **76.1** | **76.0** |\n\n>The figures in general can be improved by increasing the font size.\n\nThanks for your suggestion! We will modify them following your suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The approach is simple yet intuitive. There are a few works in this line of research but most of them focus on the vocabulary in a post-hoc manner, while this work focuses on the representation in each language, where they improve the latent space representation in terms of the rate distortion. Also, the paper is well-written, in a sense that one can understand the proposed method by just reading the paper with little struggle.", "reasons_to_reject": "I understand the scope remains in language understanding, but would be nice to see how it works in more general setting like XTREME multilingual benchmark, or at least common multilingual tasks such as NER or QA. Also the experiment relies on mBERT as the baseline, which is less common in practice compared to XLM-R. Nevertheless, I am leaning toward accepting the paper even with lack of these aspects.", "questions_to_authors": "* The figures in general can be improved by increasing the font size.\n* Why not compare the proposed method to XLM-R in addition to mBERT? I understand the method won't scale along with the model size, but at least XLM-R should fit, and it is worth to be added in the baseline given its popularity and superior performance to mBERT.", "paper_topic_and_main_contributions": "Multilingual language models (LMs) is a useful tool to tackle NLP tasks in the multilingual setting, or in a situation where there are no reliable pre-trained LMs available in the language of the desired tasks. However, one of the well-known bottlenecks of such multilingual LMs is the gap between a high-resource and low-resource language in performance when they are fine-tuned. The paper proposes a method to reduce this bias in multilingual LMs to ensure the performance in a low-resource language. The method is based on the coding rate function and the compression of mixed language representation is studied with the method. The claim is supported by the superior accuracy in multilingual language understanding tasks to the benchmark.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f3ttih0UBf", "reviewer": "Reviewer_hT3i", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">While the paper claims that the proposed Macedon approach outperforms state-of-the-art methods, the comparison might be limited to a specific set of existing approaches. A more comprehensive comparison across a wider range of methods, such as XLMR, would provide a more complete understanding of the proposed approach's performance.\n\nThank you for pointing that out! Given that the proposed Macedon is a versatile framework applicable to various backbones, we opted to feature one of the most representative multilingual pre-trained language models, mBERT, in our paper to illustrate the effectiveness of our framework. Furthermore, we introduced XLM-R-base as an additional backbone and presented the results for PAWS-X and XMLI in Tables A and B, respectively. Notably, our findings demonstrate that the proposed Macedon consistently outperforms the baselines when utilizing the XLM-R backbone.\n\nTable A: Performance comparison on the PAWS-X dataset with XLM-R-base as the backbone. The highest scores per category are in bold.\n|          |en   | de   | es   | fr   | ja   | ko   | zh   | AVG  |\n|-----------|------|------|------|------|------|------|------|------|\n|XLM-R     | 94.6 | 87.5 | 89.0 | 89.4 | 75.5 | 74.6 | 80.4 | 84.4 |\n|XLM-R+wTD | 94.1 |87.1 | 89.3 | 88.7 | 77.4 | 76.5 | 80.4 | 84.8|\n|XLM-R-adv | 94.5 | 87.1 | **89.5**| 89.1 | 77.2 | 77.3 | 80.7 | 85.1 |\n|Macedon   | **94.8** | **88.7** | 89.3 | **90.1** | **77.9** | **78.0** | **81.3** | **85.7** |\n\nTable B: Performance comparison on the XNMI dataset with XLM-R-base as the backbone. The highest scores per category are in bold.\n|           | en   | ar   | bg   | de   | el   | es   | fr   | hi   |\n|-----------|------|------|------|------|------|------|------|------|\n| XLM-R     | 84.8 | 71.7 | 77.9 | 77.5 | 76.2 | 79.1 | 78.2 | 70.1 |\n| XLM-R+wTD | 85.0 | 72.1 | 78.4 | 77.2 | 76.4 | 79.2 | 79.1 | 71.7 |\n| XLM-R-adv | 84.8 | 73.2 | 78.5 | 77.6 | 76.5 | 78.6 | 78.5 | 71.8 |\n| Macedon   | **85.3** | **73.5** | **79.0** | **78.8** | **77.6** | **80.9** | **79.6** | **72.0** |\n|           | **ru**  | **sw**   | **th**   | **tr**  | **ur**   | **vi**   | **zh**   |**AVG**  |\n| XLM-R     | 75.8 | 64.9 | 72.5 | 73.1 | 66.5 | 75.0 | 73.4 | 74.4 |\n| XLM-R+wTD | 76.3 | 66.5 | 73.9 | **74.1** | 68.0 | 75.4 | 75.2 | 75.2 |\n| XLM-R-adv | 76.6 | 65.6 | 73.3 | 73.8 | 68.3 | 75.7 | 74.8 | 75.2 |\n| Macedon   | **77.0** | **66.6** | **74.5** | 73.9 | **68.8** | **76.6** | **76.1** | **76.0** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a novel approach to improving cross-lingual natural language understanding (NLU) performance, particularly in scenarios where data for low-resource languages is scarce. \n\nThe introduction of the Macedon approach is innovative. It proposes a novel method for learning cross-lingual representations that minimizes the reliance on extra codes to encode language-related information. This approach is unique in its focus on rate-distortion functions to guide the representation learning process.\n\n", "reasons_to_reject": " While the paper claims that the proposed Macedon approach outperforms state-of-the-art methods, the comparison might be limited to a specific set of existing approaches. A more comprehensive comparison across a wider range of methods, such as XLMR, would provide a more complete understanding of the proposed approach's performance.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the challenge of training cross-lingual natural language understanding (NLU) models in situations where there is a scarcity of data for low-resource languages. \n\nThe primary focus of the paper is to propose a novel approach called \"Macedon\" that aims to improve cross-lingual NLU performance in scenarios where low-resource languages are involved. The authors highlight that existing methods, which often rely on adversarial training and mutual information estimation to learn language-agnostic representations, can struggle when data is extremely limited for a particular language. Accurately estimating data distribution becomes challenging in such cases.\n\nThe key contribution of the paper is the introduction of the \"Macedon\" approach, which stands for \"minimizing representation coding rate reduction.\" The proposed method seeks to eliminate language-associated information in a model's representations. This is done by minimizing the need for extra codes to encode language-related information, measured using the rate-distortion function. In other words, the goal is to create a model that can effectively represent language-neutral features without introducing additional complexity related to language-specific aspects.\n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SsRzChIEWn", "reviewer": "Reviewer_C8eb", "summary": "", "questions": "\n1.What is the difference between mbert and Macedon-NoCM in the Table-1? \n2. See the weaknesses.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">No mention about the base model architecture of the Macedon in this paper. Does it follow mbert? If yes, then why mbert was chosen? XLM-Roberta perfroms better than mbert in most of the tasks. Why it was not selected?\n\nYes, we selected mBERT as the backbone for our paper. Given that the proposed Macedon is a versatile framework applicable to various backbones, we specifically chose one of the most representative multilingual pretrained language models, mBERT, to demonstrate the effectiveness of our framework. Additionally, we introduced XLM-R-base as another backbone and presented the results for PAWS-X and XMLI in Tables A and B, respectively. Notably, our findings indicate that the proposed Macedon consistently outperforms the baselines when utilizing the XLM-R backbone.\n\nTable A: Performance comparison on the PAWS-X dataset with XLM-R-base as the backbone. The highest scores per category are in bold.\n|          |en   | de   | es   | fr   | ja   | ko   | zh   | AVG  |\n|-----------|------|------|------|------|------|------|------|------|\n|XLM-R     | 94.6 | 87.5 | 89.0 | 89.4 | 75.5 | 74.6 | 80.4 | 84.4 |\n|XLM-R+wTD | 94.1 |87.1 | 89.3 | 88.7 | 77.4 | 76.5 | 80.4 | 84.8|\n|XLM-R-adv | 94.5 | 87.1 | **89.5**| 89.1 | 77.2 | 77.3 | 80.7 | 85.1 |\n|Macedon   | **94.8** | **88.7** | 89.3 | **90.1** | **77.9** | **78.0** | **81.3** | **85.7** |\n\nTable B: Performance comparison on the XNMI dataset with XLM-R-base as the backbone. The highest scores per category are in bold.\n|           | en   | ar   | bg   | de   | el   | es   | fr   | hi   |\n|-----------|------|------|------|------|------|------|------|------|\n| XLM-R     | 84.8 | 71.7 | 77.9 | 77.5 | 76.2 | 79.1 | 78.2 | 70.1 |\n| XLM-R+wTD | 85.0 | 72.1 | 78.4 | 77.2 | 76.4 | 79.2 | 79.1 | 71.7 |\n| XLM-R-adv | 84.8 | 73.2 | 78.5 | 77.6 | 76.5 | 78.6 | 78.5 | 71.8 |\n| Macedon   | **85.3** | **73.5** | **79.0** | **78.8** | **77.6** | **80.9** | **79.6** | **72.0** |\n|           | **ru**  | **sw**   | **th**   | **tr**  | **ur**   | **vi**   | **zh**   |**AVG**  |\n| XLM-R     | 75.8 | 64.9 | 72.5 | 73.1 | 66.5 | 75.0 | 73.4 | 74.4 |\n| XLM-R+wTD | 76.3 | 66.5 | 73.9 | **74.1** | 68.0 | 75.4 | 75.2 | 75.2 |\n| XLM-R-adv | 76.6 | 65.6 | 73.3 | 73.8 | 68.3 | 75.7 | 74.8 | 75.2 |\n| Macedon   | **77.0** | **66.6** | **74.5** | 73.9 | **68.8** | **76.6** | **76.1** | **76.0** |\n\n>Selection of baselines are also not clear.\n\nThanks for pointing it out. We use following baselines: (1) zero-shot on mBERT, (2) mbert-wTD (fine-tuning mBERT with English and few-shot data), (3) Syn. (one of the state-of-the-art approaches for cross-lingual NLU tasks via data augmentation), X-MAML (one of the state-of-the-art approaches for cross-lingual NLU tasks based on meta-learning), and mbert-adv (a popular and strong baseline which learns language-invariant representation based on adversarial training with generative adversarial networks).\n\n>What is the difference between mbert and Macedon-NoCM in the Table-1\n\nMacedon-NoCM is a variant of the proposed Macedon. Macedon-NoCM utilizes Multilingual BERT as the backbone without incorporating covariance matrix modification. The comparison between Macedon and Macedon-NoCM constitutes an ablation study. The entry labeled \"mbert\" in the performance comparison table refers to zero-shot training on Multilingual BERT, achieved through fine-tuning Multilingual BERT with English data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper includes\n\n1. Novel approach: The paper proposes a novel approach called Macedon that uses a coding rate reduction technique to minimize the loss of information during representation learning to align different language representations.  \n\n2. Evaluation: The paper evaluates Macedon on three public cross-lingual benchmark datasets and compares it with existing state-of-the-art methods.  \n\n3. Contributions: The paper makes several contributions, including the proposal of two criteria for representation learning, the introduction of Macedon, and the evaluation of Macedon on benchmark datasets. \n", "reasons_to_reject": "The potential weaknesses of this paper includes: \n\n1.No mention about the base model architecture of the Macedon in this paper. Does it follow mbert? If yes, then why mbert was chosen? XLM-Roberta perfroms better than mbert in most of the tasks. Why it was not selected?\n\n2.Selection of baselines are also not clear. ", "questions_to_authors": "\n1.What is the difference between mbert and Macedon-NoCM in the Table-1? \n2. See the weaknesses.", "paper_topic_and_main_contributions": "This paper proposes an approach called Macedon for cross-lingual natural language understanding (NLU) tasks that aims to overcome the challenges of training models with limited low-resource language data. The paper addresses the problem of learning language-agnostic representations that can be used for cross-lingual transfer learning, which is important for many NLU tasks. \n\nThe main contributions of the paper are: \n\n1. The introduction of Macedon, a new approach that uses a coding rate reduction technique to minimize the loss of information during representation learning, and adversarial training to align different language representations. \n\n2. The evaluation of Macedon on three public cross-lingual benchmark datasets, including PAWS-X, QADSM, and XNLI, which shows that Macedon outperforms existing state-of-the-art methods for cross-lingual NLU. \n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses cross-lingual natural language understanding and proposes to minimize representation coding rate reduction between languages in order to learn representations that do not use extra codes to encode language-specific information.

Pros / Strengths:
- The idea is novel and interesting as the approach does not require parallel data or accurate distribution estimates.
- The paper is well written.
- The approach is evaluated on three datasets against some baseline models based on mbert.

Cons / Weaknesses:
- The baselines are not convincing (important models such as XML-R are missing), weakening the experimental sections. (Note: the authors have extended their experiments during the response period, including XML-R for at least two of their datasets. Why not all three?)
- The number of tasks is small considering that there are many more popular multilingual benchmark datasets available.

Action items for improving the paper:
- The experimental section is a bit weak. To increase the value of the paper for other researchers and its influence on future research, the authors should add results with stronger multilingual models (such as XLM-R (as already partly provided during the rebuttal phase)) as well as results on (at least a few) more datasets.
Decision: Accept-Findings