Clustering Pseudo Language Family in Multilingual Translation Models with Fisher Information Matrix
Authors: Xinyu Ma
Xuebo Liu
Min Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: Multilingual Translation
Low-resource
TL;DR: We propose a novel language family clustering method for multilingual models.

Abstract: In multilingual translation research, the comprehension and utilization of language families are of paramount importance. Nevertheless, clustering languages based solely on their ancestral families can yield suboptimal results due to variations in the datasets employed during the model's training phase. To mitigate this challenge, we introduce an innovative method that leverages the fisher information matrix (FIM) to cluster language families, anchored on the multilingual translation model's characteristics. We hypothesize that language pairs with similar effects on model parameters exhibit a considerable degree of linguistic congruence and should thus be grouped cohesively. This concept has led us to define pseudo language families. We provide an in-depth discussion regarding the inception and application of these pseudo language families. Empirical evaluations reveal that employing these pseudo language families enhances performance over conventional language families in adapting a multilingual translation model to unfamiliar language pairs. The proposed methodology may also be extended to scenarios requiring language similarity measurements. The source code and associated scripts can be accessed at https://github.com/ecoli-hit/PseudoFamily.

Clustering Pseudo Language Family in Multilingual Translation Models with Fisher Information Matrix
Xinyu Ma Xuebo Liu" Min Zhang
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China mxinyuma@gmail com, (liuxuebo, zhangmin2021 ) t.. edu.
Abstract
Current academic investigations in this domain gen- erally fall into two distinct categories: Firstly, the integration of diverse prior knowledge, which incor- porates the utilization of various forms of existing knowledge and language family classifications, as evidenced in research such as Bakker et al. (2009); Chen and Gerdes (2017). Secondly, the exploration and computation of language similarity center examining language representations and their com- parative resemblances (Tan al., 2019; Oncevay et al.. 2020; Maurya and Desarkar, 2022).
In multilingual translation research, the com- prehension and utilization of language families of paramount importance. Nevertheless, clustering languages based solely on their an- cestral families can yield suboptimal results due variations in the datasets employed dur- ing the model's 's training phase. To mitigate this challenge. we introduce innovative method that leverages the fisher information matrix (FIM) to cluster language families, anchored on the multilingual translation model's charac- teristics. We hypothesize that language pairs with similar effects on model parameters ex- hibit considerable degree of linguistic con- gruence and should thus be grouped cohesively. This concept has led us to define pseudo lan- guage families. We provide in-depth dis- cussion regarding the inception and application of these pscudo language families. Empirical evaluations reveal that employing these pseudo language families enhances performance over conventional language families in adapting multilingual translation model to unfamiliar language pairs. The proposed methodology may also be extended to scenarios requiring language similarity measurements. The source code and associated scripts can accessed at https://github.com http.//cooocoic//eeeettteeo
Conventional methodologies predominantly rely supplemental resources or the alteration of model architecture to categorize language fami- lies. However, those approaches need more adapt- ability, especially when dealing with recently de- veloped pre-trained large MNMT models, e.g., m2m100 (Fan et al., 2021), where the original train- ing data and model architecture are inaccessible. When it comes to significantly hampers the effectiveness of existing methodologies for language family clustering.
scenario this limitation
To address this gap. we introduce a novel ap- proach to pseudo language family clustering, which inherently depends on the MNMT model itself. Our proposed methodology hypothesizes that lan- guage pairs exhibiting similarities in their impact on model parameters are likely to possess a high level of congruence and should, consequently, be grouped together. We employ the FIM to quan- tify such similarities between language pairs. We demonstrate the efficacy of our approach by en- hancing low-resource translation with pseudo lan- guage family clustering, yielding superior results compared traditional language family clustering. Our main contributions are as follows:
1 Introduction
Multilingual neural machine translation (MNMT) aims to construct a single model to translate multi- ple languages and has proven its effectiveness (Aha- roni et al., 2019). The application such models for low-resource languages has revealed that lever- aging supplementary languages during the fine- tuning phase can yield results that surpass tradi- tional methods (Lakew al., 2018; Gu et al., 2018). Nevertheless, the practical challenge lies in the de- velopment of the optimal strategy identify the most beneficial auxiliary languages that can bol- ster the translation of low-resource language pairs.
We present innovative methodology for clustering language families uniquely de- signed to function without the need for data access or architectural modifications.
Co--firtt and Corresponding Author

---

We unveil three efficient methods anchored in the FIM comprehensively compute similar- ities amongst language pairs.
Balto-Slavic
Indo-Iranian Japonic ja
bg sr Korcanic
Empirical evaluations indicate that our clus- tered pseudo language families exhibit supe- rior performance compared to conventional language families. Furthermore, approach exhibits versatility, effortlessly adapting to scenarios that necessitate language clustering.
hr
uk hi
ko
sk
be
bn Germanic de
Austronesian
Turkic sv
ms
kk
tr
2 Language Clustering with FIM
Figure 1: The used language families in experi- ments.
The accurate representation of language pairs is foundational for computing their pairwise similar- ities and ensuring efficacious clustering. As such, devising profound and robust representations for these pairs becomes imperative to attain excellence in this arena. Drawing inspiration from contem- porary strides in parameter-efficiee fine-tuning where innovative strategies have emerged har- ness model parameters for representation (Ben Za- ken et al., 2022; Gong et al., 2022; Zhang et al., 2023) we propose the deployment of the FIM. Our approach seeks to capitalize on the task-specific parameters embedded within the model.
consistency, we adhere to the model configuration and training settings delineated in Section 3.2.
Fisher Information The concept of fisher infor- mation occupies pivotal position in information theory, serving instrumental measure in dis- cerning the importance and prospective value of tuning specific parameters. Essentially, quan- tifies the variability of the first derivative of the likelihood function's 's logarithm. Through gauging the magnitude of this metric, one can deduce the necessity of fine-tuning particular parameters for subsequent tasks (Xu et al., 2021).
2.1 Preliminary
Formally, the FIM has been defined and em- ployed in prior works (Theis et al., 2018; Thomp- son al., 2019; Zhong et al., 2022)
Dataset an initial step, we curated a com- mon language pool comprising translations of 17 languages to English (en), sourced from the TED Corpus (Qi et al.. 2018). These language pairs col- lectively serve as the complete set for selection and function as the auxiliary language for low-resource languages. These languages span seven distinct families: Balto-Slavic, Austronesian, Indo-Iranian, Turkic, Japonic, Koreanic, and Germanic. The lan- guages we employed are depicted in Figure 1.
Fo = E[(Vo log P(Y|X:6)
(I)
(Vo log P(Y[X;8))
Where X and Y denote the input and output respec- tively, and @ denotes the parameters of the model. For the i-th parameter in & we empirically estimate its fisher information using the diagonal elements of the FIM. The formula derives as follows:
Model For empirical validation of our pro- posed methodologies, we elected employ the m2m100 418M model (Fan al., 2021). It is underpinned by vast, many-to-many dataset, en- compassing an impressive compilation of 100 lan- guages. The innovation of this model lies in its unique data mining strategy, it capitalizes on lin- guistic similarities, thereby circumventing the need for exhaustive mining across all conceivable di- rections. Given the extensive breadth and innate multilingual capabilities of the model, it offers an apt platform for our investigative endeavors, fur- nishing a robust foundation for the appraisal and af- firmation our methodological propositions. For
Fj - ((V log (Y(X;
(2)
2.2 Calculating Similarity for Language Pairs
While leveraging a diagonal matrix aids in estimat- ing FIM, arriving at precise probability estimations remains non-trivial task. In light of this, we resort to the subsequent equation to approximate FIM:
F
D Σ(V log
(3)
Here, D symbolizes the complete dataset, with [D] representing the data amount, and the entire dataset is divided into mini-batches.

---

E.-
E,
hr
Da
uk
D.
sk
Dr
be
bg sr uk sk be ms kk de sv ja ko
bg hr sk bn ms de sv
Figure 2: The parameters distribution of FIM for the training languages. We divide the model into five parts: encoder attention layer (E.), encoder fully connected layer (Ez), decoder self-attention (Da). decoder cross attention layer (D2) and decader fully connected layer (Dz)-
Figure 3: The similarity computed with Overlap method, with darker colors representing the higher similarity between language pairs.
((a)) Σ(F, Fa)² Fe
(4)
We input parallel corpora corresponding to lan- guage pairs into the model and estimate the FIM us- ing Equation 3 for just one epoch. We accumulate the FIM using the formula for each ing the model training process without backward propagation. After the completion of one epoch, we calculate the average FIM obtained from each mini-batch as the final estimation. Appendix A. provides the pseudo-code we use to estimate FIM.
Here, t denotes the target language direction. and a represents the auxiliary language pair. Fur- thermore, F denotes the FIM, Fa corresponds to the matrix associated with the auxiliary lan- guage, F, corresponds to the target language. and symbolizes the matrix's 's size.
dur-
2. KL Divergence (KL): By leveraging the FIM, we compute the KL divergence from other lan- guages in relation to the target language, thereby capturing
Significance of FFN in FIM Estimation The model was then divided into five sections, and the distribution of the highest 40% parameters is de- picted in Figure 2. It is pivotal note that the feed-forward networks (FFN) contain over 60% of the selected parameters. comparison was made with related study (Meng et al., 2022) regarding the fine-tuni mun-ttuning performance on the MNLI-m task. In this study, the researchers solely fine-tuned se- lected type of weight matrix from each transformer layer, and the best results were achieved when only the FFN was fine-tuned. From these observations, we conclude that the optimal strategy for this task involves utilizing the FIM derived from the FFN.
nuanced measure of the dis- tances between language pairs.
F
(( Σ Fa log
(5)
F.
The nomenclature used here mirrors that of the MSE, with indicating absolute value.
3. Overlap Similarity (Overlap): In further lever- aging the FIM, we construct a fisher information mask. Here, the uppermost K parameters are attributed value of 1, whilst the others are des- ignated value of 0. The similarity amongst each language pair ascertained using:
Variants for Similarity Calculation To effec- tively leverage the results derived from the FIM and judiciously select auxiliary languages for mul- tilingual training of specific target language pair, we devised three distinct strategies for calculating the similarity or distance between language pairs:
Stt.a) Overlapping(M1, Ma) Activate(M:)
(6)
Here, M symbolizes the fisher information mask, with Overlapping and Activate respec- tively quantifying the number of parameters that are simultancously activated and those activated in the target direction.
Mean Square Error (MSE): Utilizing MSE offers an intuitive depiction of the distributional disparities among different language pairs. It is formally represented as:
2.3 Clustering Pseudo Language Family
Selection of Pseudo Language Families For the selection of auxiliary languages, we designed a

---

3 Improving MNMT with Pseudo Family
South Slavic
East Slavic
Russian(ru) Ukrainian(uk) Belarusian(be)
3.1 Searching the Best K for Overlap Method To ascertain this threshold, we embarked upon series of experimental procedures with the range spanning from 30% to 80%. Our meticulous anal- yses culminated in the selection 40% as the de- fault parameter for the Overlap method, given its superior performance during the evaluation of the overall performance on the developmental set.
Eastern Subgroup
Bulgarian(bg)
Macedonian(mk)
West Slavie
Western Subgroup
Csech-Slovak(sk)
Slovene(sl)
Croatian(hr)
Lekhitic
Serbian(sr)
Bosnian(bs)
3.2 Main Experiments
Montenegrin
Sorbian
Model and Dataset We employ the same m2m100_44 418M model as outlined in (Fan et al.. 2021) the foundational and utilize the datasets delincated in Section 2.1. For our experiments, we selected languages from the Indo-Iranian and Aus- tronesian language families to test our methods, considering all language pairs as potential auxil- iary languages. All the data has been cleaned and tokenized with fairseq script and mosesdecoder',
Figure 4: The phylogenetic of Slavic languages.
pragmatic algorithm. Beginning with the sorting of similarities between language pairs, we set an initial search radius. Within this pre-defined bound- ary, the language pairs in closest proximity are integrated into the auxiliary language roster. The radius is then adjusted, taking cues from the simi- larity metric of the latest added language pair. This process is iterated until the list ceases expand with new language pairs. As a result, we identify cluster that we term the pseudo language family.
Baselines For our evaluation, we considered the following baselines based on the primary model: 1) Vanilla: Direct translations were conducted using the pre-trained model for the target language pairs without fine-tuning: 2) FT: The primary model underwent a fine-tuning phase using bilingual data specific to the target language pairs; h3) LF: The primary model fine-tuned by employing the traditional language family delineated in Figure 1, with temperature sampling and set temperature 1.5; 4) LF+FT: Expanding on the LF method, we integrated an additional tine-tuning phase that used the target language pair data.
This algorithm progressively broadens its selec- tion boundary but at decelerating rate, aiming assimilate languages beneficial for the auxiliary set while sidestepping those exhibiting pronounced dissimilarities. For a comprehensive insight into the algorithm, refer to Appendix A.2.
Analysis We examined the efficacy of FIM in gauging linguistic similarity by juxtaposing it with the Slavic language family structure in Figure This comparison was made using the similarity ma- trix derived from the Overlap method, as illustrated in Figure 3.
Training and Inference For the training stage, set the batch size as 4096; for our methods, we up-sample the training data to be the same to en- sure that the proportion of each language within each mini-batch is equal except Hindi (hi), which is aligned with LF when using Overlap method. For optimization, we deployed the Adam opti- mizer (Kingma and Ba. 2015) with B1 0.9. B₂ 0.98, and 10e-6 A learning rate of Ir 3e was chosen for this process.
The insights obtained from FIM are consistent with the findings of phylogenetic tree analysis. For instance, Serbian (sr) and Croatian (hr), situated the same terminal branch, are an- ticipated exhibit higher similarity score.
In contrast, languages such as Ukrainian (uk), which belongs to the East Slavie linguistic group. are situated remotely the phylogenetie tree. This geographical and linguistic distance results in only moderate similarity scores when compared with non-Fast Slavic languages in our matrix. These observations are aligned with the insights presented by Kudugunta et al. (2019).
For the inference stage, beam search was em- ployed with beam size set 5 and length penalty 1.0 applied consistently across languages. Evaluation metrics were based on BLEU scores,
/https://github. womn/faccbookeeeerddaee tree/maiii htte//mieeeaaele//mmma htttps: / thub. com/moses- satteesssst/mmmee satt/oosssdeooer

---

Model
hi
bn
id
ms
KL
id de
bg de
sv uk id hi be fa
bg
id bg sv
MSE
bg sv de
id bg sk hg sv de uk
Overlap
ko
bn fa
ir
bg ms sv
Table 1: The pseudo language family formed by the similarities computed using our methods.
calculated using LLU (Post, 2018). Our ex- perimental framework was built upon the fairseq toolkit (Ott et al., 2019).
languages identified through our proposed methods and the baselines, are showcased in Table 2. Our pseudo family strategies delivered notable perfor- mance uplifts across all three methods, clearly out- pacing the conventional language family method.
Model
fa
hi
bn
Avg.
Baselines
Averagely, noted a boost of 1.7 BLEU scores when juxtaposed against the traditional language family approach, rise of 1.1 BLEU when com- pared to pure fine-tuning, and a 1.0 BLEU elevation against the hybrid approach of fine-tuning post mul- tilingual training. A notable leap in performance was witnessed for several low-resourced language pairs, underpinning the efficacy of our strategy.
Vanilla
25.0
19.5
10.7
28.0
28.6
22.4
36.5
35.1
23.0
39.4
35.3
33.9
LF
33.9
35.6
26.3
38.4
32.5
33.3
LF+FT
36.8
35.6
26.3
39.0
32.5
34.0
Our Methods
KT
36.1
35.5
25.2
40.0
37.6
34.9
MSE
36.9
27.1
39.9
35.9
34.9
Overlap
36.9
35.6
27.2
39.7
35.5
35.0
Furthermore, we assessed methodology's resilience across diverse datasets. Our proposed method adeptly marries linguistic pertinence with data-related similarity, facilitating the astute selce- tion of auxiliary languages for optimized transla- tion outcomes. For comprehensive details, please refer to the Appendix A.5.
Table 2: The main result of our experiment on baselines and our pseudo family method with three different methods to calculate similarities. The threshold of K in Overlap method is set as 40%.
3.3 Result and Analyse
Comparison between Language Families A deeper investigation into language families, as de- picted in Table 1, reveals a critical insight: within the traditional structure of a language family, it is not guaranteed that all languages will provide su- perior translation outcomes, even precipitating significant compromise in the translation quality. Intriguingly, within the pseudo language family, our analysis identified language pairs that were lin- guistically divergent from the target language. For instance, Korean (ko) was selected as an auxiliary language for translation from Farsi (fa) to en. Data similarity metrics show that fa exhibits relatively higher similarity with ko than with others, as we have detailed in Appendix A.3.
4 Conclusion and Future Work
In this study, we introduce pioneering approach and three methods to quantify the similarities be- tween language pairs and subsequently propose pseudo family, based on this measurement, enhance the selection of auxiliary languages for multilingual training in low-resource language sce- narios. Our strategies for assessing language simi- larity leverage fisher information to sieve through model parameters, subsequently identifying task- specific ones and utilizing them to formulate an FIM gauge the similarity between language pairs. The experimental outcomes significantly improve BLEU scores on low-resource languages, affirming our proposed method's feasibility and accuracy.
This phenomenon highlights that methodol- ogy is proficient in tapping into dimensions beyond traditional linguistic boundaries, thereby selecting language pairs or datasets that are more conducive to optimal model training. We have delved deeper into the comparative analysis substantiating the su- periority of our method in Appendix A.4.
Future works include: 1) expanding the appli- cation of the proposed FIM methodology to other tasks that require the calculation of language sim- ilarity and 2) integrating our findings into cur- riculum learning frameworks (Liu et al., 2020; Zhan al., 2021) and consistency learning frame- works (Li et al., 2022; Wang et al., 2022; Liu et al., 2023; Li et al., 2023).
Effect of Pseudo Language Family Our main results, comparing the outcomes between auxiliary

---

Limitations
Eric Holman. 2009, Adding typology to lexico- statistics: A combined approach to language classifi- cation. Linguistic Typology.
In the present study, our methodology and exper- iments predominantly concentrated on the anal- ysis of low-resource language pairs. An explo- ration of the generalization performance of our approach across more comprehensive range of language pairs would be of significant value. Moreover, while our experiments employed the m2m100_41180 model, future investigations could benefit from incorporating larger m2m 100 models. Testing the generalization capacity of our method with diverse models, such as the NLLB (Costa- jussà et al., 2022) and Sea (Barrault al., 2023), could yield more valuable insights. Furthermore, is pertinent to note that the FIM retains extensive information. In forthcoming stud- ies, we intend to implement innovative, parameter- efficient fine-tuning strategies akin to those pro- pounded by (Gong et al., 2022). These strategies aim to ameliorate the detection of pertinent parame- ters while concurrently managing the total number of parameters with greater efficacy.
Loic Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al. 2023. Scamlessm44-masiveee mul- tilingual & multimodal machine translation. ArXiv preprimt, abs/2308. 11596............
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. BitFit: Simple parameter-efficient fine-tuning for transformer-bet Proceedings of the 60th Anmual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 1-9, Dublin, Ireland. Associa- tion for Computational Linguistics.
masked language-models, In
Xinying Chen and Kim Gerdes. 2017. Classifying lan- guages by dependency structure. typologies of delex- icalized Universal Dependency treebanks. In Pro- ceedings of the Fourth International Conference on Dependency Linguistics (Depling 2017), pages 54- 63, Pisa.Italy. Linköping University Electronic Press.
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. ArXiv preprint, abs/2207.046%
Acknowledgment
This work supported in part by the National Natural Science Foundation of China (Grant No. 62206076). Shenzhen College Stability Support Plan (Grant Nos. GXWD2022081 173340003,
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma. Ahmed El- El-Kishky, Siddharth Goyal. Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Ar- mand Joulin. 2021. Beyond english-centric multi- lingual machine translation. J. Mach. Learn. Res., 22:107:1-107:48
GXWD202208111211150000 Shenzhen Sci-
ence and Technology Program (Grant No.
XCBS2022100809112005 Xuebo Liu was sponsored by CCF-Tencent Rhino-Bire Open Research Fund. The computational resources of this research were supported by the Education Center of Experiments and Innovations at Harbin Institute of Technology, Shenzhen. We would like to thank the anonymous reviewers and meta- meta-reviewer for their insightful suggestions.
Zhuocheng Gong, Di He, Yelong Shen, Tie- Yan Liu, Weizhu Chen, Dongyan Zhao, Ji-Rong Wen, and Rui Yan. 2022. Finding the dominant winning ticket in pre- traitrained language models. In Findings of the As- sociation for Computational Linguistics: ACL 2022, pages 1459-1472, Dublin, Ireland, Association for Computational Linguistics.
References
Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor O.K.
Li. 2018. Universal neural machine tramslation for extremely low resource languages. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I [Long Pa- pers), pages 344-354, New Orleans, Louisiana. As- sociation for Computational Linguisties.
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North
American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 3874-334 -3844, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations. ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Dik Bakker, André Müller, Viveka Velupillai, Søren Wichmann, Cecil Brown, Pamela Brown, Dmitry Egoroy, Robert Mailhammer, Anthony Grant, and

---

Sneha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan Firat. 2019. Investigating multilingual NMT
Empirical Methods in Natural Language Processing (EMNLP)), pages Computational Linguistics.
Online. Association for
representations at scale. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP)), pages 1565-1575, Hong Kong, China. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Con- ference of the North American Chapter of the Associa- tion for Compulational Linguistics (Demonstrations), pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Surafel M Lakew, Marcello Federico, Matteo Negri, and Marco Turchi. 2018. Multilingual neural ma- chine translation for low-resource languages. IJCoL. Italian Journal of Computational Linguistics, 4(4- 1):11-25.
Matt Post. 2018, A call for clarity in reporting BLEU scores, In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguisties.
Yinghao Li, Xuebo Liu, Shuo Wang, Peiyuan Gong, Derek F. Wong. Yang Gao, Heyan Huang, and Min Zhang. 2023. TemplateGEC: Improving grammatical error correction with detection template. In Proceed- ings of the st Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers), pages 0878-6892, Toronto, Canada. Association for Computational Linguistics.
Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Pad- manabhan, and Graham Neubig. 2018. When and why pre-trained word embeddings useful for neu- ral machine translation?
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Short Papers), pages 529-535, New Orleans, Louisiana. Associa- tion for Computational Linguistics.
Zhaocong Li, Xuebo Liu, Derek F. Wong, Lidia S. Chao, and Min Zhang. 2022. ConsisiTL: Modeling consis- tency in transfer learning for low-resource neural machine translation. In Proceedings of the 2022 Con- ference Empirical Methods in Natural Language Processing, pages 8383-8394, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Xu Tan, Jiale Chen, Di He, Yingce Xia, Tao Qin, and Tin Liu. 2019. Multilingual neural machine translation with language clustering. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNIP-IJCNLLP), pages 963-973, Hong Kong, China. Association for Computational Lin- guistics.
Shudong Liu, Xuebo Liu, Derek F. Wong, Zhaocong Li, Wenxiang Jiao, Lidia S. Chao, and Min Zhang. 2023. KNN-TL: k-nnerest-neighhor transfer learn- ing for low-resource neural machine translation. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics ( (Volume I: Long Pupers), pages 1878-1891, Toronto, Canada. Association for Computational Linguistics.
Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. 2018. Faster gaze prediction with dense networks and fisher pruning. ArXiv preprint.
Xuebo Liu, Houtim Lai, Derek F. Wong, and Lidia S. Chao. 2020. Norm-based curriculum learning for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguisties, pages 427-436, Online. Associ- ation for Computational Linguistics.
Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh, and Philipp Koehn. 2019. Overcoming catastrophic forgetting during domain adaptation of neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long and Short Papers), pages 2062-2000 266-2068, Minneapolis, Min- nesota. Association for Computational Linguistics.
Kaushal Maurya and Maunendra Desarkar. 2022. Meta- XNI.C A lreta-learning approach based on language clustering for zero-shot cross-lingurt transfer and generation. In Findings of the Association for Com- putational Linguistics: ACL 2022, pages 269-284, Dublin, Ireland. Association for Computational Lin- guistics.
Zhijun Wang, Xuebo Liu, and Min Zhang. 2022. Break- ing the representation bottleneck of Chinese charac- ters: Neural machine translation with stroke sequence modeling. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 6473-6484, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yu Meng. Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022. Generating training data with language mod- els: Towards zero-shot language understanding. In Advances in Nearal Information Processing Systems.
Arturo Oncevay, Barry Haddow, and Alexandra Birch. 2020. Bridging linguistic typology and multilingual machine translation with multi mue language repre- sentations. In Proceedings of the 2020 Conference on
Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang.

---

2021. Raise a child in large language model: To- wards effective and generalizable fine- tuning. Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 9514- 9528, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Runzhe Zhan, Xuebo Liu, Derek F. Wong, and Lidia S. Chao. 2021. Meta-curriculum learning for domain adaptation in neural machine translation. Proceed- ings of the AAA/ Conference on Artificial Intelligence, 35(16)14310-14315
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. Adaptive budget allocation for parameter-efficiet fine-tuning. In The Eleventh In- lernational Conference on Learning Representations.
Qihuang Zhong, Liang Ding, Li Shen, Peng Mi, Juhua Liu, Bo Du, and Dacheng Tao. 2022. Improving sharpness- aware minimization with fisher mask for better generalization on language models. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pages 4064 4085, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.

---

A Appendix
A.2 Algorithm for Language Selection
A.1 Process of Similarity Calculation
The algorithmic details for the selection auxiliary language pairs are as follows:
Algorithm FIM Estimation
Sort the similarities in descending (ascending for MSE and KL) order create list L.
Data: θ parameters weight; D: training
data; IDI: number of mini-batcher in dataset D; ( loss function for parameters θ; sample, mini-batch for t-th step; FIM,: fisher
2. Initialize Gap [ - L[O]], add the first language into the auxiliary list,
3. Iterate from i = 2 to the end of L, for each loop select language as follow:
information matrix initialization: Begin training for epoch one;
1) If LLi 1] L Gap add the i- th language into the auxiliary list and update the Gap [[ 1 -[]]
θ inherent from the pre-trained model; FIMo < 0: t <- O;
2) If LLi 1] - L - Gap add the i-
for sample; D do
th language into the auxiliary list and update the Gap Gap,
L(O)= step_treetrepppp (r___rrnning());
// trraiinng for mini-batch of dataset D;
3) If L - - L[(] > Gap, terminate the loop;
aL(G)
91= 26
// Get gradients;
4. The language pairs in the auxiliary list with the target pair are composed of the pseudo language family for the target language pair.
+1 + FIM, o
7
D
<-1+1:
// update FIM;
A.3 Data Similarity Analysis of Pseudo Family
// do not backpropagation;
end
From a linguistic vantage point, ko and lan- guages demonstrate stark disparitics. They neither share analogous characters nor exhibit proximal grammatical structures. Despite these disparities, it is pertinent to highlight the prevalence of word embedding as method to abstractly represent char- acters and sentences, thereby bridging the gap be- tween linguistically disparate languages. Through word embedding, characters or phrases with analo- gous meanings may find themselves closer prox- imity within the embedded space.
return FIM;
In order to identify pertinent language pairs for multilingual training, we advocate for the applica- tion of the FIM similarity methodology, which has demonstrated commendable efficacy. Our proposed approach can be delineated into three phases:
1. Construction of Shared Language Pool: A shared language pool, embodying a plethora of languages across diverse language families, is constructed.
Taking this into consideration, our analysis tran- scends mere linguistic similarities and delves into the dataset's similarities as well. We have em- ployed sampling methodology to ascertain the data similarity between fa and both traditional and pseudo language pairs:
2. Derivation of FIM: The fisher information trix for each language in this pool is then com- puted using the estimation strategy detailed in Section 2.2, and the pseudo-code shown in Algorithm
1. Randomly select 10% data points from each auxiliary language training set, denoted as T.
3. Computation of Language Pair Similarity: When identifying suitable auxiliary languages for multilingual training with designated tar- get language pair, we employ three methods described Section 2.2 to calculate the simi- larity between each language pair.
2. Randomly select 10% data points from the training set of fa denoted as S.
For each data point in T, compute the cosine distance to all data points in S.

---

bg
1
0.85
0.85
0.86
0.87
0.77
0.81
0.75
0.71
0.85
0.81
0.69
0.8
0.85
0.86
0.79
0.83
ST
0.83
0.91
0.86
0.87
0.78
0.79
0.75
0.71
0.81
0.78
0.66
0.79
0.83
0.84
0.73
0.79
hr 0.83
0.91
1
0.85
0.87
0.78
0.79
0.75
0.71
0.81
0.78
0.66
0.79
0.83
0.84
0.73
0.78
uk 0.84
0.85
0.85
0.86
0.78
0.79
0.75
0.7
0.81
0.78
0.69
0.78
0.83
0.83
0.77
0.81
sk 0.85
0.88
0.87
0.87
0.77
0.79
0.75
0.7
0.82
0.79
0.68
0.79
0.84
0.84
0.77
0.81
be 0.76
0.8
0.79
0.8
0.78
0.77
0.76
0.73
0.76
0.7.
0.69
0.77
0.76
0.76
0.72
0.76
Ca 0.79
0.8
0.8
0.79
0.79
0.76
0.79
0.75
0.81
0.78
0.68
0.83
0.8
0.79
0.76
0.81
hi 0.75
0.78
0.77
0.77
0.76
0.77
0.81
1
0.82
0.76
0.75
0.69
0.8
0.77
0.76
0.73
0.78
bn 0.72
0.75
0.74
0.74
0.73
0.75
0.78
0.84
0.73
0.72
0.68
0.77
0.74
0.73
0.71
0.75
id
0.84
0.82
0.82
0.82
0.82
0.7.
0.82
0.75
0.71
1
0.84
0.68
0.81
0.84
0.84
0.77
0.82
ms 0.8
0.79
0.8
0.75
0.74
0.85
0.78
0.8
0.76
kk
0.72
0.71
0.71
0.74
0.72
0.72
0.72
0.71
0.69
0.72
0.72
0.72
0.72
0.71
0.73
0.73
tr 0.79
0.8
0.79
0.79
0.76
0.78
0.75
0.78
0.79
0.76
0.82
de
0.84
0.84
0.83
0.84
0.84
0.75
0.8
0.75
0.71
0.83
0.79
0.68
0.79
0.87
0.77
0.81
SV 0.85
0.85
0.84
0.84
0.84
0.75
0.79
0.74
0.7
0.84
0.8
0.67
0.79
0.87
1
0.76
0.81
ja
0.8
0.76
0.76
0.8
0.79
0.73
0.78
0.74
0.7
0.79
0.78
0.71
0.77
0.79
0.78
0.85
ko
0.83
0.81
0.8
0.83
0.82
0.76
0.82
0.77
0.73
0.83
0.8
0.7
0.83
0.83
0.82
0.84
bg
sr
hr
uk
sk
be
hi
bn
id
ms
kk
tr
de
SV
ja
ko
Figure 5: The full similarities computed TED datasets using Overlap method. We can directly infer the similarity between language pairs by observing the darkness or lightness of colors. Red boxes cover a language family, yellow boxes demonstrate the potential subfamily and black boxes contain some auxiliary languages beyond the language family that can leverage the target languageppait translation.
4. For each data point in T, average the cosine distances of its top closest data points in S as similarity.
A.4 Supplementary Analysis Language Families
Figure 5 elucidates that one can discern subgroups within vast language family, thereby alluding to the intricate web of language relationships and the potential for additional sub-categorizzatios. Cat- egorizing sub-language families solely based on linguistic methodology can be a formidable task.
5. Finally, average the calculated similarities for all data points in T.
The conclusive outcomes demonstrate that within the dataset, ko and fa share similarity of 32.6%,
Furthermore, it is apparent that languages chosen due to their intrinsic mutual similarities may not necessarily be part of the same traditional language family. This phenomenon echoes our methodol- ogy, wherein the model's parameters serve as a conduit for representation, thereby facilitating the amalgamation of multifaceted information span- ning languages and data, which could potentially unveil novel insights that could bolster our compre- hension of pre-trained models.
comparable to the likeness between Bengali (bn) at 32.6% and hi at 33.5%, with the latter duo form- ing part of the traditional language family of fa. Significantly, ko boasts more substantial data volume in comparison to bn and hi, factor that could considerably bolster the training procedure. This observation underscores that methodology does not exclusively hinge upon linguistic similari- ties; instead, it amalgamates information gleaned from the dataset, a process that facilitated by the model's representational form.
Moreover, traditional methodologics often em- ploy stringent classifications for language pairs,

---

bg
sr
hr
uk
sk
be
fa
hi
bn
id
ms
kk
de
SV
ja
ko
bg
sr
hr
uk
sk
be
hi
bn
id
mss
kk
tr
SV
ja
ko
Figure 6: The full similarities computed with OPUS100 and TED datasets. Even when computing the similarity between language pairs across various datasets, most similarities remain highest for the same language pair within different datasets.
which may prove inadequate when navigating lan- guages that hover near classification boundaries, potentially leading incongruities. Additionally, such a rigid approach might fall short when faced with shifts in the data domain.
sis to scrutinize the variations in similarities gen- erated by our approach across diverse data combi- nations. For the purposes of our experimentation and analysis, we employed the OPUSI00 dataset facilitate our experimental and research endeavors. Subsequently, we calculated Stt,a) where is de- rived from the TED dataset and a originates from the OPUSIO 100 dataset. Figure documents this full similarity matrix.
In contrast, our approach embraces more fluid and dynamic selection algorithm that meticulously curates choices tailored to each target language pair. Crucially, our method is versatile, accom- modating variations across different models and datasets, thereby ensuring its seamless applicabil- ity across a spectrum of scenarios. This versatility is highlighted by the observation that numerous languages can seamlessly integrate into multiple pscudo-language families.
Notably, findings demonstrate that even when languages derived from distinct corpora, there remains pronounced similarity among lin- guistically akin ones. This underscores the robust- ness of our methodology in discerning intrinsic linguistic affinities.
A.5 Robustness Analysis Across Different Datasets
To more comprehensively demonstrate the perfor- mance of our methodology across diverse domain- specific datasets, we conducted an in-depth analy-


Reviews:
[{"review_id": "5OKIgzhfXS", "reviewer": "Reviewer_BkKF", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review.\n\n> Q1: The $S_{(t,a)}$ is using the activated neurons and binarized them into 0 or 1. This seems to be adhoc, and if the authors can provide using floats of these activated neurons for similarly/distance computing for clustering, the compare and analysis will be great.\n\nThanks for highlighting the comparison with other distance algorithms. In our initial tests, we indeed explored various clustering strategies. We utilized fisher information, a feature intrinsic to neuron activation, to conduct these experiments. For distance measurement between language pairs, we applied MSE ($MSE = (F_t - F_a)^2/|F|$, where $F$ denotes the Fisher information matrix). While its performance was noteworthy, it was surpassed by the overlapping similarity on the fa-en task, as illustrated below: \n\n********************\n|      | BLEU          |Auxilary       | \n| ------------- | -------- | --------- |\n| Baseline      | 25       |           | \n| MSE           | 36.3     | id tr de  | \n| Pesudo family | **36.9** | tr id ko  |\n********************\n\nWe will incorporate a dedicated section elaborating on the outcomes of various clustering techniques in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Idea and experiment wise, the approach is simple and straightforward. Though some follow up details can be added to enrich the paper further, I think it is a good asset for the research community. \n\nThe authors claimed they will release code & data. ", "reasons_to_reject": "The $S(t, a)$ is using the activated neurons and binarized them into 0 or 1.  This seems to be adhoc, and if the authors can provide using floats of these activated neurons for similarly/distance computing for clustering, the compare and analysis will be great. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper detailed a simple approach of computing FIM, which in-turn was converted into activated neurons to represent each language for clustering.  The results, as summarized in table 2, seem to be quite reasonable, and experiments showed positive improvements. \n\nIdea and experiment wise, the approach is simple and straightforward. Though some follow up details can be added to enrich the paper further, I think it is a good asset for the research community. \n\nThe authors claimed they will release code & data. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z9d37F69MN", "reviewer": "Reviewer_yzL1", "summary": "", "questions": "- Is there any way to use the Fisher Information itself (e.g. through some distance function or similarity of gradients) instead of relying on this masking? \n- Table 2 is very interesting, and shows some unintuitive relationships (e.g. fa with ko, id with de), any hypotheses about this?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review.\n\n> Q1: The usage of a Fisher Mask and a similarity criterion based on this seems somewhat arbitrary, including choice of taking top 30% of parameters (was this tuned? why not another number).\n\nThe decision to use the top 30% of parameters was made after extensive experimentation. We evaluated the model's performance on clustering pseudo language families using different thresholds. It was observed that the threshold of 30% yielded the best performance among the tested values, providing a good balance between capturing significant parameters and method efficiency.\n\n> Q2: Is there any way to use the Fisher Information itself (e.g. through some distance function or similarity of gradients) instead of relying on this masking?\n\nThanks for emphasizing the thought of using Fisher Information itself. In our initial tests, we indeed explored various clustering strategies. We utilized fisher information, a feature intrinsic to neuron activation, to conduct these experiments. For distance measurement between language pairs, we applied MSE ($MSE = (F_t - F_a)^2/|F|$, where $F$ denotes the Fisher information matrix). While its performance was noteworthy, it was surpassed by the overlapping similarity on the fa-en task, as illustrated below: \n\n********************\n|      | BLEU          |Auxilary       | \n| ------------- | -------- | --------- |\n| Baseline      | 25       |           | \n| MSE           | 36.3     | id tr de  | \n| Pesudo family | **36.9** | tr id ko  |\n********************\n\nWe will incorporate a dedicated section elaborating on the outcomes of various clustering techniques in the revised version.\n\n> Q3: Table 2 is very interesting, and shows some unintuitive relationships (e.g. fa with ko, id with de), any hypotheses about this?\n\nThanks for pointing out this unusual outcome. It is truly unintuitive and absurd if we want to understand it from the perspective view of linguistics. However, once we feed those data to the pre-trained M2M model, they would be first represented by vectors containing semantic information. At the same time, in the word vector, words with similar meanings in different languages will have a closer distance, which can play a synergistic role in training. Therefore, take **fa** as an example. We measured the similarity using Euclidean distance between the training set of auxiliary languages and the training set of fa.\n\nTo calculate the similarity:\n1. Randomly select 100 data points, denoted as T, from the training set of auxiliary languages.\n2. Randomly select 100 data points, denoted as S, from the training set.\n3. For each data point in T, compute the cosine distance to all data points in S.\n4. For each data point in T, average the cosine distances of its top 5 closest data points in S.\n5. Finally, average the calculated similarities for all data points in T.\n\nThe similarities are shown below:\n********************\n| | Langauge Family| |Pseudo Family | | |\n| ---- | -------------------- | ------------------- | ------------------ | ------------------- | ------------------- |\n| |**bn-en**                      | **hi-en**                     | **tr-en**                    | **id-en**                     | **ko-en**                    |\n| **fa-en**   | 0.11492737485758579 | 0.11611198283623779 | 0.12014097928752948 | 0.12008474558327929 | 0.12465920288568932 |\n********************\n**bn** and **hi** are in the language family and the rest languages in the table are auxiliary languages. It shows that the auxiliary language that has high similarity in data could lead to better performance. The observation is also in lined with **id**.\n********************\n||Language Family |Pseudo Family ||\n| ---- | ------------------- | ----------------- | ------------------ |\n| | **ms-en**                  | **bg-en**                | **de-en**                 |\n| **id-en**   | 0.11061211301192167 | 0.11564603335590569 | 0.11717251327650846 |\n********************\nTherefore, our clustering results make sense from the perspective of model representation and data similarity. Languages with more similar data are more likely to be selected even if their similarity in linguistics is very low and better results can be obtained.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- contribution towards the insight that the beneficial effect of auxiliary languages in multilingual training can be largely explained by the model parameters rather than external linguistic or typological constraints. ", "reasons_to_reject": "- the usage of a Fisher Mask and a similarity criterion based on this seems somewhat arbitrary, including choice of taking top 30% of parameters (was this tuned? why not another number). ", "questions_to_authors": "- Is there any way to use the Fisher Information itself (e.g. through some distance function or similarity of gradients) instead of relying on this masking? \n- Table 2 is very interesting, and shows some unintuitive relationships (e.g. fa with ko, id with de), any hypotheses about this?", "paper_topic_and_main_contributions": "This paper presents a way to select auxiliary languages to improve machine translation performance within a multilingual setting. Specifically, the paper proposes to compute similarity between a language pair t, a as a function of the Fisher Information Matrix of the parameters when translating into t and the Fisher Information Matrix when translating into a. Intuitively, the Fisher Matrix encapsulates the variance of the (log) derivative, so if two target languages overlap a lot in terms of which parameters have high variance, then information from the languages should help each other. The authors provide justification for the subset of parameters they choose based on an analysis, and also show that their method improves over several reasonable baselines (fine-tuning, language family-based, both together), but by less than a BLEU point on average. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "84TAdASGV2", "reviewer": "Reviewer_kKvr", "summary": "", "questions": "Q A: If I understood correctly, you did not try to fine-tune each of the languages on all of the languages of the TED dataset?\n\nQ B: In Figure 4, the similarity is always 1 if the language is itself. Would it still be 1, if the material would not be the same?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review.\n\n> Q1: I'm not convinced that the presented method gives considerable enhancements and that its superiority is proven by the given experiments. It would need more evaluation. There are so many things here that can affect the end result. For example, why is ms so much better with FT than with others?\n\nYour query is valid and we understand the need for clarity on this matter. A key aspect contributing to the performance difference is the similarity between the training and test data. When there's a high similarity between the two datasets, the FT model tends to outperform the LF. This is evident from the computed similarities for the language pairs we experimented with. However, the use of Fisher information in our pseudo LF method allows it to consider data similarity and other related factors at the same time, thereby providing results superior to both LF and FT.\n\nTo calculate the similarity between training and test datasets, we used a sampling method:\n1. Randomly select 1000 data points, denoted as T, from the training set.\n2. Randomly select 100 data points, denoted as S, from the test set.\n3. For each data point in T, compute the cosine distance to all data points in S.\n4. For each data point in T, average the cosine distances of its top 5 closest data points in S.\n5. Finally, average the calculated similarities for all data points in T.\n\nTable 1: The similarity between the training set and test set in each language pair.$ \\Delta = BLEU_{FT} - BLEU_{Vanilla}$\n\n|            | fa                  | hi                  | ms                  |\n| ---------- | ------------------- | ------------------- | ------------------- |\n| similarity | 0.10810209134725653 | 0.12462727531056876 | 0.13408131563654796 |\n| $\\Delta $  | +11.5               | +15.6               | +16.7               |\n\n\n\n> Q2: The paper ignores the fact that some of the languages with traditional language families are written with different writing systems. But then, so are they in the generated pseudo families in this paper. I'm not convinced that Farsi model can learn anything from the Korean text which does not share single unicode character. Would the results be the same if just the English translations would be used to fine tune the models and there would not be any source text or the same sentence in Korean every time?\n\nYour skepticism, rooted in linguistic differences, is understandable. However, when using multilingual pre-trained models, the characters and sentences are transformed into vector representations post the embedding layer. This abstract representation allows languages to aid one another in the learning process, regardless of script differences.\n\nTo illustrate, we computed the distance between common words in Korean and Farsi using the vanilla M2M model. Specifically, Korean word embeddings were utilized to identify the nearest Farsi word via Euclidean distance.\n\nTable 2: Korean words and their closest Farsi counterparts based on embeddings.\n\n| ko     |           | fa    |               |\n| ------ | --------- | ----- | ------------- |\n| 만렙   | max level | تفریح | entertainment |\n| 마음   | mind      | ذهن   | mind          |\n| 하나   | one       | یکی   | one           |\n| 평화   | peace     | آرامش | peace of mind |\n| 불     | fire      | آتش   |fire          |\n| 학교   | school    | مدرسه | school        |\n\nFurthermore, to highlight the proximity of semantically similar words, we calculated distances between corresponding Korean and Farsi words, juxtaposed against the average distance between the Korean word and a broader set of commonly-used Farsi words.\n\nTable 3: Distances between semantically similar words vs. average distances.\n\n|     ko     |   fa    |  distance   | average  distance |  $\\Delta$   |\n| :--------: | :-----: | :---------: | :---------------: | :---------: |\n| 안녕하세요 |  آفتاب  | 8317.399414 |    9544.922852    | 1227.523438 |\n| 고마워하는 | خداحافظ | 15522.97461 |    17568.11328    | 2045.138672 |\n|     예     |  منون   | 4474.33252  |    8007.143066    | 3532.810547 |\n|   아니요   |   بله   | 10817.47852 |    11549.48438    | 732.0058594 |\n|    제발    | ببخشید  | 10472.74121 |    11903.8916     | 1431.150391 |\n|     큰     |  لطفاً   | 3131.273193 |    7231.745117    | 4100.471924 |\n|    작은    |  بزرگ   | 2274.627197 |    6987.947754    | 4713.320557 |\n|    친구    |  کوچک   | 3417.742676 |    7057.731934    | 3639.989258 |\n|    좋다    |  دوست   | 9683.666992 |    10981.41797    | 1297.750977 |\n|     집     |   عشق   | 3705.327393 |    7983.23877     | 4277.911377 |\n|    학교    |  خانه   | 2197.190186 |    7114.350586    |  4917.1604  |\n|     책     |  مدرسه  | 2706.901855 |    7249.868652    | 4542.966797 |\n|     펜     |  کتاب   | 10441.48047 |    11023.87305    | 582.3925781 |\n|    색상    |   قلم   | 8703.549805 |    11031.95801    | 2328.408203 |\n|     물     |   رنگ   | 4471.234375 |    7806.318848    | 3335.084473 |\n|    음식    |   آب    | 3287.777832 |    7130.495117    | 3842.717285 |\n\nOur findings indicate that words with parallel meanings tend to have reduced distances in vector space, validating the efficacy of multilingual machine translation even across script-divergent languages.\n\n> Q3: The paper mentions variations in the domain of the data utilized in the abstract, but it is not later investigated further. Does having different domain help or hinder the outcome? Are the results better because the model is seeing new English sentences and does not have that much to do with the source language?\n\nOur paper did indeed mention variations in the data domain in the abstract, and we apologize for not delving deeper into this in the main content. This was primarily due to constraints on paper length and our focus on other key aspects of the study.\n\nTo directly address your query on domain variations: Yes, domain similarity between the training set and test set does play a decisive role in model performance. Table 1 presents this correlation: as the similarity between the two sets increases, the performance during fine-tuning also tends to rise.\n\nFor a more specific illustration, we examined the fa language. We assessed its training set's similarity against the training sets of the language family (bn, hi) and the pseudo language family (tr, id, ko). The results, as displayed in Table 4, demonstrate that the auxiliary training data's similarity to the target language's training data is proportional to improved translation outcomes.\n\nTable 4: Similarity between language family and pseudo language family\n\n|      | bn                  | hi                  | tr                  | id                  | ko                  |\n| ---- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- |\n| fa   | 0.11492737485758579 | 0.11611198283623779 | 0.12014097928752948 | 0.12008474558327929 | 0.12465920288568932 |\n\nImportantly, our methodology primarily emphasizes enhancing the translation capability across bilingual pairs. This approach is not just about exposing the model to new English sentences; it's about fortifying its foundational bilingual translation competence. Our results suggest that the model's performance is influenced both by domain similarity and by the nuances of source languages. Our research underscores that a careful balance between these elements is pivotal for the optimal effectiveness of translation tasks.\n\n> Q4: If I understood correctly, you did not try to fine-tune each of the languages on all of the languages of the TED dataset?\n\nIndeed, we hadn't initially elaborated on that specific approach. Based on your suggestion, we fine-tuned the fa-en task across the entirety of the TED dataset. We subsequently used the resulting universal model as the foundation for further experiments. Here are the results:\n\nTable 5: BLEU Scores for the fa-en with Different Fine-Tuning Strategies\n\n|                   | **fa-en**   | \n| ----------------- | ---- | \n| Pseudo Family           | **36.9** | \n| Entire TED Dataset       | 36.2 | \n| TED Dataset + Pseudo Family | 36.7 | \n\n> Q5: In Figure 4, the similarity is always 1 if the language is itself. Would it still be 1, if the material would not be the same? \n\nThanks for the question. We split the training set into two subsets of equal data size and recalculated the fisher information matrix for analysis. Of course, the similarity on different subsets will not be 1, but it is still much higher than other languages. We also test them on a different corpus, we calculate $S_{(t,a)} $ where $t$ is in the TED dataset and $t$ is in the OPUS dataset, the result shows in different materials the language itself still contains better similarity and our method has involved the information about language itself. \n\n> Q6: In 2.1 on lines 110 and 111 it said that the respective volumes of the dataset would be presented on Table 1. They are not presented there.\n\nWe sincerely apologize for this oversight. We'll address this issue in the upcoming revision. Here is the missing table detailing the number of sentences for each language:\n\nTable 6: Number of Sentences for Each Language in the Dataset\n| language | sentences |\n| -------- | --------- |\n| **bg**       | 174,444    |\n| **sr**       | 136,898    |\n| **hr**       | 122,091    |\n| **uk**       | 108,495    |\n| **sk**       | 61,470     |\n| **be**       | 150,965    |\n| **fa**       | 18,798     |\n| **hi**       | 4,649      |\n| **bn**       | 4,509      |\n| **id**       | 87,406     |\n| **ms**       | 5,220      |\n| **kk**       | 3,317      |\n| **tr**       | 182,470    |\n| **de**       | 167,888    |\n| **ja**       | 204,090    |\n| **ko**       | 205,640    |\n| **sv**       | 56,647     |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of using FIM to calculate similarity and use that similarity to create new pseudo families is certainly an interesting one.", "reasons_to_reject": "I'm not convinced that the presented method gives considerable enhancements and that its superiority is proven by the given experiments. It would need more evaluation. There are so many things here that can affect the end result. For example, why is ms so much better with FT than with others?\n\nThe paper ignores the fact that some of the languages with traditional language families are written with different writing systems. But then, so are they in the generated pseudo families in thia paper. I'm not convinced that Farsi model can learn anything from the Korean text which does not share single unicode character. Would the results be the same if just the English translations would be used to fine tune the models and there would not be any source text or the same sentence in Korean every time?\n\nThe paper mentions variations in the domain of the data utilized in the abstract, but it is not later investigated further. Does having different domain help or hinder the outcome? Are the results better because the model is seeing new English sentences and does not have that much to do with the source language?", "questions_to_authors": "Q A: If I understood correctly, you did not try to fine-tune each of the languages on all of the languages of the TED dataset?\n\nQ B: In Figure 4, the similarity is always 1 if the language is itself. Would it still be 1, if the material would not be the same?", "paper_topic_and_main_contributions": "The paper introduces a way to calculate translation pair similarity using Fisher information matrix (FIM) in order to create pseudo language-families. The families are used to choose auxiliary data in fine-tuning tranlastion models. The method is evaluated against using traditional language families.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Give a reference to FIM already in the Introduction where it is mentioned first.\n\nThe given reference for the TED corpus on line 104 is wrong. Chronopoulou et al. use the TED corpus in a similar manner as the current paper, but for the TED talks the reference should be this one which is used by Chronopoulou et al.: https://aclanthology.org/N18-2084/", "Typos_Grammar_Style_and_Presentation_Improvements": "Space missing before citation on lines 35 and 210.\n\nIn 2.1 on lines 110 and 111 it said that the respective volumes of the dataset would be presented on Table 1. They are not presented there.\n\nIn Tables 2 and 3 and in their captions: \"Pesudo\" -> \"Pseudo\"."}}]

Author rebuttal:

Meta-review: This paper presents a new method for selecting auxiliary languages to improve machine translation. Instead of using traditional language families, they compute similarity between each pair of languages with Fisher Information Matrix, so that languages will be considered similar if they overlap a lot in terms of which parameters have high variance, and use this to compute pseudo-families based on the empirical data. The technique seems like it should be also applicable for other situations that would benefit from measuring language similarity.

The reviewers appreciated that the apprach is straightforward and effective.
Decision: Accept-Main