Investigating the Effectiveness of Multiple Expert Models Collaboration
Authors: Ikumi Ito
Takumi Ito
Jun Suzuki
Kentaro Inui
Venue: EMNLP 2023 Findings (2023)
Keywords: machine translation
multi-domain translation
multiple model collaboration
TL;DR: This paper investigates the effectiveness of several machine translation models and aggregation methods in a multi-domain setting under a fair condition.

Abstract: This paper aims to investigate the effectiveness of several machine translation (MT) models and aggregation methods in a multi-domain setting under fair conditions and explore a direction for tackling multi-domain MT. We mainly compare the performance of the single model approach by jointly training all domains and the multi-expert models approach with a particular aggregation strategy. We conduct experiments on multiple domain datasets and demonstrate that a combination of smaller domain expert models can outperform a larger model trained for all domain data.

Investigating the Effectiveness of Multiple Expert Models Collaboration
Ikumi Ito'
Takumi Ito 1.2 Jun Suzuki'.3
Kentaro Inui
Tohoku University *Langsmith Inc.
"IKENN MBZUAI
ikumi ito. p8edc. tohoku. jc.
(t-ito, Jun.suzukijJetohoku ac. jp kentaro. inui aww.zuuaiaccc...... ae
Abstract
Multi-
This paper aims to investigate the effectiveness of several machine translation (MT) models and aggregation methods in multi-domain set- ting under fair conditions explore a direc- tion for tackling multi-doman MT. We mainly compare the performance of the single model approach by jointly training all domains and multi-expert models approach with particular aggregation strategy. We conduct experiments multiple domain datasets and demonstrate that combination of smaller domain expert models can outperform a larger model trained for all domain data.
Mixed
Domain
one oulpui by one model
Data
(MD))
VS
DEMs
Damtiin
Expert
Output selection
Fisemblle Mra
Domain
Expert 2
OR
Domiin
Expert N
Figure 1: Comparative overview of MDM (top) and DEMs (bottom) settings.
Morcover, DEMs may have several advantages over MDMs. Thus, verifying DEMs' potential ad- vantages would be worthwhile.
1 Introduction
The machine translation (MT) research community has been paying much attention to multi-domain evaluation (Saunders, 2022; Pham et al., 2021) as well as document-level evaluation these days. This trend is supported by the fact that the main trans- lation task in the WMT competition' shifted to multi-domain setting after 2022. We explore more effective and efficient approach since this is a more realistic and desirable setting actual MT systems.
In this study, aim compare MDM and DEMs with various configurations in fair condi- tion and reveal the effectiveness of DEMs in the current MT technologies. We conduct experiments on multiple domain datasets and demonstrate that a combination of smaller DEMs can outperform a single larger MDM.
Related Work
Multi-domain.
The straightforward approach to addressing multiple domains is training on data en- compassing various domains. Much research is un- derway develop multi-domain model, such adding domain-specinn tags to inputs (Kobus al., 2017: Tars and Fishel, 2018), sampling of training data (Pham et al., 2022), and developing multi-head attention networks for each domain (Jiang et al., 2020). These approaches aim to enable a single model to handle various domains.
There are two major approaches for tackling multi-domain adaptation: a Multi-Domain Model (MDM), and Domain Expert Models (DEMss. In MDM, the model is trained on all domain data; in DEMs, expert models are trained for each domain (Figure 1). Most previous studies have focused on MDM (Kobus et al., 2017; Britz et al., 2017) since MDM is the identical approach when we do not consider domains. In contrast, DEMs have yet to be investigated in-depth due to not much attention in the community until recent years. We hypothesize that DEMs have the potential to achieve superior performance if each expert model well-trained in the corresponding domain and better combined.
Instead of multi-domain model, expert model for specific domains has also been ex- plored (Neves al., 2022) to enhance in-domain performance. To address multiple domains, collab- orating multiple expert models is feasible strategy. However, previous studies have yet to examine the
hhttp://ww22 statmt. org

---

effectiveness of this strategy due to the cost of run- ning multiple models. In light of the growing trend toward larger models, it is valuable to explore the performance potential of collaborative (smaller) models.
Dataset
#Sent Pairs
JParaCrawl v3.0
25.7M
The Kyoto Free Translation Task (KFTT)
440k
Sspanese-Engliste Legal Paraliel Corpus (LAW)
260k
TED talks (TED)
225k
Asian Scientific Paper Excerpt Corpus (ASPEC)
200k
The Business Scene Dialogue corpus (BSD)
20k
Collaboration of multiple models. Collabora- tion methods for multiple models (ensemble) have been explored. The typical method entails averag- ing the probabilities of each output token across multiple models. Kobayashi (2018) proposed the post-ensemble, which compares the outputs from multiple models and selects the one closest to the majority among the outputs. Ensemble methods are often applied to multiple models trained on the same data but with different random seeds, whereas we perform ensembles models trained on dif- ferent domains. Furthermore, output selection (or re-ranking), which selects the best one from out- puts, can benefit DEMs. Fernandes et al. (2022) investigated the effects of output selection using minimum bayes risk decoding (Kumar and Byrne, 2004) and quality estimation methods (Kepler et al.. 2019; Ranasinghe et al., 2020) in MT task. How- ever, their experiment only examines the effect of output selection when single model has multiple outputs. We investigate the gain when these en- semble and output selection methods are applied to multiple expert models.
Table Number of parallel sentences in training data.
Thus, n expert models are developed. At infer- ence time, either an ensemble is applied, or all expert models generate each output, and then the output selection algorithm determines the final out- put. Section 4.2 describes the output selection and ensemble methods used in experiments.
Experiments
We performed experiments Boggish-to-Jappanee (En-Ja) and topnnes (Ja-Ei) transla- tions with five domains.
4.1 Datasets, models, and evaluation
As shown in Table 1, we used JParaCrawl v3.0 (Morishita et al., 2022) for pre-training and the following five datasets for fine-tuning: The Ky- oto Free Translation Task (KFTT) (Neubig, 2011) 1), Japanese-Fnglish Legal Parallel Corpus (LAW)), TED talks (TED) (Cettolo al., 2012), Asian Sci- entific Paper Excerpt Corpus (ASPEC) (Nakazawa et al., 2016), and The Business Scene Dialogue corpus (BSD) (Rikters et al., 2019)3. The concept of 'domain' can be defined from several perspec- tives (Saunders, 2022). In this study, we equated differences in datasets with variations in domains. Thus, in the DEMs setting, fine-tuned five mod- els as the domain experts from the identical pre- trained model.
3 Investigation Focus
The multi-domain translation task assumes do- mains, with data Di E {1, ...,..,, n)) correspond- ing to each domain. Besides the domain data, gen- eral domain data D Dgeneral is assumed to be avail- able. Based on the existing domain adaptation ap- proach (Luong and Manning, 2015; Sennrich et al., 2016), the model is pre-trained in D Dgeneral: then fine-tuned in the domain data. At test time, no domain information provided with the source, although is an instance one of the n-domains. Following a description of MDMs and DEMs.
We built transformer-basee els (Vaswani et al., 2017) of three different sizes: SMALL (90M), BASE (290M), and LARGE (IB). The differences in model configurations for each size are shown in Table 2. To make fair comparison in both MDM and DEMs settings, (i) for pre-training, the same number of epochs and batch sizes were used for both settings, and (ii) for fine-tuning, MDM was updated 10k times, and each expert model in DEMs was updated 2k times, with the same batch size.*
MT mod-
Multi- Model (MDM). In MDM, sin- gle translation model is pre-trained on D geeneral, followed by fine-tuning with domain-mixed data Daul Dj +... D,,-
Domain Expert Models (DEMs). In DEMs, single model is pre-trained on D Dyeneral, as in MDM setting. Then, the pre-trained model is fine- tuned as an expert model on each domain data D,.
Ptttp:// http:://wwwwhhtottt 3 "Appendix A shows details of the datasets. "Appendix B shows the details of the models.
com/jaen-1 law/

---

Encoder & Decoder
83
OE MBR - -ENS
DM *'rale
MDM
82.61 x
Name
Params
layers
duodel
dmz
heads
MS-COMET-22
82
81.78
81.73
SMALL
90M
6
512
2048
81
80.97
BASE
290M
6
1024
4096
16
79.78
80.57 A
LARGE
IB
6
2048
8192
32
80
79.52
A 79.87
79.59 T 79.
79
78.10
Table 2: Model Configurations.
78
E 78.39
78.17
VIDII
VDM DEMS
MDM 1B
DEMS Rase 290M x
Small
Rase
Small
We chose MS-COMET-22 (Koemi et al., 2022)
90M
290M 90M x
Model size
primary evaluation metric. Since MS- COMET-22 is evaluation model developed by additionally training COMET (Rei et al., 2020)
Figure 2: Evaluation results for En-Ja test data.
81
AQE MBR ENS
DM
Oracle
MDM
with variety of domain data, it is considered suit- able for a multi-domain setting.
80.34 X
MS-COMET-22
79.31
79.31 +
79
4.2 Output selection and Ensemble for DEMs
78
78.35
77.72
78.12
76.97
77.23
77.30 D
Let y" represent the output of the i-th expert model, and let Y be the set of y for all i. The following explains the selection method for the final outputs in the DEMs setting:
76.75
76
75.56
76.27
75.50
75
MDM
MDM DEMs Bose Small
MDM Lange 1B
DEMs
Small
Base
Model size
Quality Estimation (QE): Select one from Y with the highest score of the quality estimation metric,
Figure 3: Evaluation results for Ja- En test data.
MISCOMMETQEE-2 (Koemi et al., 2022).
Results, Analyses and Discussions
Minimum Bayes Risk (MBR): Select one as fol- lows, UMER argmax,#ct [e] Σー= u(x; y², y²), where u MS-COMET-22, a reference-based met- ric measuring the similarity between hypothesis y E Y and a pseudo reference y" E Y, using also information of source r.
Figures 2 and 3 show the results for the En-Ja and Ja-En test data, respectively." Comparing MDM and DEMs for the same model size, DEMs scored higher for almost all methods. In addition, DEMs-SMALL (90M x 5) achieved competitive performance to MDM-LARGE (1B) by Oracle and MBR. We assessed statistical significance through the paired t-test and bootstrap resampling (Koehn, 2004). The results indicated that Oracle and MBR outperformed compared to MDM settings. These findings suggest that effective combinations of small expert MT models can surpass the perfor- mance larger MT model.
Ensemble (ENS): The token probability is calcu- lated as follows, p(ysel=) र Σ/= Py (veeel>) where Pj denotes the token probability of the expert model, and r represents the input. Subsequently, beam search is employed generate the output. Domain Match (DM): Manually select one accord- ing to the domain of the input. This is a cheat method because the domain information gener- ally unknown, as Section 3 described.
Comparison of output selection methods and Ensemble for DEMs: Across all model sizes and language directions, the five methods ranked Oracle MBR QE DM ENS. The differ- ence between QE and Oracle can be ascribed to the performance gap between MS-CON DE--CM-----2222 and M---00MT- seems counter-intuitive that MBR scored higher than QE Considering the supe- rior translation performance of the expert model in the target domain, it is reasonable to expect that QE,
Oracle (Oracle): Select with the highest score according the MS-COMET-22.. taking the refer- ence into account as well. This method yields an upper bound in the DEMs setting.
QE, MBR, DM, and Oracle are the output selection methods. Given that QE and MBR require only the output of each model, and ENS can be performed if the weights of each model are accessible, QE, MBR, and ENS practical settings for real-world appli- cations. In contrast, DM needs to obtain domain information for input data, and Oracle needs to ac- cess references, so DM and Oracle are experimental settings.
'Appendix C shows the results for metrics other than MS- COMET-22. Appendix D shows the results across all model and domain combinations. Appendix shows results on devel- opment data. Appendix F shows examples of DEMs output.

---

Data) KFTT
LAW
TED
ASPEC
BSD
DatalMModd
KFTT
LAW
TED
ASPEC
BSD
KFTT
155
92
102
118
128
KITT
127
120
117
105
123
LAW
97
260
97
106
101
LAW
119
211
133
96
117
TED
105
104
112
66
132
TED
95
141
76
151
ASPEC
99
100
79
134
ASPEC
127
115
149
BSD
98
116
105
100
179
BSD
71
134
104
195
Table 3: Confusion matrix of domains selected by DEMs (Oracle) in the En-Ja test data. Each number count. If multiple models get the same best score, they get a count.
Table Confusion matrix of domains selected by DEMs (QE) in the En-Ja test data. Each number count. If multiple models get the same best score, they all get count.
Data'l Model
KFTT
LAW
TED
ASPEC
BSD
which selects a single high-quality output, would surpass MBR, which chooses the sentence most se- mantically similar to others in Y. Therefore, im- proving QE methods is crucial for the success of DEMs setting. Even the expert model that matches the input domain can generate low-quality output. In such cases, MBR and QE select better output from other models. This may account for the DM results being lower than the MBR and QE.
KFTT
128
86
111
136
160
LAW
148
133
151
179
141
TED ASPEC
152
130
70
84
105
135
112
77
73
127
BSD
153
155
95
158
99
Table 5: Confusion matrix of domains selected by
DEMs (MBR) in the En-Ja test Each number is a count. If multiple models get the same best score, they all get count.
ENS scored lower than QE and MBR. This result suggests that in the DEMs setting, ENS is negatively impacted by models with poor performance. Hence, weighted ENS could improve the performance (Dur- rani et al., 2016; Sajjad al.. 2017). However, determining the appropriate weights typically ne- cessitates training or tuning, and any alteration in the number domains requires a reconfiguration of these weights. QE and MBR would be more flexi- ble than the ENS in multi-domain MT tasks.
performance superiority of DEMs: 1) the of multiple models, and 2) domain-based data sep- aration. To verify these two factors separately, we prepared five MDMs trained with various ran- dom seeds. We then evaluated the performance of QE, MBR, ENS, and Oracle, which are the identi- cal methods used for DEMs. Note that the initial model for fine-tuning is identical across all models in both DEMs and MDMs, with differences emerg- ing solely in the fine-tuning data. Table 6 shows the performance DEMs and MDMs on En-Ja test data.
Selected outputs in Oracle, QE, and MBR: Ta- bles 3, and 5 show which expert model's outputs were selected by Oracle, QE, and MBR for each do- main of input in En-Ja, respectively. For Oracle, the corresponding expert model's output for the input domain was consistently the most selected, but this was not the case for MBR and QE. Even in Oracle, outputs from models other than the expert corresponding the input were often selected. This suggests that the DM did not lead high perfor- mance. We assume that this is an effect of including instances whose domain is not far distant from the different domain data. Since Oracle simply selects good-quality sentences without being aware of the domain, it can select the appropriate model's out- put flexibly. This appears to explain the significant difference in scores between Oracle and DM.
In MDMs, the average score (AVG) of the five models was lower than all other methods (QE, MBR, ENS, Oracle). This indicates that the use of multi- ple models is effective. Also, the output selection methods (QE and MBR) scores were higher than ENS.
This result suggests that the output selection ap- proach more effective than the typical ensemble approach, even when each model is similar (since the MDMs have the same training data and differ only in seed).
Comparing DEMs and MDMs. it is intriguing note that DEMs outperformed MDMs in all meth- ods except AVG, even though DEMs uses less data for fine-tuning than MDMs. In addition, none
Efffett of multiple models and data separating: As discussed in the previous paragraph, we can as- sume that there are two contributing factors to the
"Each model in MDMs was fine-tuned one epoch for each model (about 1.5 times the number of updates for each model in DEMs). Appendix G shows the results when training model in MDMs setting with different epochs.

---

AVG
Oracle
91
OF MBR
ENS
DM
XOracee MDM
DEMs
77.20
79.87
80.97
78.17
81.78
90
89.87 X
MDMs
78.15
78.61
78.63
78.15
78.97
wm222-comet-
diff
-0.95
+1.26
+2.34
+0.02
+2.81
89
88.89
88. 06
88.01 87.96 f
88
87.68
87.21
Table Performance comparison DEMs and MDMs with SMALL En-Ja test data. The 'diff" row is the DEMs score minus the MDMs score. The scores of DEMs are same DEMS-SMALL 90M x in Figure 2.
87
86.48
86
86
87.07
86.84
87.89
86
85,99
85
MDM
MIM
DEMS
MDM
DEMS 290M
Large
90M
290M 9UM x Model
Figure Evaluation results for En-Ja test data. QE conducted with wm122-cometkiwi-de and MBR, Oracle, and evaluation was conducted with wmt22-comete
of the methods for MDMs exceeded the MDM- LARGE score (79.78) in Figure 2. These results suggest that domain-based data separation is effec- tive. The difference between DEMs and MDMs scores for the output selection methods (QE and MBR) stems from a heightened probability of in- cluding good outputs in the candidate set. This is because DEMs can more dependably acquire
4 QB MBR DDM Oraalee MDM
86
wmt12 -comeet-m
85
84.90 X
84
36 83,90
$2.96
82.
84
83
82.57
82. 52
81.85
speciic knowledge through domain- based
82
81.29
81.53
81.68
data separation.
81
81.83
80.80
80
Effects of selection model performance on DEMs: We conducted experiments using wmt22- cometkiwi-da (Rei et al., 2022b) and wmt22-comet- da (Rei et al., 2022a) in place of MS-COMET-QE- 22 and MS-COMET-22, respectively. We also used wmt22-comet
MDM
KDM DEMs
MDK
DLMs Base 290M
Sanil
Base Small
Large 1B
X Model size
Figure Evaluation results for Ja-En test data, QE was conducted with wmt22-comerkiwi-da. and MBR, Oracle, and evaluation was conducted with wm122-comet-da.
evaluation metric. Figures 4 and show the results for the En-Ja and Ja-En test data, respectively.
han al., 2016; Passban et al., 2022), for example. more comprehensive study will be conducted in the future.
Compared to the MS-COMET-QE-- 22 and MS- COMET-22 experiments shown in Figures 2 and 3, QE and MBR showed a smaller increase in scores. This result can be attributed to the fact that MS- COMET-QE-22 and MS-COMET-22 have been trained multi-domain data, discussed in Sec- tion 4.1, whereas umm222comerriwwidda and wmt22- comet-da are not. This indicates that the met- rics used in a multi-domain setting must multi- domain capable.
Conclusion
We investigated the effectiveness of MT models in multi-domain setting in fair conditions and ex- plored direction for tackling multi-domain MT. We mainly compared the performance of the two main approaches, MDM and DEMs, on English to/from Japanese tasks with five domains. Our results revealed that the domain tion strategy is effective in a multi-domain setting. Additionally, we found that when we want to use metrics for selection in multi-domain setting, the metrics must be multi-domaie capable. Further- more, combining five 90M DEMs based on an out- put selection approach has the potential perform better than single 1B MDM, which may suggest a new direction for developing MT models the multi-domain setting.
Potential of DEMs in practice: We believe that DEMs approach offers several advantages over MDM, especially in practical scenarios. For ex- ample, DEMs approach provides benefits for multi- organizational collaboration in cases where privacy issues preclude sharing training data. Since when QE and MBR applied in DEMs, it is enough to share only the model outputs. Although not ex- plored in our experiment, because DEMs would work even with different model architectures, may facilitate collaboration among multiple orga- nizations compared to Federated Learning (McMa-
data separa-

---

Limitations
work was supported by JST Moonshot R&D Grant Number JPMJMS2011 (fundamental research) and JST CREST Grant Number JPMJCR20D2.
Our experiments were conducted only on the En- glish to/from Japanese translation task. Also, we only experimented with five domains. The num- ber of domains increases, as does the number of expert models, and the running cost of the DEMs in- creases as well. In addition, both MDM and DEMs approaches can be applied to tasks other than trans- lation. However, in our experiments, DEMs de- pends on the performance of MS COMET22222 and MS-COMET-QE-22, and in tasks where such tools are not available, our scenario used may not be applicable. Furthermore, we did not include in our comparisons any studies such as domain tagging, training data sampling, or model architecture that would improve the performance of MDM, as de- scribed in Section 2. Experimentation in more comprehensive setting is future work.
References
Denny Britz,, Quoe Le, and Reid Pryzant. 2017. Effec- tive Domain Mixing for Neural Machine Translation. In Proceedings of the Second Conference on Machine Translation (WMT), pages 118-126.
Mauro Cettolo, Christian Girardi, and Marcello Fed- crico. 2012. WIT3: Web Inventory of Transcribed and Translated Talks. In Proceedings of the 16th Annual Conference of the European Association for Machine Translation (EAMT), pages 261-268.
Nadir Durrani, Hassan Sajjad, Shafiq Joty, and Ahmed Abdelali. 2016. A Deep Fusion Model for Domain Adaptation in Phrase-basse MT. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics (COLING), pages 3177- 3187.
We discussed the trade-off between the perfor- mance and size of the models in MDM and DEMs. Model compression methods such as quantization and knowledge distillation could impact the trade- off. However, such compression techniques are outside our focus.
Patrick Fernandes, António Farinhas, Ricardo Rei, José G. C. de Souza, Perez Ogayo, Graham Neubig, and Andre Martins. 2022. Quality-Aware Decoding for Neural Machine Translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies (NAACL-HLT)), pages 1396-1412
To achieve good results with DEMs settings, there needs to be at least one high-quality candi- date among the expert model' outputs. If the performance of expert models inadequate, the output selection methods discussed might not be as effective.
Haoming Jiang, Chen Liang, Chong Wang, and Tuo Zhao. 2020. Multi-Domain Neural Machine Transla- tion with Word-Level Adaptive Layer-wise Domain Mixing. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics (ACL), pages 1823-1834.
In our experiments, models were evaluated us- ing only automatic evaluation metrics. While we recognize that human evaluation is necessary to perform precise evaluation in translation task, we only performed automatic evaluation due the large number of models to compare.
Fabio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. 2019. OpenKiwi: An Open Source Framework for Quality Estimation. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics (ACL), pages 117-122.
Ethics Statement
Hayato Kohayashi. 2018. Frustratingly Easy Model En- semble for Abstractive Summarization. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4 4175-4176.
Our experiments were conducted only on the datasets already published to the community and well-studied in many previous studies. Moreover, even though our target task is machine translation, which involves text generation, we cannot believe the methods presented here further amplify biases for any other ethical concerns that implicitly exist in the datasets and tasks. Therefore, we foresce no ethical concerns in this work.
Catherine Kobus, Josep Crego, and Jean Senellart. 2017. Domain Control for Neural Machine Translation. Proceedings of the International Conference Recent Advances in Natural Language Processing (RANLP). pages 372-378.
Tom Koemi, Hitokazu Matsushita, and Christian Fed- ermann. 2022. MS-COMET: More and Better Hu- man Judgements Improve Metrie Performance. In Proceedings of the Seventh Conference on Machine Trunslation ( (WMT), pages 541-548.
A cknowledgements
We would like to thank the members of the Tohoku NLP Group for their insightful comments. This

---

Philipp Koehn. 2004. Statistical Significance Tests for Machine Translation Evaluation. In Proceedings of the 2004 Conference Empirical Methods in Natu- ral Language Processing (EMNLP), pages 388-395.
Association for Computational Linguistics (NAACL). pages 53.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a Method for Automatic Eval- uation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics (ACI.), pages 311-318.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 66-71.
Peyman Passban, Tanya Roosta, Rahul Gupta, Ankit Chadha, and Clement Chung. 2022. Training Mixed- Domain Translation Models via Federated Learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (NAACL-HLT), pages 2576-28 2586.
Shankar Kumar and William Byrne 2004. Minimum Bayes-Rist Risk Decoding for Statistical Machine Trans- lation. In Proceedings of the Human Language Tech- nology Conference of the North American Chapter of the Association for Computational Linguistics (IILT- NAACL), pages 169-177 196-
Stephan Peitz, Sarthak Garg, Udhay Nallasamy, and Matthias Paulik. 2019. Cross+Self-Ateention for Transformer Models. https://github com/ pytooch/fairrse/ffless/// /ttpll/e//551122//ppee/ 2wtp/////pppppppp
Minh-Thang Luong and Christopher Manning. 2015. Stanford neural machine translation systems for spo- ken language domains. In Proceedings of the 12th International Workshop on Spoken Language Trans- lation (IWSLT), pages 76-79.
Minh-Quang Pham, Josep Crego, and François Yvon. 2022. Multi Moltii-DDmain Adaptation in Neural Machine Translation with Dynamic Sampling Strategies. Proceedings of the 23rd Annual Conference of the Eu- ropean Association for Machine Translation (EAMT), pages 13-22.
H. B. McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. 2016. Communication Effficient Learning of Deep Net- works from Decentralized Data. In International Conference on Artificial Intelligence and Statistics (AISTATS).
MinhQuang Pham. Josep Maria Crego, and François Yvon. 2021. Revisiting Multi-Domain Machine Translation. Transactions of the Association for Com- putational Linguistics (TACL), 9:17-35.
Makoto Morishita, Katsuki Chousa, Jun Suzuki, and Masaaki Nagata. 2022. JParaCrawl v3.0: A Large- scale English-Japanes Parallel Corpus. In Proceed- ings of the Thirteenth Language Resources and Eval- uation Conference (LREC), pages
Maja Popović. 2015. chrF:
character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation (WMT), pages 392-395.
Toshiaki Nakazawa, Manabu Yaguchi, Kiyotaka Uchi- moto, Masao Utiyama. Eiichiro Sumita, Sadao Kuro- hashi, and Hitoshi Isahara. 2016. ASPEC: Asian Scientific Paper Excerpt Corpus. In Proceedings of the Tenth International Conference on Language Re-
Matt Post. 2018. Call for Clarity in Reporting BLEU Scores. In Proceedings of the Third Conference on Machine Translation (WMT), pages 186-191.
sources and Evaluation (LREC), pages 2204-2208.
Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020. TransQuest: Translation Quality Es- timation with Cross-lingual Transformers. In Pro- ceedings of the 28th International Conference on Computational Linguistics (COLING), pages 5070- 5081.
Graham Neubig. 2011, The Kyoto Free Translation
Task. http://www.hhotroo.comkkkk
Mariana Neves, Antonio Jimeno Yepes, Amy Siu, Roland Roller, Philippe Thomas, Maika Vi- cente Navarro, Lana Yeganova, Dina Wiemann, Gior- gio Maria Di Nunzio, Federica Vezzani, Christel Gerardin, Rachel Bawden, Darryl Johan Estrada, Salvador Lima-lopez, Eulalia Farre-maduel, Mar- tin Krallinger, Cristian Grozea, and Aurelie Neveol. 2022. Findings of the WMT 2022 Biomedical Trans- lation Shared Task: Monolingual Clinical Case Re- ports. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 694 723.
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C. Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a. COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task. In Proceedings of the Seventh Conference on Machine Translation (WMT). pages 578-585.
Myle Ott. Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of the 2019 Conference of the North American Chapter of the
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A Neural Framework for MT Evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685-2702.

---

Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. Martins. 2022b. CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task. In Proceedings of the Seventh Conference Ma-
chine Translation (WMT), pages 645.
Matiss Rikters, Ryokan Ri, Tong Li, and Toshiaki Nakazawa. 2019. Designing the Business Conversa- tion Corpus. Proceedings of the 6th Workshop on Asian Transiation ( (WAT), pages 54-61.
Hassan Sajjad, Nadir Durrani, Fahim Dalvi, Yonatan Belinkov, and Stephan Vogel. 2017. Neural Machine Multi-Domain Scenario. In Proceedings of the 14th International Conference on Spoken Language Translation (/WSLT), pages 66 66-73.
Translation Training
Danielle Saunders. 2022. Domain adaptation and multi- domain adaptation for neural machine translation: survey. Journal of Artificial Intelligence Research, 75:351-424.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning Robust Metries for Text Genera- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties (ACL),
pages 7881-7892.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving Neural Machine Translation Mod- els with Monolingual Data. In Proceedings the 54th Annual Meeting of the Association for Compu- tational Linguistics (ACL), pages 86-96.
Sander Tars and Mark Fishel. 2018.
Neural Machine Translation. In Proceedings of the 2tt Anmual Conference of the European Association for Machine Translation (EAMT), pages 279-288.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention All You Need. In Proceedings of the 31 International Conference Neural Information Processing Sys- tems (NIPS), page 6000 6010-6010.

---

A Details of dataset construction
coding, beam size of and maximum length of 200.
For fine-tuning on KFTT, TED, and BSD, we used the original training data. The original LAW con- tains 262,449 instances without distinction between train, development, and test, we used all but the first 2,000 instances for fine-tuning. The original ASPEC contains 3,008,500 instances in the train- ing set, but we used only the first 200,000 instances for fine-tuning.
C
Results of performance comparison by multiple metrics
Tables 9 and 10 show the results with MS- COMET-22, BLEU (Papineni et al., 2002). chrF (Popović, 2015), and BLEURT (Sellam et al., 2020) for the En-Ja and Ja-En test data. We use sacreBLEU (Post, 2018) for the BLEU and chrF calculations. The sacreBLEU signatures are nrefs: case: mise: mixed exp version:::3:
We created the development and test data by concatenating the five datasets used for fine-tuning: KFTT, LAW, TED, ASPEC, and BSD. They were each extracted from the top 500 instances after duplicates were removed from each raw data. Con- sequently, there were 2,500 unduplicated instances each for development and test data. For the raw data on KFTT, ASPEC, and BSD, we used the orig- inal development and test data. the case of the raw data on TED, we used dev2010 for develop- ment data and tst2015 for test data. For the raw data on LAW, we used the first 1-1000 instances of the original data for development data and the next 1001-2000 instances for test data.
tok: 1 3a smooth and
off:yes!! Inc: 6 Inw: 0 0 ce: ce:nol ve:sonvers:::: respectively. We used BLEURT-20 for BLEURT.
D Results for all combinations of domains and models
Tables 11, 12, 13 and 14 show the results with MS-COMET-22 for all combinations of data and models.
We tokenized all data into subwords processing step. We trained subword segmen- tation models using byte pair encoding on the JParaCrawl v3.0 as the training dataset for En- glish and Japanese, respectively. In training, we applied byte fallback, ensuring essentially the ab- sence of unknown words in the language used for any data, including fine-tuning data. We chose vocabulary size of 32k for each language. We used Sentencepiece tool (Kudo and Richardson, 2018) for these implementations.
pre-
E Results of MS-COMET-22 for development data
Figure 6 shows the evaluation results by MS- COMET-22 for the En-Ja development data, and Figure 7 shows the evaluation results by MS- COMET-22 for the Ja-En development data. For both En-Ja and Ja-En, the results for the develop- ment data showed a similar trend the results for the test data.
83
AQ MBR ENS
DM
Oracle
MDM
82.47 x
Details of model configurations
MS-COMET-22
81.72
81.50 +
81
The basic model configuration was the transformer defined in fairseq toolkit (Ott et al., 2019). From the default settings, moved the layer normaliza- tion to the beginning of each transformer block, made sharing the input and output embedding the decoder, and implemented the cross+self- attention mechanism (Peitz et al., 2019).
80.75
80
79.46
80.29 A
79
78.33
78.90
A 79.52
79.24
78.48
77.69
77
MDN Small
MOM DEMS Hasc
MOM 18
DEMS Bagc
Small
90M
200M 90M
Model size
Figure 6: Evaluation results for En-Ja development data.
We trained neural MT models with the fairseq tool in version 02.2. In pre-training, we used the configuration for SMALL. BASE, and LARGE except for the learning rate (See Table 7). In fine- tuning, we used the same configuration for SMALL, BASF, and LARGE (See Table 8).
Examples of outputs for each expert model and output selection
Table 15 shows examples of each expert model's 's output and the result of output selections for the TED test data in En-Ja.
We used the same generation configuration for SMALL, BASE, and LARGE, with beam search de-

---

Configurations
Selected Value
Optimizer
Adam (3) = 0.9, 822 0.98, == x 10 10"*)
Weight Docay
Criterion
label smoothed cross entropy
Label Smoothing
0.1
Learming Rate
SMALL: 0.001, BASE: 0.0005, LARGE: 0.00025
Learning Rate Schedule
inverse square root
Warmup Steps
30000
Gradient Clipping
1.0
Dropout
0.1
Mini-bateh Size
512k
Number of Epochs
50
Averaging
Save checkpoint for every epochs and take average of last checkpoints
Table 7: List of hyperparameters in pre-training.
Configurations
Selected Value
Optimizer
Adam (91 = 0.9, 82 = 0.98, = 10
Weight Decay
0
Criterion
label smoothed cross entropy
Label Smoothing
0.1
Learning Rate
0,00001
Learning Rate Schedule
constant
Gradient Clipping
1.0
Dropout
0.2
Mini-hatch Size
16k
Number of Updates
MDM: 10000, DEMs: 2000
Averaging
Save checkpoint for every 100 updates and take an average of last 5 checkpoints
Table 8: List of hyperparameters fine-tuning.
COMET
BLEU
chrF
BLEURT
COMET
BLEU
chrF
BLEURT
QE 79.87
24.1
31.3
65.62
QE
77.23
22.4
52.2
66.02
MBR
80.97
22.3
30.1
65.05
MBR
78.35
22.5
52.0
65.98
DEMs- SMALL
DEMs- SMALL
ENS
78.17
24.6
31.6
65.55
ENS
75.50
23.6
52.5
66.11
DM
78.39
30.6
36.3
67.39
DM
76.27
28.1
55.8
67.76
Oracle
81.78
25.8
32.7
66.94
Oracle
79.31
25.0
53.9
67.88
QE
80.57
25.7
32.9
66.98
QE
78.12
23.5
53.6
67.25
MBR
81.73
24.4
32.0
66.42
MBR
79.31
24.0
53.6
67.57
DEMs- BASE
DEMs- BASE
ENS
79.13
26,9
33.8
67.24
ENS
76.75
25.3
54.2
67.57
DM
79.59
33.0
38.5
69.28
77.30
30.3
58.0
69.52
Dracle
82.61
28.2
35.0
68.94
Oracle
80.34
27.0
55.8
69.82
MDM SMMM-
78.10
29.8
35.6
66.18
MDM- 75.56
27.8
55.8
67.27
MDM-BASE
79.52
32.5
38.0
68.41
MDM-BASE
57.7
69.25
MDM-Large
79.78
34.1
39.4
69.30
MDM-Large
77.72
30.6
58.5
69.92
Table 9:
Performance comparison of all settings on En- Ja data using metrics COMET* (MS-CC MAS-COMET-22), BLEU, chri, and BLEURT.
Table 10: Performance comparison of all settings on Ja- En test data using metrics COMET* (MS-COMET-22), BLEU, chrF, and BLEURT.
G Whether the MDMs models are sufficiently trained
and additional increments in the number of epochs did not enhance the translation performance. There- fore, the MDMs model did not appear to under- trained, and it is more likely that domain-based separated data is more effective than multi-domain data for training.
We investigate the impact of the number of train- ing epochs the performance of the MDMs in the section 5 experimental setting. Table 16 shows the translation performance with one epoch, two epoch, and three epoch fine-tuning for the concate- nated data KFTT, LAW, TED, ASPEC and BSD. The epoch-1 demonstrated the highest performance,

---

Data' Model
MDM
KFTT
LAW
TED
ASPEC
BSD
MIX
78.10
-1.08
-0.70
-1.62
-1.11
-0.02
KFTT
77.06
-0.89
-2.37
-2.58
-1.37
-0.30
LAW
81.51
-2.54
+0.86
-4.51
-1.95
TED
73.31
-0.80
-0.35
+0.13
-0.57
+0.86
ASPEC
80.17
-0.16
-1.01
-1.46
+0.27
+0.21
BSD
78.47
-1.01
-0.60
+0.34
-0.86
+1.08
Table 11: Results for all combinations of data and mod- els with SMALL on En-Ja test data. each row, the score of each expert model is listed based on the MDM score.
Data Data Model
MDM
KFTT
LAW
TED
ASPEC
BSD
MIX
79.52
-1.39
-1.53
-1.86
-1.49
-0.77
KFTT
78.14
+0.09
-1.93
-1.92
-0.28
-1.16
LAW
83.01
-2.00
-0.35
-5.83
-2.66
-2.40
TED
75.70
-1.86
-2.04
+0.10
-1.75
-0.50
ASPEC
80.98
-0.91
-1.43
-1.54
+0.02
-0.30
BSD
79.76
-2.26
-1.89
40.13
-2.77
+0.51
A QE MBR - ENS B DM
Oralle MDM
81
Table Results els with BASE En-Ja Ja test data. In each the score of each expert model is listed based on the MDM score.
combinations of data mod-
80
80.03 *
MS-COMET-22
* 79.10
79.02
78
78.10
77.27
77.77 A
76.85 88
77
A 76.85
77.08 m
75.90
76.39
76
75.82
Data' Datal/Model
MDM
KFTT
LAW
TED
ASPEC:
BSD
75.23
Small 90M
Small 290M 90M x
Large
Rase
MIX
75.56
-0.24
-0.18
-2.13
-1.65
-0.21
Model
290M 5
KFTT
71.03
-0.02
-2.04
-5.96
-2.45
-2.33
LAW
78.73
-1.41
+0.45
-2.39
-2.71
-1.33
Figure 7: Evaluation results for Ja-En development data.
TED
73.67
-0.39
-0.15
+0.22
-1.41
+0.07
ASPEC
79.05
-0.03
+0.07
-1.10
-0.09
-0.43
BSD
75.33
+0.66
+0.77
-1.44
-1.58
+2.96
Table 13: Results
combinations of data and mod-
els with SMALL on Ja-En test data. In each row, the score each expert model is listed based on the MDM score.
Data Model
MDM
KFTT
LAW
TED
ASPEC
BSD
MIX
76.97
-0.46
-0.66
-2.56
-2.11
-0.30
KFTT
73.27
-0.35
-2.09
-6.06
-2.48
-2.38
LAW
79.57
-1.23
+0.29
-3.13
-3.06
-1.06
TED
75.03
-0.24
-0.80
+0.21
-1.98
+0.55
ASPEC
79.59
0.00
+0.10
-1.33
-0.26
-0.38
BSD
77.38
-0.46
-0.83
-2.49
-2.80
+1.78
Table 14: Results for all combinations of data and mod- els with BASE on Ja-En test data. In each row, the score of each expert model is listed based on the MDM score.

---

D22 Li:
souree
F. (i S
This is the Livingston Public Library that was completed in 2004 in my hometown, and, you know, it's got dome and it's got this round thing and columns, red brick. and you can kind of guess what Livingston is trying to say with this building: children, property values and history.
reference
KFTT model's output
Livingstone Public Library was completed 2004 my hometown.
Livingstone Public Library was completed 20004
Library was completed 2004 my hometown,
the Livingstonee
LAW model's output
The Livingstone Public Library was completed 2004.
The Livingstone Public Library was completed 2004.
The Livingstone Public Library was completed 2004 in my hometown,
TED model'ss output (selected by MBR, DM, Oracle)
with dome, a round decoration, a column, red brick,
and you'tn re going to see what Livingstone wants to tell you about in this building:
children, asset value, history.
ASPEC model'ss 's output
Livingstone Public Library was completed in my hometown in 2004.
The Livingstone Public Library was completed in 2004 in my hometown, and it's dome with round decorations and columns with red bricks Livingstone will be able to consider what you want to tell this building Children, asset value, history
BSD model's output (selected by (E)
Table 15: Examples of DEMS-SMALL'S output. The source and reference sampled from the TED dataset. The MBR, MD, and Oracle methods selected the expert model fine-tuned on TED, The QE method selected the output of the expert model fine-tuned on BSD.
AVG
QE
Oracle
epoch-1
78.15
78.61
78.63
78.15
78.97
epoch-2
-0.23
-0.23
-0.14
-0.14
-0.13
epoch-3
-0.12
-0.08
-0.02
-0.09
+0.04
Table 16: Performance comparison of SMALL with vary- ing training epochs in MDMs En-Ja test data.


Reviews:
[{"review_id": "WG9iqoemrG", "reviewer": "Reviewer_wUg4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your invaluable comments.\n\n## About experiments with more language pairs and domains\nAs discussed in the limitation, we also recognize that the language pairs are only English from/to Japanese and that only five domains are a limitation of our paper. \nHowever, we believe that our experimental setting is well designed for the research question and that our paper provides a useful case study because English from/to Japanese is one of the major language pairs in the translation task, which is included in the WMT General (News) translation task. We also believe that we have provided sufficient evidence to support our claims in light of the review policy for a short paper (https://aclrollingreview.org/reviewertutorial#4-how-to-read-for-review).\n\n\n## About examining the failure conditions of the winning strategy\n\nThank you for your valuable comments.\n\nDue to the nature of the Minimum Mayes Risk (MBR) algorithm, the inclusion of poor quality translations in the candidate set may result in a poor final score.\n\nIn the camera-ready version, we will add discussions by showing examples of instances where MBR failed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths above: clean, focused examination, results could be practical, interesting (consolidated) ideas for aggregating translations.", "reasons_to_reject": "Weaknesses above, for example, experiments with more language pairs, with loose domains.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper compares two model strategies for designing multi-domain systems: multi domain experts and their aggregation vs a single model that is trained on the mixed data. It introduces several ways of aggregating the domain experts. Experiments conducted with 5 domains show that multi experts strategy is a better fit in the scenario concerned in the paper. \n\nOverall, I feel that the paper's result is due to a well-defined scenario: general domain is not a concern in this setup, the 5 domains in experiments seem to be each a focused/concentrated domain. As a result, I wonder if there is a chance that mixing the data in this scenario may not further boot the performance of each -- gradient could even cancel each other if domains are distant. Therefore, I wish that paper provided more discussions and analysis in this regard.\n\nStrengths:\n- Paper's writing is clear. \n- Thorough experiments to support the final claim.\n- The idea proposed in paper will be helpful for practitioners with the same setup.\n\nWeaknesses:\n- Wish to see results in more language pairs, and looser (less concentrated domains).\n- Examine the failure condition of the winning strategy.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "A62TGbfPZX", "reviewer": "Reviewer_6uVB", "summary": "", "questions": "line 189 : you seem to be suprised by MBR outperforming QE, but according to line 156 y_MBR does optimise MS-COMET-22, which is your evalaution metric ( not sure how fair this comparison to other methods, which may not have access to the reference)", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive comments.\n\n## About the Minimum Mayes Risk (MBR) algorithm\nWe are afraid to say Reviewer #6uVB may have misunderstood that the computation of MBR needs the references. __Minimum Mayes Risk (MBR) uses reference-based metrics such as COMET, but does not require access to reference text, as described in Section 4.2 (line 156). Therefore, MBR is also a \"realistic\" approach.__\nFurthermore, as shown in Figure 2-5, the MBR achieves 90M x 5 DEMs (Domain Expert Models) outperforming 1B MDM (a Multi-Domain Model).\n\nIn the camera-ready version, we will improve the presentation to specify that MBR is the method that does not use reference texts. Moreover, we will clearly state that the practical methods are MBR, QE (Quality Estimation), and ENS (Ensemble), and that the virtual settings are DM (Domain Match) and Oracle. \n\nWe would appreciate it if you could reconsider the scores If misunderstanding about MBR primarily affected the decision of the scores.\n\n## About domain settings\nAs reviewer #6uVB mentioned, the domain is not very well defined in our experiment, and we agree that this is one of the reasons why DM is lower than QE and MBR. Domain is an arbitrary delimitation, and it is not easy to separate it exactly, and based on practical situations. Therefore, for convenience, we considered each dataset a different domain, which we believe is a very realistic setting. We will highlight these discussions in the camera-ready version.\n\n## About baseline settings\nThank you for the suggestion of many related works. We will add these in the related work section in the camera-ready version. As discussed in the limitation section, research on multi-domains is a very active research area, and we recognize that various approaches can improve MDM performance even more.  There would also be other approaches to DEMs, such as combining classifiers that predict domain, as you suggested. However, these comparisons are outside the focus of this study. \n\nOur research question is to compare output selection and ensemble strategies in a simple setting where the main experimental variables are model size and number of models.\n\n## Significance test\nWe calculated the statistical significance with Paired T-Test and bootstrap resampling (Koehn, EMNLP 2004), using the COMET library implementation.\nThe following tables show that MBR is significantly better than MDM.\n\n> If system_x is better than system_y then:\nNull hypothesis rejected according to t-test with p_value=0.05.\nScores differ significantly across samples.\n\n\n| system_x \\ system_y | QE | MBR | ENS | DM | Oracle | MDM-Small (90M) | MDM-Base (290M) | MDM-Large (1B) |\n|----|----|----|----|----|----|----|----|----|\n| MBR (En-Ja) | True | | True | True | False | True | True | True |\n| MBR (Ja-En) | True | | True | True | False | True | True | True |\n\n\n- [Statistical Significance Tests for Machine Translation Evaluation](https://aclanthology.org/W04-3250) (Koehn, EMNLP 2004)\n\n## Other evaluation metrics\nWe do not use BLEU or chrF because they have been reported in recent years to have low evaluation performance (Freitag et al., WMT 2022)  and are likely to mislead to conclusions. We chose MS-COMET as the evaluation metric because MS-COMET has the highest correlation with human judgment (please see the Table 14 and 15 of Freitag et al., WMT 2022). Moreover, MS-COMET is a suitable model for a multi-domain setting, as noted in footnote 5. We believe that MS-COMET is an appropriate metric in our experimental setting, but we will add several additional metrics in the camera-ready version.\n\n- [Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust](https://aclanthology.org/2022.wmt-1.2) (Freitag et al., WMT 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Paper is clearly written, experimental choices are well documented \n- Authors compare different ensembling strategies which could be of interest to some researchers\n - The finding that QE could be an interesting strategy for ensembling several models seems interesting (if the significance of these results is confirmed)", "reasons_to_reject": "IMO several important baselines are missing :  \n  - model with adapter layers seems  (and AdapterFusion ) seem to be a natural baseline in these settings\n  - we know that naive concatenation of different domains is subpoptimal baseline, several works have tried different sampling techniques for different domains (Eg. Pham et al. EAMT 2022) , it would have been useful at least to mention such works, and even better to use them as stronger baselines\n - one very simple baseline would be to train a classifier that would be able to predict the domain for input sentence, and report it as a more realistic alternative to DM version\n\nSignificance testing would be useful to asses the gains. Moreover, reporting the results in terms of other metrics (Eg. BLEU, chrf, BLEURT?) would have been helpful to asses the validity of the findings", "questions_to_authors": "line 189 : you seem to be suprised by MBR outperforming QE, but according to line 156 y_MBR does optimise MS-COMET-22, which is your evalaution metric ( not sure how fair this comparison to other methods, which may not have access to the reference)", "paper_topic_and_main_contributions": "This paper addresses the problem of domain-specific Machine Translation, and is specifically interested in the scenario when we have access to different domains data during training, but during inference the input domain is unknown. It compares  (1) default multi-domain model setting, where single model is trained on the concatenation of all the datasets coming from differeint domains and (2) different ensembling methods combining several (smaller) domain-specific models. Domain specific models are trained in 2 steps: first general model is trained on concatenation of all the data, then general model is finetuned on the domain of interest. The ensembling methods this work is comparing are \n1) Oracle: simply selecting the translation with highes MS-COMET-22 score \n2) QE: selecting the translation with highes MS-COMET-QE (referenceless) score\n3) MBR: minimum bayes risk optimisation relying on MS-COME-22 estimator\n4) ENS: each token probablilty is an avg probablity across all domain-specific models\n5) DM : provide model with golden domain and select appropriate domain during inference \n\nAuthors report results in MS-COMET-22 scores for en-ja and ja-en datasets containing 5 different domains and demonstrate that Oracle > MBR > QE > DM > ENS\n\nTo summarize, looks like that methods that have direct access to the final evaluation metric (Oracle, MBR) outperform the  methods that aim to optimize different metric (which is not that surprising in itself). The face that DM is lower than QE probably indicates that the notion of the domain is not very well defined ( as highligheted by Saunders 2022), and it is not enough to provide model with golden domain label to obtain optimal translation.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Very complete overview of recent related work by Saunders 2022 http://arxiv.org/abs/2104.06951\nPham et al, 2022 https://aclanthology.org/2022.eamt-1.4.pdf\nPham et al 2021 https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00351/97775/Revisiting-Multi-Domain-Machine-Translation\nGu, Feng 2020 https://arxiv.org/pdf/2011.00678.pdf"}}, {"review_id": "Z3pfd4kGN2", "reviewer": "Reviewer_Va6z", "summary": "", "questions": "If we selected different outputs of the MDM model using the MBR method,  can the MDM method achieve higher performance than DEMs?\nThis paper only compares two multiple-domain settings under English-Japanese translation tasks, which are dissimilar between the two languages. I wonder if we conduct experiments on similar languages,  can we achieve the same conclusion as the En-Ja translation task?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review.\n## About the practicality of DEMs settings\nWe believe that the DEMs (Domain Expert Models) approach is suitable for the real MT systems.\n\n(i) As Reviewer #Va6z mentioned, QE (Quality Estimation) and MBR (Minimum Mayes Risk) require external evaluation models. However, evaluation metrics such as COMET are available as open source, and access to them is easy. Also, for example, the use of external evaluation models is allowed in the WMT competition (https://www2.statmt.org/wmt23/translation-task.html).\n\n(ii) During inference, in a DEMs setting, each output of the expert model can be generated independently and can easily be processed in parallel. So, for example, DEMs with each model being small would be able to be inferred on multiple small-memory GPU machines. Although the appropriate approach will vary depending on the situation of computational resources and other factors, DEMs would be useful in a real-world setting.\n\n(iii) The DEMs setting is one of the most common approaches in shared tasks such as WMT translation (Kiyono et al., WMT 2020).\n\n(iv) We believe DEMs would be beneficial in a multi-organizational collaboration scenario, as discussed in lines:260-271.\n\nWe add these discussions in the camera-ready version.\n\n- [Tohoku-AIP-NTT at WMT 2020 News Translation Task](https://aclanthology.org/2020.wmt-1.12) (Kiyono et al., WMT 2020)\n## About the appropriateness of using an external evaluation model in a fair comparison of MDM and DEMs\nOracle is a setting to know the upper bound in the DEMs approach, not a baseline for fair comparisons. Although QE and MBR require an external evaluation model, our research question is to investigate the performance improvement of those evaluation models in the DEMs setting. We believe that our experiment setting is a fair comparison for our research questions.\n\n## Applying MBR to MDM\nIt is expected that applying MBR to MDM (a Multi-Domain Model) will also improve performance (Fernandes et al., NAACL 2022). Also for DEMs, increasing the number of hypotheses generated by each model may improve performance. Note that our experiment was conducted in a setting where each model has a single hypothesis for both MDM and DEMs.\n\n- [Quality-Aware Decoding for Neural Machine Translation](https://aclanthology.org/2022.naacl-main.100) (Fernandes et al., NAACL 2022)\n## About experiments with more language pairs\nAs discussed in the limitation, we also recognize that the fact that the language pairs are only English from/to Japanese is a limitation of our paper. There is no doubt that our paper would be better if we experimented with more language pairs.\nHowever, we believe that our experimental setting is well designed for the research question and that our paper provides a useful case study because English from/to Japanese is one of the major language pairs in the translation task, which is included in the WMT General (News) translation task. We also believe that we have provided sufficient evidence to support our claims in light of the review policy for a short paper (https://aclrollingreview.org/reviewertutorial#4-how-to-read-for-review).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper sheds light on the underexplored domain expert models (DEMs) and verifies the hypothesis that DEMs have the potential to achieve superior performance if each expert model is well-trained for the corresponding domain and properly combined. This paper conducts experiments on different output selection approaches and shows under what method DEMs outperform MDMs.", "reasons_to_reject": "The authors only conduct experiments on En-Ja translation tasks, which is not sufficient to support the conclusion in this paper.\nIn the DEMs method, the output of different models should be selected or ensembled in the postprocess period, which is not suitable in the actual NMT system.\nTo acquire results of QE and Oracle methods, the external evaluation model or references would be relied on. I do not think it is a fair comparison.", "questions_to_authors": "If we selected different outputs of the MDM model using the MBR method,  can the MDM method achieve higher performance than DEMs?\nThis paper only compares two multiple-domain settings under English-Japanese translation tasks, which are dissimilar between the two languages. I wonder if we conduct experiments on similar languages,  can we achieve the same conclusion as the En-Ja translation task?", "paper_topic_and_main_contributions": "The focus of this paper is to compare the approach of training a single machine translation (MT) model that covers all domains and the approach that aggregates models of multiple domains under a fair condition. The authors conduct experiments on multiple domain datasets and demonstrate that a combination of smaller Domain Expert Models (DEMs) can outperform a single larger Multi-Domain Model(MDM).", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a study comparing the performance of different combinations of domain-expert models against a big model trained on a combination of the data of all domains. All the reviewers agree that the work is sound. One common concern raised by the reviewers is that results are only presented on one language pair (en-ja), however the authors pointed out that according to the guidelines this should be enough for a a short paper and I agree with that. It presents some initial experiments that can trigger further research.

It is also worth noting that there was some confusion among the reviewers concerning the access to references. The authors addressed these issues in their response and I want to reiterate this here: MBR and QE do *not* have access to the references, thus they are fair methods to include in the comparison. Oracle does have access to the references, and therefore it is marked as such and provides an upper bound to the performance of the model. This does not compromise the soundness of the paper.

The weak point of this work in the excitement. None of the reviewers was overly enthusiastic, and I tend to agree with their judgement. There are no new methods presented in the paper, and it mainly consists of the experimental comparison between the presented ones. It can constitute a useful resource for practitioners, but I'm a bit reluctant to allocate a slot in the main conference for it. I think Findings would be a better destination, where it can constitute a reference to trigger further research.
Decision: Accept-Findings