An Empirical Study of Translation Hypothesis Ensembling with Large Language Models
Authors: António Farinhas
José G. C. de Souza
Andre Martins
Venue: EMNLP 2023 Main (2023)
Keywords: Translation
Large Language Models
hypothesis ensembling
reranking
MBR decoding
TL;DR: We investigate how hypothesis ensembling can improve the quality of the generated text for LLM-based machine translation.

Abstract: Large language models (LLMs) are becoming a one-fits-many solution, but they sometimes hallucinate or produce unreliable output. In this paper, we investigate how hypothesis ensembling can improve the quality of the generated text for the specific problem of LLM-based machine translation. We experiment with several techniques for ensembling hypotheses produced by LLMs such as ChatGPT, LLaMA, and Alpaca. We provide a comprehensive study along multiple dimensions, including the method to generate hypotheses (multiple prompts, temperature-based sampling, and beam search) and the strategy to produce the final translation (instruction-based, quality-based reranking, and minimum Bayes risk (MBR) decoding). Our results show that MBR decoding is a very effective method, that translation quality can be improved using a small number of samples, and that instruction tuning has a strong impact on the relation between the diversity of the hypotheses and the sampling temperature.

An Empirical Study of Translation Hypothesis Ensembling with Large Language Models
António Farinhas 1.2
José G. C. de Souza André F. T. Martins 1.2.3
Instituto Superior Técnico (Lisbon ELLIS Unit)
Instituto de Telecomunicações Unbabel
{antonio. far.o.fariii farinhas, andre. tondre.t.mrrtiiiii.. mamamian)tteecmieec ulisboa. pt. jose. wwwza@unbabel..comm
Abstract
models, the same cannot be said for LLM-based MT. There is, however, a lack of exploration in this casc. Our paper fills this gap by providing a com- prehensive study on ensembling translation hy- potheses ($2), encompassing multiple LLMs such as ChatGPT, LLaMA, and the instruction-tuner Alpaca (Taori et al., 2023). We consider different strategies to generate hypotheses (prompt-base based en- sembling. temperature sampling, beam search), and several techniques to produce the final translation, including ChooseBest, GenerateBest, reranking based on quality estimation, and minimum Bayes risk decoding. The last two approaches have been successful at improving translation quality with task- models (Fernandes et al., 2022; Fre- itag et al., 2022a), but it is unclear whether the findings hold for LLM-based MT.
Large language models (LL.Ms) are becoming mne-fits-many solution, but they sometimes hal- lucinate produce unreliable output. In this pa- per, we investigate how hypothesis ensembling can improve the quality of the generated text for the specific problem of LLM- based machine translation. We experiment with several tech- niques for ensembling hypotheses produced by LLMs such as ChatGPT, LLaMA, and Alpaca. We provide comprehensive study along multi- ple dimensions, including the method to gener- ate hypotheses (multiple prompis, temperature- based sampling, and beam search) and the strat- egy to produce the final translation (instruction- based, quality- baseed reranking, and minimum Bayes risk (MBR) decoding). Our results show that MBR decoding is a very effective method, that translation quality can be improved using a small number of samples, and that instruc- tion uning has a strong impact on the relation between the diversity of the hypotheses and the sampling temperature. Our code is avail- able at https://githhb. com/ demp/spi/ araasslation-hypothee setppppteeeei-eeeelmllllllllii ensembl ing.
Our main findings can be summarized as fol- lows. First, we demonstrate that translation qual- ity can be enhanced with small number of samples (e.g.. 20), especially when translating out of English (Fig. 1). In the case of ChatGPT, the cost in terms of paid tokens grows sublincarly with the number of samples ($3.2). Second, we show that similar findings apply to LLaMA and Alpaca ($3.3). We discuss in which conditions beam search remains a reliable baseline for single-hypothesis translation ($3.3.1) and how ensemble transla- tions ($3.3.2). Moreover, we find that there exists significant gap in the quality of ensembles of un- biased samples from LLaMA and Alpaca ($3.3.3). We attribute this disparity to how instruction tun- ing affects the relationship between the diversity of the hypotheses and the sampling tempera- ture, which ultimately impacts translation quality. Lastly, show that hypothesis ensembling duces the number of generated hallucinations, thereby improving the model's robustness source perturbations ($3.3.4). Ensembling predictions and increasing the model size narrows the quality gap between these models and ChatGPT.
1 Introduction
Significant research effort has been devoted to task- specific neural machine translation (NMT) models trained in a fully supervised manner with large vol- umes of parallel data. Their performance has been enhanced through techniques such as fine-tuning on in-domain data, model ensembling, and rerank- ing during decoding (Koemi et al., 2022). The recent achievements of general-purpose large lan- guage models (LLMs) such as GPT and LLaMA (OpenAI, 2023; Touvron et al., 2023) offer a fresh perspective on the problem, demonstrating the fea- sibility of generating high-quality translations with- out explicit training for the specific task, even in challenging zero-shot scenario (Hendy et al., 2023),
While techniques such as greedy decoding or sampling from the distribution often prove inade- quate for generating translations with tpec-fpe

---

COMET
BLEURT
shot scenarios where no examples are provided in the prompt, we can consider (1) choosing sin- gle prompt and sampling with a temperature such that the resulting predictions are diverse, (2) fixing the sampling temperature and considering multiple prompt templates, or (3) choosing a single prompt that makes the model generate multiple predictions. Refer to App. A.1 for specific prompt templates. While this paper does not cover in-context learning, these strategies can also be applied in few-shot sce- narios where in-context examples are provided in the prompt. In such cases, multiple prompts can be created by providing different in-context examples.
0e
A2en sken
2.2 Generating final translation
Let y C y be set of N hypotheses, possibly generated with one of the methods discussed $2.1. When it comes to providing a final output, a commonly used approach involves aggregating the hypotheses in Y by selecting the most frequent with majority voting (Wang al., 2023a). How- ever, this approach is not well suited for generation tasks such as MT given that the output consists sequence of multiple tokens. Therefore, we explore alternative methods that incorporate both external models ($2.2.1) and the LLM itself ($2.2.2).
Greedy Sampling
Ensemblingg mbr
Ensembling, ranking
Figure I: COMET and BLEURT scores for translations produced by ChatGPT. The greedy search output is indi- cated by a blue bold line, and single sample baseline by red bold line. Ensembles of multiple (20) predic- tions marked with dashed lines: orange for ranking with COMETKIWI and green for MBR decoding with COMET, Top: EN-X. Bottom: X-BN.
2 Ensembling Hypotheses
2.2.1 Using external models
Ensembling has long history in machine learn- ing, being well known for leveraging multiple com- plementary systems to improve performance given task and provide good/robust generaliza- tion (Hansen and Salamon, 1990; Ting and Witten, 1997; Breiman, 2001; Zhou et al., 2002). While there have been efforts in prompt ensembling and multi-prompt learning within the context of L.LMs, this area is largely unexplored for text generation tasks, where the output is string rather than just single token (Liu et al., 2023). See $4 for further details. In this section, we delve into the process of generating multiple translation hypotheses ($2.1) and explore different methods for ensembling them to produce a single translation ($2.2).
Assuming access to an external model f that pro- vides an estimated quality score for hypothe- sis y y without requiring ground truth (e.g.. COMETKIWI (Rei et al., 2022b)), simple ap- proach consists of choosing the hypothesis that maximizes this score (Fernandes et al., 2022),
Tranking = arg max f(y). DEY
(1)
Another alternative is minimum Bayes risk (MBR) decoding. which aims to find the output that maximizes an expected utility function u(y", y) that measures the similarity between a hypothesis y and a reference y E ) (Kumar and Byrne, 2002; Eikema and Aziz, 2020). For MT, this can be an automatic evaluation metric such COMET (Rei et al., 2020). MBR decoding seeks for
2.1 Generating multiple hypohess
There are several ways of generating multiple pre- dictions from a single language model. In zero-
Imbr == arg max Ey~pu(liz)) [u(Y,)), (2) DEÿ
where the expectation in Eq. is typically approxi- mated as a Monte Carlo (MC) sum,
'Ensembling predictions in this context should not be con- fused with the practice of model ensembling, which involves using multiple models (e.g.. with different initializations) and combining their outputs. In this paper, we focus on combining hypotheses generated from a single model. The framework re- mains valid if the hypotheses originate from different models.
Ey~po [u R3
Σ u(y 1, (3)
M

---

using M model samples ,...,,. (20))
(
A closed-source setting using ChatGPT², an LLM developed by OpenAI, which has been shown to provide high quality translation (Hendy et al., 2023; Peng et al., 2023a). The system is restricted behind API walls, with undisclosed training data/regime and limited documentation. According to their documentation, it is an In- structGPT model (Ouyang et al., 2022), trained follow instructions with reinforcement learn- ing from human feedback (Christiano al., 2017; Stiennon et al., 2020) using proximal policy opti- mization algorithms (Schulman et al., 2017).
yielding an unbiased MC estimate of the expected utility. Alternatively, we may obtain y(!),.... (M)) from temperature/nucleus sampling (Holtzman et al., 2020), resulting in a biased estimate. While the number of samples used to approximate the expected utility of each hypothesis can be smaller (Eikema and Aziz, 2022), we set M N.
2.2.2 Using the LLM
While the techniques above rely external mod- els for assessing quality, we also propose alterna- tive methods which do not need any external (task- specific) model. We consider two scenarios:?
An open-source scenario using LLaMA (Tou- vron et al., 2023) and Alpaca (Taori et al., 2023). The latter was finetuned from a LLaMA model on an cntrruction-followinn dataset with 52K ex- amples generated with text-davinci-003. fol- lowing Wang et al. (2023b). We use the versions with 7B parameters unless otherwise stated.
using the LLM to select the most appropriate hypothesis from y (formulated as a multiple choice question), which we call ChooseBest;
asking the LLM generate final prediction based on the hypotheses in y (i.e., a less re- strictive scenario where the model has the free- dom to either generate a new prediction or to choose one hypothesis from )), which we call GenerateBest.
As our translation baseline, we employ greedy decoding since it generally produces higher-quality outputs, in line with the findings of Peng et al. (2023a), who demonstrate that using a lower sam- pling temperature leads to improved performance." In this work, we use COMETKIW/ (Rei et al., 2022b) for ranking according to Eq. I and COMET(- 22) (Rei et al., 2022a) the utility function in MBR decoding, following Eq. 2. We consider different translation directions, including lan- guages such as English (EN), German (DE), Rus- sian (RU), Czech (CS), and Ukrainian (UK). We use the WMT22 test sets (Koemi et al., 2022), (4) which are recent, and thus less likely to have been part of ChatGPT's training (see footnote 4). Fol- lowing Freitag et al. (2022b), we evaluate each system with COMET(-22) (Rei et al., 2022a) and BLEURT (Sellam al., 2020).
The prompt templates for these methods are pro- vided in App. A.2.
2.3
Measuring hypothesis diversity
Inspired by the work of Fomicheva et al. (2020) quantifying model uncertainty, measure the semantic diversity between different translations for the same source sentence by computing
N
u(y ()) (0).
N(N
It is worth noting that when u is the same utility function used in Eq. 2, this quantity can be com- puted without any additional cost during inference with MBR decoding, as already provides scores for all the necessary pairwise comparisons.
3.2 Closed-source setting
We generate a set of translation hypotheses for each source sentence by sampling from the model with
3 Experiments
3.1 Setup
Aettps://ppenie compblog///eeeeeeceeeeeoco chatgpt. bott/blg/chhttpt... Our experi- ments were conducted with the gpt-3. 5-tuboo model be- tween April and June 2023.
We study different methods for generating transla- tions in two regimes:
"For more information, see httthtt//ppaateeee openai.. com/ com/docs /ocs//ccc////ccoo model- -en--iddxxeeee ror-reeearchers. According
Anotteer possibility. which we leave for future work, in- volves a sequential architecture that samples predictions non- independently by providing in the prompt the answers gen- erated in previous steps and taking the last prediction as the final output. This is known in statistics as stacking (Breiman, 1996) and is related to the work of Madaan et al. (2023).
this, ChaiGPT was trained on data from before Q4 2021.
"We encountered some APl/server errors when prompting ChatGPT for translation with a lemperature of 0, as reported by Guerreiro et al. (2023). Using temperature of 0.1 helps alleviate these issues, which we use as proxy for greedy decoding. This is not done when using I.LaMA/Alpaca.

---

EN-DE
EN-RU
EN-CS
EN-UK
N
METROD
COMET
BLEURT
Cost
COMET
BLEURT
Cost
COMET
BLEURT
Cost
COMET
BLEURT
Cost
Greedy
87.01
77.15
87.77
75.61
90.04 89.20
80.98
87.66
76.08
Sampling
86.56
76.67
87.15
74.62
79.81
86.63
74,13
using ChaiGPT
ChooseBest
87.15
77.38
87.77
75.47
7
90.21
$1.07
87.56
75.60
GenerateBest
87.16
77.69
5
87,30
75.17
6
90.05
80.89
87.54
75.59
asing external models
Ranking
87.58
77.70
88.72
76.70
91.02
62.14
88.
76.86
5
MIR decoding
87.77
77.71
3
88.88
76.37
3
91.37
81. 7.788
89.
76.87
COMET oracle
88.85
78.91
77.96
92.18
83.07
89.23
78.39
Ranking MBR decoding
87-64
77.86
8
88,96
76.89
10
91 40
82. 24.64
10
89.29
77.47
11
20
88
78.
89,41
76.73
10
82 45.45
10
90.03
77.77
COMET orucle
89.88
80.22
8
90.61
79.41
10
92.26
81.55
10
91.54
80.29
11
Ranking
78.06
19
89.17
77.17
24
91.52
82.80
23
89
50
MBR decoding
88.25
78.14
19
89.64
77.04
24
92.21
82,66
23
90.31
78.03
27
COMET oracle
90.39
80.89
19
91.74
80.66
24
93.75
85,29
23
92.70
81,22
27
DE-EN
RU-EN
CS-EN
UK-EN
COMET
BLEURT
Cost
COMET
BLEURT
Cost
COMET
BLBURT
COMET
BLEURT
Cost
Greedy
85.45
74.50
1
85.99
77.92
87.13
77.42
85.63 85.23
76.50
1
Sampling
85.18
74.06
1
85.68
77.48
86.72
76.88
75.88
using CharPP 77.82
5
ChooseBest
85.37
74.43
85,90
4
87.09
77.33
85.51
76.30
GenerateBest 85.46
74.54
4
85.66
77.56
87.08
77.48
85.46
76.40
axing models
Ranking
85.62
74.61
86.22
78.08
87.34
77 7788
85.85
76.72
5
MBR decoding
85.73
74.58
86,27
78.05
87.49
77.51
85.97
76.55
COMET oracle
86.83
75.73
2
87.46
79.57
2
88.65
79.12
2
87.27
78.27
2
Ranking
85.79
74.81
6
86.36
78.22
87.46
77.79
85.99
76.86
20
MBR decoding
85.95
74.69
6
86.49
78.27
5
87.70
77.71
6
86.22
76.77
COMET oracte
76.88
81.03
80,
79.66
Ranking'
85.79
74.78
13
86.32
78.13
12
87.47
77.67
13
85.
76.81
50
MBR decoding
86.03
74.80
13
86.60
78.39
12
87.79
71.73
86.28
76.87
COMET oracle
88.18
77.49
13
88.95
81,86
12
90.02
81,58
13
88.80
80.41
11
Table 1:
Automatic evaluation metrics for ChatGPT and total cost in terms of relative number of tokens normalized within each translation direction, rounded to the nearest unit. Sampling multiple predictions does not increase the prompt's cost, hence the total cost does not increase (approxinately) linearly with the number of samples. Ranking uses COMETKIWI and MBR decoding uses COMET. Best overall values are bolded.
temperature of (unbiased sampling) using the prompt of Hendy et (2023):
generating translations using different prompt tem- plates (sce App. A.1) yields translations of compa- rable quality. Furthermore, ensembling predictions from different prompts does not lead to improved results compared to ensembles generated using the same prompt. Therefore, we use only one prompt in our further analysis.
Translate this sentence from [source language] to [target language].
Source:
[source sentence].
Target:
We observe that using multiple samples results in at least one significantly higher-quality transla- tion compared to single prediction, as indicated by automatic evaluation metrics (see oracles for COMET in Table 1). Increasing the number of hy- potheses in the set consistently leads to an oracle translation of superior quality, thus highlighting the potential of ensembling predictions.
3.2.1 How should we ensemble translations?
We compare the methods introduced in $2.2, which include approaches that provide a final answer using only the LLM (ChooseBest and GenerateBest) or that require access to an ex- temall model (ranking with COMETKIW/ and MBR decoding with COMET as the utility function). Ta- ble shows the results for the automatic evaluation with COMFT and BLEURT, along with the relative cost of each method compared to greedy decoding. We normalize the values within each translation di-
Using different prompts. Based on preliminary experiments conducted on a subset of the language directions mentioned in 1, we observed that

---

EN-DE
EN-RU
EN-CS
EN-UK
N
METHOD
COMET
BLEURT
COMET
BLEURT
COMET
BLEURT
COMET
BLEURT
LLaMA greedy
77.33
62.86
71.57
50.80
71.56
52.33
69.06
Alpacu greedy
76.67
64.06
75.59
71.40
56.61
72.76
LLaMA beam
77.30
59.99
62.25
Alpaca beam
78.59
65.95
77.71
61.62
76.34
60.81
76.68
55.09
I.LMMA unbiased sampling
51.98
36.02
42.79
21.91
42.11
21.36
42.39
20.66
Alpaca unbiased sampling
68.15
54.86
61.50
44.15
54.90
37.45
56.80
36.44
LLaMA biased sampling
69.78
55.68
63.41
42.41
60.26
39.78
59.59
36.28
Alpaca biased sampling
73.42
60.65
70.25
53.19
63.97
47.70
66.45
45.91
umbiased sampling
LLaMA ranking
77.68
65.25
75.29
57.71
69.45
52.08
71.19
50.78
I.LaMA MBR decoding
79.45
63.78
76.85
54.70
72.02
49.11
73.18
48.25
Alpaca ranking
82.70
71.35
81.63
65.33
78.24
62.36
78.72
56.43
Alpaca MBR decoding
84.23
70.58
83,94
65.97
81.09
62.53
81.70
59.33
biased sampling
LLaMA ranking
83 04
71.91
82,93
67.41
81.07
66.88
81.12
62.24
LLaMA MBR decoding
84.06
69.84
83.72
64.75
82.87
63.61
82.01
60.14
Alpaca ranking
83.58
72.30
84.12
68.75
82.42
68.50
82.22
61.09
Alpaca MBR decoding
84.54
71.18
85.44
68.32
84.82
68.16
84.30
63.42
DE-EN
RU-EN
CS-EN
UK KK-
COMET
BLEURT
COMET
BLEURT
COMET
BLEURT
COMET
BLEURT
LLaMA greedy
82.36
70.19
81.58
71.62
81.26
69.90
81.37
71.13
Alpaca greedy
82.31
70.14
81.65
71.89
81.14
69.69
81.34
70.90
LLaMA beam
82,56
70.49
82.19
72.50
82,08
70.83
81.97
71.99
Alpaca beam
82.53
70.40
82.08
72.29
81.69
70.26
81.55
71.09
LLaMA unbiased sampling
73.26
60.23
70.63
58.62
70.19
57.20
70.72
59.13
Alpaca unbiused sampling
81.04
68.66
79.92
69.62
79.03
67.20
79.52
68.85
LLaMA biased sampling
79.82
67.60
78.75
68.14
78.10
66.07
78.71
67.84
Alpaca biased sampling
81.86
69.71
81.09
71.02
80.44
68.81
80.72
70.33
unbiased sampling
LLaMA ranking
82,90
70.64
82,12
71.74
81.85
69.97
82.17
71,70
T.I LMM MBR decoding
84.25
70.75
83.22
71.61
82.92
69.62
83.40
71.51
Alpaca ranking
83.97
72.22
83.62
74.07
83.75
72.79
83.40
73.47
Alpaca MBR decoding
84.47
71.78
83,95
73.50
84.00
71.78
83.58
72.46
50
biased sampling
I.I.MA ranking
84.03
72.15
83,44
73.73
83.58
72.32
83.50
73.47
LLaMA MBR decoding
85.03
72.17
84.22
73.30
84.4
71.87
84.23
72.82
Alpaca ranking
84.10
72.43
83.70
74.32
83,92
72.97
83.56
73.67
Alpaca MBR decoding
84.02
71.31
83.56
73.33
83.61
71.47
83.08
72.22
Table Automatic evaluation metrics for LLaMA (7B) and Alpaca (7B). Ranking uses COMETKIWI and MBR decoding COMET. Best overall values are bolded and best within each group are underlined.
rection by dividing by the cost of greedy decoding.
Using external models. Table shows that the two methods that use external models for ensem- bling predictions are effective at increasing the fi- nal translation quality over the baselines. Notably, these methods achieve significant improvements without requiring extensive number of unbiased samples from the model's distribution, especially when translating of English. Fig. provides visual representation of the gains achievable with 20 samples. This differs from the findings of previ- ous research using tpsk-spe NMT models (Fer- nandes et al., 2022; Freitag et al., 2022a), where is typically necessary to bias the model's dis- tribution using techniques like nucleus sampling (Holtzman et al., 2020) to train models without
Using ChatGPT. Although the performance of ChooseBest and GenerateBest with samples is slightly better than the single sample baseline, these approaches still fall short of both the greedy decoding output and the methods that use external models for selecting the final translation, according to both COMET and BLEURT. Furthermore, the incorporation of all translation hypotheses in the prompt (see App. A.2) significantly increases the cost, making these approaches less scalable. For that reason, we chose not pursue this direction further and instcad focused our efforts on exploring the methods described in 1.

---

label smoothing (Eikema and Aziz, 2020; Freitag al., 2022a), which often leads to an impractical increase cost due to the need for more translation hypotheses. MBR decoding consistently achieves the best results according to COMET across all translation directions. Although the differences in quality are small, this pattern does not hold when evaluating with BLEURT for EN-RU and EN-CS, for which ranking with COMETKIWI appears to have an edge.
LLaMA, greedy LLaMA, beam
Alpaca, greedy Alpaca,
Figure 2: COMET scores for LLaMA and Alpaca with greedy (blue) and beam search (orange). We represent ILaMA with solid lines and Alpaca with dashed lines. Left: EN-X. Right: X-EN.
3.3 Open-source setting
We obtain sets of both biased and unbiased trans- lation hypotheses for each source sentence from LLaMA and Alpaca. The former is obtained by sampling with temperature of 1, while the lat- ter uses temperature and nucleus sampling (with t 0.8 and p 0.95), which are the defaults for LLaMA. We use a variation of the prompt of Hendy et al. (2023) which stresses the transla- tion direction (crucial for the non instruction-tus LLaMA to understand the task) as follows,
EN-DE
EN-RU
EN-CS
EN-UK
Greedy
11.2
25.0
21.5
31.6
LLaMA
Beam
25.2
43.4
46.4
41.2
Ranking
1.3
2.5
3.6
11.3
MBR
6.4
6.4
8.8
12.1
Greedy
2.1
2.7
4.5
13.6
Alpaca
Beam
4.0
6.4
10.6
21.3
Ranking
0.4
1.8
3.1
17.1
MBR
1.4
0.8
2.6
10.7
Translate this sentence from [source language] to [target language].
Table 3:
Percentage of translations in the wrong tar- get language when translating from EN. Ranking with COMETKIWI and MBR decoding with COMET ased samples. We do not show the values for X-EN given that only a few translations (<1% for all lan- guages) are not in EN. Best overall values bolded and best for each model are underlined.
[source language] Source: [source sentence].
bi-
[target language] Translation:
While most works on translation with general- purpose LLMs typically present results using unbi- ased sampling or greedy decoding, as it has been observed that reducing the sampling temperature generally enhances translation quality (Peng et al.. 2023a), it is worth exploring the impact of using beam search (Reddy, 1977), the go-to search strat- egy for decoding with speeifpei models. Thus, in addition to greedy decoding, we employ beam search (with a beam size of 5) as a single hypothe- sis baseline. Notably, this is not possible for Chat- GPT ($3.2) because its API does not include beam search. We report results in Table 2, and use them to answer specific research questions next.
results not as favorable when applying beam search to the non instruction-tuned LLaMA for EN-X (particularly for EN-RU and EN-UK). contrast, for Alpaca, beam search is consistently better in both language directions.
We hypothesize that this discrepancy is related to LLaMA's relative inability to generate text in languages other than English. To validate this hy- pothesis, we automatically identify the language of the provided translations using the language iden- tification model of Joulin et al. (2016a,b). While both LLaMA and Alpaca, with both greedy and beam search, correctly provide translations En- glish when requested (X-EN), the same cannot be said for other languages (see Table 3). In particu- lar, LLaMA frequently provides translations in the wrong target language (mostly in English), espe- cially with beam search. Interestingly, ensembling predictions (e.g., with ranking or MBR decoding) effectively mitigates this issue. Providing a few in-context examples, which we leave for future
3.3.1 Greedy vs. beam search baseline
Fig. compares greedy search and beam search for both LLaMA and Alpaca for X-EN (right) and EN-X (left) translation tasks. For X-EN, beam search outperforms greedy search, with LLaMA achieving the highest overall quality. However, the
"Wo use the implementation in https: httee:////ttthhtcceee com/ facebookresdd wwwpoookkeeeercccoo wwweeeeeearreeeee hawph///ammaaaeeeeeeeeeeeeeeaeae aaree//lama..... _v..

---

work, another alternative that may help improve LLaMA's performance, alleviating the impact of decoding with alternative methods.
0.50
Diversity
0.25
3.3.2 How should ensemble translations?
0.00
Table 2 shows that, for all models and translation directions, single sample baseline not competi- tive at all with the greedy and beam search outputs, with the latter achieving the best overall quality, as discussed in $3.3.1. Sampling's poor performance is, however, more noticeable when translating from English. Given that the overall quality scores lower than that of ChatGPT (Table 1), in App. B we report results for larger version of LLaMA (with 30B parameters), for which we also observe performance gains from ensembling translations. Besides, App. C contains additional results consid- ering few-shot learning scenario where in-context examples are provided in the prompt.
83
COMET
80
72
BLEURT
0.0
0.2
0.4 Tempereture
0.6
0.8
1.0
ranking
diversity alpaca, diversity
alpaca, mbr
alpaca, ranking
Figure 3: Values for BLEURT (bottom) and COMET (middle) for MBR decoding with COMET (green) and ranking with COMETKIWI (orange), and diversity be- tween hypotheses (top; blue) as we increase the pling temperature for EN-DE. We represent LLaMA with solid lines and Alpaca with dashed lines. The dot- ted black lines (top) mark the increasing diversity gap.
For EN-X, ensembles of unbiased samples from LLaMA do not perform well, a topic will further study in $3...3. Overall, Alpaca performs better, and the final quality of the ensemble can be boosted by biasing the samples (although the difference is not very significant for EN-DE). MBR decoding with COMET attains the best results in terms of COMET, while ranking with COMETKIWT is better in terms of BLEURT for most language pairs.
the diversity between hypotheses, depicted by the blue lines, vary with the sampling temperature for LLaMA (solid lines) and for Alpaca (dashed lines). As expected, the diversity between hy- potheses increases as the sampling temperature in- creases. However, this occurs at a different rate for LLaMA and Alpaca, indicating that instruction tuning changes the relationship between hypothe- sis diversity and sampling temperature. Ultimately, this affects the final quality of the ensemble, which may help explain the aforementioned quality gap for ensembles of unbiased samples.
For X-EN, while biased sampling is still advan- tageous and the best results in terms of BLEURT are still obtained with Alpaca (ranking with ComEtKIWT) the best COMET scores are attained using LLaMA (MBR decoding).
3.3.3 Biasedness, diversity, and quality
There exists a significant gap the final quality of an ensemble of unbiased samples from LLaMA and Alpaca, especially in the case of EN-X X transla- tions, where LLaMA's performance is notably poor. For example, shown in Table 2, the disparity in COMET and BLEURT scores for EN-DE is and points, respectively. In this section, we study how instruction tuning influences the relationship between candidate diversity and sampling tempera- ture, and its impact on final translation quality. We consider translations from English to German (see App. D for the reversed direction) as a case study and measure translation diversity using the method described in $2.3, with COMET as the similarity function u in Eq.
An interesting observation is the noticeable in- crease the diversity gap (i.e., the length of the dot- ted black lines increases for temperatures ranging from 0.6 to 1.0 in Fig. 3), which coincides with di- vergence in the translation quality between LLaMA and Alpaca ensembles (the solid and dashed lines begin to separate). Additionally, it is worth not- ing that the optimal COMET scores are attained at candidate diversity of approximately 0.25 for both models; however, this optimum corresponds to different sampling temperatures for each model.
Overall, we conclude that instruction tuning has a notable impact on the relation between hypothe- ses diversity and sampling temperature, influencing
Fig. shows how the final translation quality, represented by the green and orange lines, and

---

EN-CS
EN-UK
pling multiple times or by using different prompts) with majority voting (Wang al., 2023a: Liévin et al., 2023; Diao et al., 2023), which not really suited for MT as argued before, or along with other complementary approaches (Wang et al., 2022; et al., 2022; Sun et al., 2023). There are several works on ensembling for NMT, where decoder uses multiple models (e.g., with different initializa- tions) and predicts output by averaging token- level predictions from each model (Sutskever al., 2014; 4; Chung et al., 2016), whereas our approach considers full translations from single model.
Greedy
10.2
21.2
ILaMA
Ranking
2.3
5.2
MBR decoding
6.9
7.4
Greedy
2.3
5.8
Alpaca
Ranking
0.8
4.1
MBR decoding
1.2
2.2
Table 4: Rate of ballucinations (the percentage over the number of sentences that passed the quality threshold for non-perturbed sources). Ranking with COMETKIWI and MBR decoding with COMET use bi- ased samples. Best overall values are bolded and best for each model are underlined.
Ranking/rescoring hypotheses. Garcia et al. (2023) train their own language models, sample multiple hypotheses and choose a final translation using MBR decoding, which has been shown to improve the translation capabilities of task-specific models (Fernandes et al., 2022; Freitag et al.. 2022a). Their work is significantly different from ours, since their models exclusively support two three languages
the final translation quality. Notably, it is simpler to set appropriate temperature for the instruction- tuned Alpaca, it is less sensitive such varia- tions. We observe that this effect is less pronounced when translating into English (refer to App. D), likely due to the higher inherent similarity between pppotheses-potentiallle attributable to the exten- sive English training data used for these models.
time. Similarly, the approach of Yang et al. (2022) includes a reranking stage (including two trained dedicated rerankers) and an edit stage, while Kadavath et al. (2022) ask mod- els to directly evaluate the probability that their self- self-generated answers are correct.
3.3.4 Hallucinations under perturbation
In this section, study how robust LLaMA and Alpaca are to perturbations in the source text by searching for hallucinations under perturbation, which correspond situations where a model pro- duces drastically different translations for unper- turbed and slightly perturbed inputs (Lee et al.. 2019; Raunak et al., 2021). We focus on EN-CS and EN-UK translations, given that hallucinations are typically more frequent when translating out of English and for lower resource languages. We follow Guerreiro et al. (2023) and apply the min- imal perturbations of Xu et al. (2023), including misspelling and title-casing words, and inserting frequent tokens at the beginning of the source sen- tence. See App. E for further details.
Editing/refining hypotheses. Raunak et al. (2023) explore translation editing with LLMs but they do not study how to use external models (e.g. COMET and COMETKIWI) to improve translation quality. Similarly, Chen et al. (2023) propose iteratively refine translations using LLMss, attaining comparable translation quality with the baseline, according to automatic translation metrics, and re- ducing (ranslationese, according to human study.
Conclusions and Future Work
Table 4 shows that the hallucination rates de- crease with instruction tuning for both EN-CS and EN-UK. Ensembling translation hypotheses further decreases the number of hallucinations, suggesting that considering multiple hypotheses is promising method for alleviating this issue.
We have conducted thorough empirical analy- sis on various techniques for generating transla- tions using LLMs. Our study encompasses eight datasets and three model classes, including closed- source and open-source models, the latter with and without instruction tuning. We have demonstrated that ensembling predictions significantly enhances translation quality and reduces hallucinations under source perturbations. Additionally, have discov- ered that instruction tuning affects the relationship between the diversity of sampled hypotheses and the sampling temperature, which in turn influences the final translation quality.
4 Related Work
Ensembling. Recently, Peng et al. (2023b) com- pare the effectiveness of different model ensemble strategies but focus on trained soft prompts and do not explore generation tasks. There is also work on ensembling predictions (produced by either sam-

---

There are several avenues for future research in addition the ones that have already been men- tioned previous sections. While ensembling pre- dictions produced by LLMs is effective at improv- ing translation quality, also presents opportunities for developing improved methods of uncertainty quantification and calibration (Fomicheva et al.. 2020; Tian al., 2023), crucial for addressing the inherent opacity of black-box LLMs.
consider better alternatives when presented with given text, resulting in the misclassification of isolated text as high quality (Bansal et al., 2021). Additionally, all evaluation benchmarks used this study are openly accessible, and annotators were allowed to label sensitive information when necessary. Lastly, it is important to note that all LLMs exhibit shared concern regarding their en- ergy consumption, particularly during the training phase (Strubell al., 2019).
Limitations
Acknowledgments
We highlight three main limitations of our work. First, we primarily focus on versions of LLaMA and Alpaca with 7B parameters, even though we have included additional results using a model with 30B parameters in App. B. It remains unclear how our findings, such as the effects of employing beam search ($3.3.1), ) or how instruction tuning influ- ences the relationship between sampling tempera- ture and hypothesis diversity ($3.3.3), generalize even larger models.
We would like thank Pedro Martins, Nuno Guerreiro, Ben Peters, John Mendonça, Duarte Alves, Sweta Agrawal, and the SARDINE lab team for helpful discussions. This work was built on open-source software; we acknowledge Van Rossum and Drake (2009); Oliphant (2006); Walt et al. (2011), and (Paszke et al., 2019). This work was supported by EU's Horizon Europe Research and Innovation Actions (UTTER, con- tract 101070631), by the project DECOLLAGE (ERC-2022-CoG 101088763), by the Portuguese Recovery and Resilience Plan through project C645008882- 00000055 (Center for Responsible AI), and by Fundação para a Ciência e Tecnologia through contract UIDB/50008/2000.
Second, we have included results from ChatGPT due to its proven ability to provide high-quality translation. Notably, CharGPT is restricted sys- tem accessible only through APIS, and its training data/regime are undisclosed. Since there is limited documentation, it is difficult to ensure that Chat- GPT did not encounter our evaluation benchmarks during training, even though they are recent ($3.1). Lastly, due to the high cost and time required for conducting a final human assessment of the translation quality, we have not included it in our evaluation. Instead, we try to address this issue by reporting results based multiple state-of-the-att automatic evaluation metrics for machine transla- tion, such as COMET and BLEURT. Despite these limitations, we believe that our findings hold sig- nificance for the ML/NLP community.
References
Gagan Bansal, Tongshuang Wu, Joyce Zhou, Ray- mond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? the effect ai explanations on complementary team performance. In Proceedings of the 2021 CHI Conference Human Factors in Computing Systems, CHI '21, New York. NY, USA. Association for Computing Machinery.
Leo Breiman. 1996. Stacked regressions. Machine learning, 24:49-64.
Ethics Statement
Leo Breiman. 2001. Random forests. Machine learning, 45:5-32,
ChatGPT and Alpaca have been finetuned using instructions and/or human feedback, which may be low-quality, contradictory, or adversarial, pos- sibly resulting in inherent biases (Fernandes et al., 2023). For example, instructions may lack speci- ficity, leading annotators to inadvertently evalu- ate slightly different task (Parmar et al., 2023). Another concern arises from using quality estima- tion/evaluation models such as COMFT KIWT and COMET, which have been finetuned on human pref- erences. In such cases, annotators may fail to
Pinzhen Chen, Zhicheng Guo, Barry Haddow, and Ken- neth Heafield. 2023. Iterative translation refinement with large language models.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Mar- tic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from humaa preferences. Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Junyoung Chung, Kyunghyun Cho. and Yoshua Bengio. 2016. A character-level decoder without explicit seg-

---

mentation for neural machine translation. In Proceed- ings the the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1693-1703, Berlin, Germany, Association for Computational Linguistics.
Yuan Gao, Ruili Wang, and Feng Hou. 2023, How to design translation prompts for chatgpt: An empirical study.
Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Fangxiaoyu Feng, Melvin Johnson, and Orhan Firat. 2023. The unreasonable effectiveness of few-shot learning for machine trans- lation.
Shizhe Diao, Pengeheng Wang, Yong Lin, and Tong Zhang. 2023. Active prompting with chain-of- thought for large language models.
Naman Goyal, Cynthia Gao. Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Kr- ishnan, Mare' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores- 1ores-101 evaluation benchmark for low-resource and multilingual ma- chine translation. Transactions of the Association for Computational Linguistics, 10:522-538.
Bryan Eikema and Wilker Aziz. 2020. Is MAP decoding all you need? the inadequacy of the mode in neural machine translation. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 4506 4520, Barcelona, Spain (Online). Inter- national Committee Computational Linguistics.
Bryan Eikema and Wilker Aziz. 2022. Sampling- -baee approximations to minimum Bayes risk decoding for neural machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 10978 11988,,9,,,8000 Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Nuno M. Guerreiro, Duarte Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. 2023. Hallucinations in large multilingual translation models.
Francisco Guzmán, Peng-Jen Chen, Mylc Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc' Aurelio Ranzato. 2019. The FLORES evaluation datasets for Now-resource ma- chine translation: Nepali-English and Sinhala- English. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLPP) pages 6098-6111, Hong Kong, China, Association for Com- putational Linguistics.
Patrick Fernandes, António Farinhas, Ricardo Rei, José G. C. de Souza, Perez Ogayo, Graham Neubig, and Andre Martins. 2022. Quality-aware decoding for neural machine translation. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1396-1444 1412. Seattle, United States. Association for Computational Linguistics.
L.K. Hansen and P. Salamon. 1990. Neural network ensembles. IEEF Transactions on Pattern Analysis and Machine Intelligence, 12100)9933-1001
Patrick Fernandes, Aman Madaan, Emmy Liu, António Farinhas, Pedro Henrique Martins, Amanda Bertsch, José G. C. de Souza, Shuyan Zhou, Tongshuang Wu, Graham Neubig, and André F. T. Martins. 2023. Bridging the gap: A survey on integrating (human) feedback for natural language generation.
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf. Vikas Raunak. Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at ma- chine translation? a comprehensive evaluation.
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Spe- cia. 2020. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539-555.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text de- generation. In International Conference on Learning Representations.
Armand Joulin, Edouard Grave, Piotr Bojanowski. Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016a. Fasttext.zip: Compressing text classification models. arXiv preprini urXiv:16/2.0330
Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022a. High quality rather than high model probability: Minimum Bayes risk decoding with neu- ral metrics. Transactions of the Association for Com- putational Linguistics, 10:811-2 10811-831 1-825.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016b. Bag of tricks for efficient text classification. arXiv preprint arXiv: I 667.01175.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Koemi, George Foster, Alon Lavie, and André F. T. Martins. 2022b. Results of WMT22 metrics shared task: Stop using BLEU- - neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46-68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan. Dawn Drain, Ethan Perez, Nicholas Schiefer. Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson,

---

Jackson Kernion, Shauna Kravec, Liane Lovitt, Ka- mal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know.
Mihir Parmar, Swaroop Mishra, Mor Geva, and Chitta Baral. 2023. Don't blame the annotator: Bias al- ready starts in the annotation instructions. In Pro- ceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 1779-1789, Dubrovnik, Croatia, As- sociation for Computational Linguisties.
Tom Koemi, Rachel Bawden, Ondiej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grund- kiewicz. Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita. Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović. 2022. Findings of the 2022 conference machine translation (WMT22). In Proceedings the Seventh Conference on Machine Translation (WMT), pages 1-45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computa- tional Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein. Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary Vito,, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, higo-perfoemance deep learning library. H. Wallach, H. Larochelle, A. Beygelz- imer, F. d'Alché-But Buc, E. Fox, and R. Garnett, editors. Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
Shankar Kumar and William Byrne 2002. Minimum Bayes-risk word alignments of bilingual texts. In Pro- ceedings of the 2002 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP 2002). pages 140-147, Association for Computational Lin- guistics.
Keqin Peng. Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023a. Towards making the most of chatgpt for machine translation.
Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fan- njiang, and David Sussillo, 2019. Hallucinations in neural machine translation.
Xiangyu Peng, Chen Xing, Prafulla Kumar Choubey, Chien-Sheng Wu, and Caiming Xiong. 2023b. Model ensemble instead of prompt fusion: a sample-specific knowledge transfer method for few-shot prompt tun- ing. In The Eleventh International Conference on Learning Representations.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2022. On the advance of making language models better reasoners.
Vikas Raunak, Arul Menezes, and Marcin Junczys- Dowmunt, 2021. The curious case of hallucinations in neural machine translation. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1172-1183, Online. Association for Computational Linguistics.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- urin, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput, Surv, 55()).
Valentin Liévin, Christoffer Egeberg Hother, and Ole Winther. 2023. Can large language models reason about medical questions?
Vikas Raunak, Amr Sharaf, Hany Hassan Awadallah, and Arul Menezes. 2023. Leveraging gpt-4 for auto- matic translation post-editing.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback.
Raj Reddy. 1977. Speech understanding systems: A summary of results of the five-year research effort at carnegie mellon university.
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a. COMET-22: Unb 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578 778,585,44 Abu Dhabi. United Arab Emirates (Hybrid), Association for Computational Linguistics.
Travis Oliphant, 2006. A guide to NumPy, volume I. Trelgol Publishing USA.
OpenAI. 2023. Gpt-4 technical report.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin. Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems.
Ricardo Rei, Craig Stewart, Ana C Farinha. and Alon Lavie. 2020, COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685-2703 Online, Association for Computational Linguistics.

---

Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro. Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b. CometKiwi: IST-unbabel 2022 sub- mission for the quality estimation shared task. In Proceedings of the Seventh Conference Machine Translation (WMT), pages 634-645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Com- putational Linguistics.
Saftyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human- centered machine translation.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023. Just ask for cali- bration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms.
Kai Ming Ting and Ian H. Witten. 1997. Stacked gener- alization: when does it work? In International Joint Conference on Artificial Intelligence.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text genera- tion. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics. pages 7881-7892, Online. Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Nisan Stiemnon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to summarize with human feedback. In Ad- vances in Neural Information Processing Systems, volume 33, pages 3008-3021. Curran Associates, Inc.
Guido Van Rossum and Fred L., Drake. 2009. Python 3 Reference Manual. CreateSpace, Scotts Valley, CA.
Stéfan van der Walt, S Chris Colbert, and Gael Varo- quaux. 2011. The NumPy array: structure for effi- cient numerical computation. Computing in Science & Engineering, 13(2):22-30.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Fdd Chi, and Denny Zhou. 2022. Rationale- augmented ensembles in language models.
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2019. Energyy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy. Asso- ciation for Computational Linguistics.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023a. Self-consistency improves chain of thought reasoning in language models. The Eleventh International Conference on Learning Representations.
Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2023. Recitation-augmentt -eettattio--agggeeeeell language models. In The Eleventh International Conference on Learning Representations.
Yizhong Wang. Yeganeh Kordi. Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language models with ceff-generated instructions.
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. 2014. Se- quence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, volume 27, Curran Associates, Inc.
Weijia Xu, Sweta Agrawal, Eleftheria Briakou, Mari- anna J. Martindale, and Marine Carpuat. 2023. Un- derstanding and detecting hallucinations in neural machine translation via model introspection.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li. Carlos Guestrin, Percy Liang. and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-followinn Ilama model. https:// gi wttpl//oootttttttttaaaaaae aattttaoodaapppeeee
Kevin Yang. Yuandong Tian, Nanyun Peng, and Dan Klein. 2022. Re3: Generating longer stories with recursive reprompting and revision. In Proceedings of the 2022 Conference Empirical Methods in Nat- ural Language Processing, pages 4393-4479, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
NLLB Team, Marta R. Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Hef- fernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan. Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan. Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers,
Biao Zhang, Barry Haddow, and Alexandra Birch, 2023, Prompting large language model for machine transla- tion: A case study.
Zhi-Hua Zhou, Jianxin Wu, and Wei Tang. 2002. En sembling neural networks: Many could be better than all. Artificial Imtelligence, (3:(1)3339-263

---

A Prompt Templates
above. For example, in the case of EN-DE trans- lation, there was an approximate gap of COMET points between the first and last translation. For that reason, we decided to discard this approach.
A.1 Translation
In addition the prompt of Hendy al. (2023), used in the reported experiments with ChatGPT,
For LLaMA and Alpaca, we use variation of the prompt of Hendy al. (2023) which stresses the translation direction (crucial for the non instruction-tunee LLaMA understand the task) as follows,
Translate this sentence from [source language] to [target language].
Source: [source sentence].
Target:
Translate this sentence from [source language] [target language].
we also tried the prompt of Peng et al. (2023a),
[source language] Source: [source sentence].
Please provide the [target language] translation for this sen- tence: [source sentence]
[target language] Translation:
the prompt of Gao et al. (2023), which provides additional information on the translation task and the language pairs involved,
A.2 Generation of the final translation
We formulate the task of choosing the most ap- propriate hypothesis (ChooseBest) a multiple choice question using the following prompt,
This [source language] [target language] translation task, please provide the [target language] translation for this sentence: [source sentence] and the prompt of Zhang et al. (2023), which is simpler but concise,
This is multiple choice question, choose single answer. What the best [target language] translation for this [source language] sentence?
Source: [source]
Option A. [hypothesis
[source language]: sentence]
[source
Option B, [hypothesis 2]
[target language]:
Correct answer: Option
In preliminary stage of this work, we observed that the results according to automatic evaluation metrics were similar for all the prompts above. In addition, ensembling translations generated with multiple prompts was not better than sampling hy- potheses using single prompt template. We also attempted to generate multiple translations (N) with the following prompt,
Besides, ask the LLM generate final prediction based on the provided hypotheses (GenerateBest) using the following prompt,
Use the following translation hypotheses generate the best possible [target language] translation for this [source language] sentence.
Translate this sentence from [source language] to [target language] in [N] different ways.
Source: [source]
Translation hypotheses:
Source: [source sentence]
[hypothesis 1]
[N] translations:
[hypothesis 2]
However, we observed a decline in translation qual- ity as the model generated subsequent translations, with the first one exhibiting lower quality compared to the ones generated using the prompts mentioned
Best possible translation:

---

EN-DE DE
EN-RU
EN-CS
EN-UK
N
METHOD
COMET
BLEURT
COMET
BLEURT
COMET
BLEURT
COMET
BLEURT
ChatGPT greody
87.01
77.15
87.77
75.61
90.04
80.98
87.66
76.08
Greedy
81.54
69.39
82.35
67.27
82.17
69.27
81.32
66.00
Biased sampling
77.47
64.87
75.89
58.62
73.59
58.47
74.29
56.50
Ranking
84.87
73.91
86.02
72.11
87.02
75.48
85.64
71.36
20
MBR decoding
85.78
73.26
87.05
71.45
88.45
74.71
86.68
70.61
COMET oracle
87.74
75.83
88.85
74.13
89.70
76.39
88.57
73.41
Ranking MBR decoding
85.16
74.37
86.75
72.93
88.11
76.84
86.29
71.61
50
86.48
73.97
87.79
72.37
89.61
76.33
87.62
71.52
COMET oracle
88.77
77.12
90.02
76.09
91.10
78.76
89.80
75.60
DE-EN
RU-EN
CS-EN
UK-EN
COMET
BLEURT
COMET
BLEURT
COMET
BLEURT
COMET
BLEURT
ChatGPT greedy
85.45
74.50
85.99
77.92
87.13
77.42
85.63
76.50
Greedy
83.12
71.01
83.00
73.48
83.64
72.60
82.86
72.50
Biased sampling
81.29
69.03
80.83
70.56
81.23
69.50
80.67
69.89
Ranking MBR docoding
84.68
72.85
84.43
74.83
85.24
74.45
84.69
74.36
20
85.18
72.64
85.06
75.00
85.74
73.91
84.91
73.87
COMET oracle
87.33
75.70
87.22
78.14
87.89
77.13
87.62
77.78
Runking
84.76
72.96
84.53
75.05
85.45
74.46
84.70
74.77
50
MBR decoding
85.48
72.97
85.34
75.18
86.17
74.29
85.33
74.29
COMET oracle
88.13
76.93
88.08
79.45
78.60
88.65
79.36
Table 5: Automatic evaluation metrics for LLaMA (30B). We use temperature and nucleus sampling (with = 0.8
and p = 0.95), which is default (see $3.3). Ranking uses COMETKIWI and MBR decoding uses COMET. Best overall values for LLaMA are bolded and best within each group underlined. Values for ChatGPT with greedy decoding are taken from Table and highlighted in red.
EN-DE
EN-RU
FEW-SHOT METHOD
COMET
BLEURT
COMET
BLEURT
Greedy
77.33
62.86
71.57
50.80
Ranking
83.04
71.91
82.93
67.41
MBR decoding
84.06
69.84
83.72
64.75
COMET oracle
86.51
73.42
86.59
69.43
Greedy 79.82
68.02
80.20
65.01
Ranking
83.72
72.90
84.98
70.38
MBR decoding
85.42
72.66
86.64
70.37
COMET orucle
87.42
75.06
88,43
72.88
Table 6: Automatic evaluation metrics for LLaMA (7B) with and without few-shot learning. For ranking with COMETKIWI and MBR decoding with COMET we use 50 samples obtained through temperature and nucleus sampling (with = 0.8 and y = 0.95), which default (see $3.3). Best overall values are bolded and best within each group underlined.
B Increasing Model Size
ing the number of model's parameters is beneficial even without instruction tuning.
Table 5 shows COMET and BLEURT scores for LLaMA (30B), along with the ChatGPT output ($3.2). The main findings discussed in $3.2 still hold: ensembling multiple translations is effective improving the overall translation quality. No- tably, these quality scores are more competitive with the ChatGPT baseline, suggesting that increas-
Few Shot Learning
As discussed in $2.1, the candidate generation strategies described in our paper can also be ap- plied in few-shot scenarios where in-context exam- ples are provided in the prompt. While this paper

---

does focus on this case and covers other or thogonal dimensions in more detail (e.g., choice of model, method to generate hypotheses, strategy to generate the final translation), we include results with few-shot examples on a subset of the language pairs (EN-DE and EN-RU), in Table 6. We con- sider 5-shot examples from the FLORES-200 dev set (Guzmán et al., 2019; Goyal et al., 2022; Team al., 2022) and use LLaMA (7B). As expected, see that hypothesis ensembling still works well for such setting, with the overall scores being higher than in a 0-shot scenario.
Biasedness, Diversity, and Quality
Fig. 4 shows that the trends observed in Fig. 3 for EN-DE translations are also observable in the re- versed translation direction (DE-EN). Once again, increasing the sampling temperature leads to an increase in the diversity of the hypotheses, and this trend varies at different rates for LLaMA and Al- paca. As the diversity gap increases, the translation quality between ensembles of samples from these models diverges. However, it is worth noting that this effect is less pronounced for DE-EN, likely due to the extensive English training data used for these models that results in lower absolute values translation diversity, as indicated by the blue lines.
0.50
Diverityy
0.00
COMET
84
83
BLEURT
0.0
0.2
0.4
0.6
0.8
1.0
Temperature
E Hallucinations
llams, mbr
llams ranking
llams. diversity alpaca, diversity
alpaca, mbr
alpaca, ranking
We follow the choices of Guerreiro et al. (2023) and detect hallucinations under perturbations as fol- lows. For each language pair, we start by obtaining source sentences for which all methods (greedy, ranking with COMETKIWI, and MBR decoding with COMET) generate unperturbed translations that meet a minimum quality threshold (BLEU 9). Then, we set a low maximum quality score for perturbed translations (BLEU- 3). A model generates a hallucination when both thresholds met. The metric for measuring lexical overlap and the threshold values we used follow previous work.
Figure 4: Values for BLEURT (bottom) and COMET (middle) for MBR decoding with COMET (green) and ranking with COMETKIWI (orange), and diversity be- tween hypotheses (top: blue) as we increase the sam- pling temperature for DE-EN. We represent LLaMA with solid lines and Alpaca with dashed lines. The dot- ted black lines (top) mark the increasing diversity gap.


Reviews:
[{"review_id": "GgfRc2quBJ", "reviewer": "Reviewer_uVRh", "summary": "", "questions": "What is the translation performance of closed-source LLM like ChatGPT?\nIf the hypotheses of powerful LLMs ensemble with results of LLMs with relatively small scale parameters, for example, LLaMA2-70B ensembles with Alpaca and LLama1-7B, the ensembling translation quality will drop?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive reviews! We will now try to address your concerns about our paper.\n\n>Their study is limited to translation pairs among languages in European, thus the generalization between other language pairs, for example, the ‘European-Asian’ pair, is unknown.”\n\nWe followed your suggestion and  ran some experiments on translation from English to Chinese on the WMT 2022 test set. The results are inline with our findings for non-Asian languages. The results in Table 1 can be complemented with the following information for En-Zh:\n\n| Model   | Method   | N  | COMET     | BLEURT    |\n|---------|----------|----|-----------|-----------|\n| ChatGPT | Sampling | 1  | 86.27     | 71.34     |\n| ChatGPT | Greedy   | 1  | 86.94     | 72.22     |\n| ChatGPT | Ranking  | 20 | 87.86     | 72.69     |\n| ChatGPT | MBR      | 20 | 88.28 | 72.96 |\n| ChatGPT | Ranking  | 50 | 88.08     | 72.78     |\n| ChatGPT | MBR      | 50 | **88.47** | **72.98** |\n\n> What is the translation performance of closed-source LLM like ChatGPT?\n\nThe results for ChatGPT are included in Table 1 (see the first two lines for the performance of single hypothesis translations). While ChatGPT’s performance alone is good (as reported by, e.g., Hendy et al. (2023)), we show that it can be further boosted by ensembling multiple hypotheses, especially for En-X translation (see Figure 1). \n\n> If the hypotheses of powerful LLMs ensemble with results of LLMs with relatively small scale parameters, for example, LLaMA2-70B ensembles with Alpaca and LLama1-7B, the ensembling translation quality will drop?\n\nThis is a very interesting question. We ran some experiments and found that combining the outputs of a powerful LLM (e.g., ChatGPT) and the outputs of a smaller scale one (e.g., Alpaca 7B) with MBR decoding actually *improves* the final quality of the ensemble. For some language pairs (En-De and En-Ru) a combination of 20 samples from each model is even better than an ensemble of 50 samples from the stronger model (see some results below). We believe this might have to do with the higher diversity of the hypotheses. We will add this new experiment to the final version of the paper.\n\n| Model               | Method | N         | En-De | En-Ru | En-Cs | En-Uk |\n|---------------------|--------|-----------|-------|-------|-------|-------|\n| ChatGPT             | Greedy | 1         | 87.01 | 87.77 | 90.04 | 87.66 |\n| ChatGPT             | MBR    | 20        | 88.09 | 89.41 | 91.97 | 90.03 |\n| ChatGPT + Alpaca 7B | MBR    | 20+20=40  | 88.49 | 89.89 | 92.15 | 90.23 |\n| ChatGPT             | MBR    | 50        | 88.25 | 89.64 | 92.21 | 90.31 |\n| ChatGPT + Alpaca 7B | MBR    | 50+50=100 | **88.62** | **90.06** | **92.38** | **90.58** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main strengths of this paper are: \n1. the authors demonstrated clear structure and logic, which is easy to follow; \n2. adequate comparisons among each set of experiments were explained in detail with tables and graphs; \n3. the problem the authors focused on is an important and state-of-the-art method for machine translation using the LLM paradigm.", "reasons_to_reject": "The weakness of this paper is that I found their study is limited to translation pairs among languages in European, thus the generalization between other language pairs, for example, the ‘European-Asian’ pair, is unknown.", "questions_to_authors": "What is the translation performance of closed-source LLM like ChatGPT?\nIf the hypotheses of powerful LLMs ensemble with results of LLMs with relatively small scale parameters, for example, LLaMA2-70B ensembles with Alpaca and LLama1-7B, the ensembling translation quality will drop?", "paper_topic_and_main_contributions": "In this paper, the authors studied the topic of improving the quality of the generated text for the specific problems of Large Language Model (LLM)-based machine translation by hypothesis embeddings, such as hallucination and unreliable output. Several comparable results were conducted through experimental pairs such as open-source scenario vs. closed-source one, different languages transformation, and searching strategies. \n\nThe main contribution of this paper is that, through a set of experiments, three findings were given: translation quality can be enhanced with a small number of samples; instruction tuning affects the relationship between the diversity of the hypotheses and the sampling temperature; hypothesis ensembling reduces the number of generated hallucinations. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NMhpmVnVFF", "reviewer": "Reviewer_aEq9", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive reviews! We are glad that you found the empirical study comprehensive and the paper well written and easy to follow.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The empirical study is comprehensive and covered many areas.\n\nThe paper is well written and easy to follow. ", "reasons_to_reject": "I do not have a reason.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper studies various techniques for ensembling translations from large language models. Experiments were conducted on several LLM models (ChatGPT, LLaMA, Alpaca) with 8 directions from English to four languages (DE,RU,CS,UK) and vice versa. The results show that ensembling increases translation quality and reduces hallucination. It is closing the gap between the studied models and ChatGPT. The results also show that generating a final translation with MBR decoding achieves high scores.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EaaUiGeP6C", "reviewer": "Reviewer_UV4z", "summary": "", "questions": "In the experiment, the LLM based approaches (ChooseBest and GenerateBest) did not perform well. This looked somewhat surprising considering that a good result about LLM-based quality estimation is recently reported (Kocmi and Federmann, Large Language Models Are State-of-the-Art Evaluators of Translation Quality). It will be better to discuss why the approaches did not work.\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and suggestions. We address below your main concerns about our paper.\n\n> The approach tried in the paper is not novel, and it is not clear if it works for other settings.\n\nPlease note that while MBR decoding and ranking approaches based on neural quality estimation metrics have been studied before for task-specific NMT models (a line of research which usually considers large sets of hypotheses), our paper focuses on a understudied and increasingly relevant scenario, LLM-based translation using general-purpose language models and a relatively small number of hypotheses. Our study provides  important  insights about  what changes in this scenario. For instance, contrary to previous findings using task-specific models, our experiments suggest that even a small number of samples can effectively improve performance; we also discuss in which conditions we do not need biased samples (see $\\S 3.2.1$ and $\\S 3.3.3$), streamlining the method's applicability. In addition, we believe that our paper sheds light on how to generate high-quality translations with LLMs, including in which conditions beam search is a reliable baseline ($\\S 3.3.1$). More notably, we find that hypothesis ensembling reduces the number of generated hallucinations (see $\\S 3.3.4$) and thus improves the model's robustness to source perturbations.\n\n> LLMs are often used with few-shot prompts, and it is not clear if the hypothesis reranking works well for such a setting.\n\nAs discussed in $\\S 2.1$, the candidate generation strategies described in our paper can also be applied in few-shot scenarios where in-context examples are provided in the prompt. We chose not to focus much on this scenario  so that we could  cover  other (orthogonal) dimensions in more detail  (e.g., choice of model, method to generate hypotheses, strategy to generate the final translation). However, we agree that including a small experiment with few-shot examples can be valuable, and therefore we decided to run some experiments to answer your question. As expected, we found that hypothesis ensembling still works well for such a setting (the numbers below are for 5-shot examples from the FLORES-200 dev set using LLaMA 7B (En-De and En-Ru)):\n\n|                  |         |    | En-De     |           | En-Ru     |           |\n|------------------|---------|----|-----------|-----------|-----------|-----------|\n| Model            | Method  | N  | COMET     | BLEURT    | COMET     | BLEURT    |\n| LLaMA 7B, 0-shot | Greedy  | 1  | 77.33     | 62.86     | 71.57     | 50.80     |\n| LLaMA 7B, 0-shot | Ranking | 50 | 83.04     | 71.91     | 82.93     | 67.41     |\n| LLaMA 7B, 0-shot | MBR     | 50 | 84.06     | 69.84     | 83.72     | 64.75     |\n| LLaMA 7B, 5-shot | Greedy  | 1  | 79.82     | 68.02     | 80.20     | 65.01     |\n| LLaMA 7B, 5-shot | Ranking | 50 | 83.72     | **72.90** | 84.98     | **70.38** |\n| LLaMA 7B, 5-shot | MBR     | 50 | **85.42** | 72.66     | **86.64** | 70.37     |\n\nWe will update our paper accordingly. \n\n> In the experiment, the LLM based approaches (ChooseBest and GenerateBest) did not perform well. This looked somewhat surprising considering that a good result about LLM-based quality estimation is recently reported (Kocmi and Federmann, Large Language Models Are State-of-the-Art Evaluators of Translation Quality). It will be better to discuss why the approaches did not work.\n\nThis is an interesting point. While Kocmi and Federmann report state-of-the-art performance on assessing system-level quality (see their Table 2), the same cannot be said for segment-level evaluation with ChatGPT (see their Table 4), in which GPT-based methods still lag behind dedicated evaluation systems like COMET or BLEURT. Notably, methods like GenerateBest (proposed in our paper) depend on the capabilities of systems to perform reliable segment-level assessments, which may explain the poor performance of these methods. However, it is important to note that machine translation evaluation/quality estimation using LLMs remains an area of active research, and we expect that the landscape might change as more advanced models/methods (with improved segment-level capabilities) are introduced. We will add this discussion to the paper.\n\n> I felt the term \"ensembling\" in this paper is confusing. In the context of ML, ensembling means to combine multiple models. This paper considers to select the best hypothesis from multiple candidates produced by a single model, which is often called reranking.\n\nIn footnote 2 we clarify the use of the term \"ensembling\". We chose to use \"ensembling\" instead of \"reranking\" since we also experiment with non-reranking techniques such as GenerateBest. While the best results are obtained with reranking, we believe that the inclusion of GenerateBest (and the finding that it is not competitive) is an important contribution of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper shows that the performance of LLM-based MT is improved with simple methods.\n- The experiments are conducted for 8 language pairs using multiple LLMs.\n", "reasons_to_reject": "- The approach is not novel. MBR decoding is widely known. The paper tries two simple approaches using LLMs but they did not work well.\n- LLMs are often used with few-shot prompts, and it is not clear if the hypothesis reranking works well for such a setting.\n", "questions_to_authors": "In the experiment, the LLM based approaches (ChooseBest and GenerateBest) did not perform well. This looked somewhat surprising considering that a good result about LLM-based quality estimation is recently reported (Kocmi and Federmann, Large Language Models Are State-of-the-Art Evaluators of Translation Quality). It will be better to discuss why the approaches did not work.\n", "paper_topic_and_main_contributions": "This paper studies how to improve LLM-based MT by reranking multiple hypotheses. In the experiment, MBR decoding with COMET improved translation performance.\n\nThis study shows that the simple approach can boost the performance of LLM-based MT through the experiments for various language pairs, and the result may be useful. On the other hand, the approach tried in the paper is not novel, and it is not clear if it works for other settings.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I felt the term \"ensembling\" in this paper is confusing. In the context of ML, ensembling means to combine multiple models. This paper considers to select the best hypothesis from multiple candidates produced by a single model, which is often called reranking.\n"}}]

Author rebuttal:

Meta-review: Paper on combination of MT outputs generated by LLMs. Different strategies to generate multiple hypotheses with LLMs, as well as methods to produce the final translation are explored. Contributions of the paper are: ensembling multiple hypotheses significantly improves translation quality;   instruction tuning affects diversity/temperature relationship; hypothesis ensembling reduces hallucinations. Experiments are provided for 8 directions of 4LPs. Paper is well written and well-organized paper. 


Reasons to accept: 
Paper investigates MT with LLMs in a scientifically solid way: they use both closed and open source models; the former to see performance impact on high end models, the latter to allow for reproducibility of results and more in depth investigations. 
Experimental results are informative and performed with up to date evaluation metrics. 

Reasons to reject: 
Limited language coverage (4 language pairs)
Limited novelty in the approaches, the authors apply (and properly cite) known methods for their experiments.    
Some methods are just tried and do not work consistently across models and metrics making this work sound very empirical. We do not know if these results will still hold for another version of the same models. 
The impact of instruction tuning is measured on one single model and LP. 

Detailed comments
Figure 1: the type of plot is misleading as it suggests a continuum between languages which does not exist. 
269: “significantly increases  the cost”. Cost of ChatGPT should not be part of the equation. If you are using external LLMs you should assume they are for free. 
303 Understanding the prompt requires reading the appendix.
Decision: Accept-Main