Document-Level Text Generation with Minimum Bayes Risk Decoding using Optimal Transport
Authors: Yuu Jinnai
Accepted at: acl25

Document-Level Text Generation with Minimum Bayes Risk Decoding using Optimal Transport

Yuu Jinnai
CyberAgent
jinnai_yu@cyberagent.co.jp

Abstract
Document-level text generation tasks are known to be more difficult than sentence-level text generation tasks as they require the understanding of longer context to generate high-quality texts. In this paper, we investigate the adaption of Minimum Bayes Risk (MBR) decoding for document-level text generation tasks. MBR decoding makes use of a utility function to estimate the output with the highest expected utility from a set of candidate outputs. Although MBR decoding is shown to be effective in a wide range of sentence-level text generation tasks, its performance on document-level text generation tasks is limited as many of the utility functions are designed for evaluating the utility of sentences. To this end, we propose MBR-OT, a variant of MBR decoding using Wasserstein distance to compute the utility of a document using a sentence-level utility function. The experimental result shows that the performance of MBR-OT outperforms that of the standard MBR in document-level machine translation, text simplification, and dense image captioning tasks. Our code is available at https://github.com/jinnaiyu/mbr-optimal-transport.

1 Introduction
Large language models (LLMs) have demonstrated remarkable capabilities across various natural language processing tasks (Stienon et al., 2020; Ouyang et al., 2022; Touvron et al., 2023; Dubey et al., 2024; OpenAI et al., 2024). While many text-generation tasks are evaluated at the sentence level, LLMs are also capable of generating text at the document level (Wang et al., 2023; Xia et al., 2024; Zhang et al., 2024). This raises the need for evaluating the performance of decoding algorithms for document-level text generation tasks.

Minimum Bayes Risk (MBR) decoding has been shown to be highly effective for sentence-level directed text generation tasks (Goel and Byrne, 2000; Freitag et al., 2022a; Eikema and Aziz, 2022; Freitag et al., 2023a). However, its effectiveness for generating longer texts is less investigated. In this paper, we evaluate the performance of MBR decoding in document-level text generation tasks. Specifically, we propose MBR-OT, a variant of MBR decoding that uses an optimal transport distance as a utility function. Optimal transport has been used in many fields to measure the dissimilarity of two probability distributions (Peyré and Cuturi, 2020; Villani, 2021). We use optimal transport as a tool to evaluate the utility between documents using sentence-level utility functions. This approach enables the use of the sentence-level utility functions that have been investigated for years to improve their accuracy by many researchers (Freitag et al., 2022b, 2023b, 2024).

We evaluate MBR-OT on multiple directed text generation tasks: document-level machine translation, document-level text simplification, and dense image captioning. Our results show that MBR decoding consistently outperforms the baselines across these tasks, showing its effectiveness for document-level text generation tasks.

I love cats. I love dogs.
A      B
I love dogs. I love cats.
B      A

![Illustrative example of a metric using Wasserstein distance over two texts "I love cats. I love dogs" and "I love dogs. I love cats.". Each output is segmented into a set of segments (e.g., sentence) and a utility function is used to compute the utility over a pair of segments from each of the outputs. Wasserstein distance is flexible to the change in the structure of the text, making it an adaptive measure for a wide range of tasks.](page_768_355_393_246.png)

Figure 1: Illustrative example of a metric using Wasserstein distance over two texts "I love cats. I love dogs" and "I love dogs. I love cats.". Each output is segmented into a set of segments (e.g., sentence) and a utility function is used to compute the utility over a pair of segments from each of the outputs. Wasserstein distance is flexible to the change in the structure of the text, making it an adaptive measure for a wide range of tasks.
2 Background

We introduce MBR decoding as one of the algorithms to solve the task. Then, we explain the concept of optimal transport which is used by the proposed method.

2.1 Minimum Bayes Risk (MBR) Decoding

Unlike maximum a posteriori (MAP) decoding (Eikema and Aziz, 2020; Holtzman et al., 2020), which aims to find the most probable output, minimum Bayes risk (MBR) decoding selects the output that maximizes the expected utility, which is equivalent to minimizing risk (Goel and Byrne, 2000; Kumar and Byrne, 2002, 2004).

MBR decoding consists of two key components: a text generation model \( P_{\text{model}} \) and a utility metric \( u \). The model estimates the probability of an output \( y \) given an input sentence \( x \). The utility metric, \( u(h, y) \), measures the quality of a candidate output \( h \) with respect to a reference output \( y \).

Let \( \mathcal{Y} \) be a set of all possible sequences. Given a set of candidate hypotheses \( \mathcal{H}_{\text{cand}} \subseteq \mathcal{Y} \), MBR decoding selects the hypothesis that maximizes its expected utility:

\[
h^{\text{human}} = \arg\max_{h \in \mathcal{H}_{\text{cand}}} \sum_{y \in \mathcal{Y}} u(h, y) \cdot P_{\text{human}}(y \mid x).
\]

(1)

Since the true human probability distribution, \( P_{\text{human}} \), is unknown, MBR approximates it using the model probability \( P_{\text{model}} \):

\[
h^{\text{model}} = \arg\max_{h \in \mathcal{H}_{\text{cand}}} \sum_{y \in \mathcal{Y}} u(h, y) \cdot P_{\text{model}}(y \mid x).
\]

(2)

For simplicity, we denote \( P_{\text{model}} \) as \( P \) throughout the remainder of this paper unless stated otherwise.

Since integrating over \( \mathcal{Y} \) is computationally intractable, Eq. (2) is typically approximated using a Monte Carlo estimate (Eikema and Aziz, 2022; Farinhas et al., 2023). This is done by sampling a set of reference hypotheses \( \mathcal{H}_{\text{ref}} \), from the model \( P \):

\[
h^{\text{MBR}} = \arg\max_{h \in \mathcal{H}_{\text{cand}}} \frac{1}{N} \sum_{y \in \mathcal{H}_{\text{ref}}} u(h, y),
\]

where \( N = |\mathcal{H}_{\text{ref}}| \).

A common practice is to use the same set of hypotheses for both the candidate pool (\( \mathcal{H} \)) and the reference pool (\( \mathcal{H}_{\text{ref}} \)). We follow the practice in this paper and assume \( \mathcal{H}_{\text{cand}} = \mathcal{H}_{\text{ref}} \).

2.2 Optimal Transport (OT)

Optimal Transport (OT; Peyré and Cuturi 2020; Villani 2021) provides a mathematical framework for quantifying the dissimilarity between two distributions. In natural language processing (NLP), OT has been widely used to measure text similarity, often referred to as the Earth Mover’s Distance (Kusner et al., 2015; Zhao et al., 2019). It has been applied in various contexts, including document similarity (Kusner et al., 2015) and summary evaluation (Zhao et al., 2019).

While existing metrics for document-level machine translation have been proposed (Vernikos et al., 2022), they are typically designed for tasks where generated documents can be segmented into a fixed sequence of corresponding segments. This limitation makes them unsuitable for scenarios where segment order or count varies across documents.

On the other hand, a key advantage of OT-based metrics is their adaptability to text structure. They can effectively handle variations such as sentence reordering and merging, which frequently occur in machine translation (Hovy and Gerber, 1997; Marcu et al., 2000). For instance, due to structural differences between Japanese and English, professional translators often restructure paragraphs during translation, significantly altering sentence order and merging content (Hovy and Gerber, 1997).

In this paper, we consider multiple OT formulations, including linear assignment, Wasserstein distance, and entropic regularized Wasserstein distance.

Linear assignment (LA). Linear assignment (LA) is a simple formulation of OT where the cost is computed as a linear sum of the cost of each element (Peyré and Cuturi, 2020; Villani, 2021). Let \( h = \{h_1, h_2, \ldots, h_m\} \) and \( y = \{y_1, y_2, \ldots, y_n\} \) be a set of sentences. Let \( p_h \) and \( p_y \) be a probability distribution over a set of elements in \( h \) and \( y \). Let \( C \) be a non-negative function representing the cost or the dissimilarity between two sentences. Using LA, the cost between two sets of sentences is defined as follows:

\[
\text{LA}_C[p_h \| p_y] =
\inf_{\gamma \in \Gamma_L(h, y)} \sum_{i \in [1..m]} p_h(h_i) C(h_i, \gamma(h_i)),
\]

(4)

where \( \Gamma_L(h, y) \) is a set of deterministic mappings from \( h \) to \( y \) that is injective if \( m \leq n \), and subjective if \( m \geq n \) (thus bijective if \( m = n \)).
LA assigns one sentence in a source to exactly one sentence in a target and computes the sum of the cost between them. One of the failure cases of this constraint is when aligning texts with and without *merge*.

(1) I like cats and dogs.

(2) I like cats. I like dogs.

Because LA cannot distribute weights of a source sentences probabilistically, it has to assign "I like cats and dogs." to only one of either "I like cats." or "I like dogs.". Thus, although the two texts have very high similarity, its utility computed by LA gets small.

**Wasserstein distance (WD).** The shortcoming of LA is that it is constrained to assign the weight of the source element to a single target element, and thus cannot adapt to the change in disclosure. Figure 1 shows the example of sentences where this is undesirable. Wasserstein distance (WD) is a generalization of LA where the source element can divide its weights and assign it to any number of target elements (Peyré and Cuturi, 2020; Villani, 2021). This enables the alignment of the sequence like in Figure 1 where one source sentence is divided into two target sentences.

WD is computed as follows:

\[
\mathrm{WD}_C[p_{\mathbf{h}} \| p_{\mathbf{y}}] =
\inf_{\gamma \in \Gamma(p_{\mathbf{h}}, p_{\mathbf{y}})} \sum_{(i,j) \in m \times n} \gamma(h_i, y_j) C(h_i, y_j),
\]

where \( \Gamma(p_{\mathbf{h}}, p_{\mathbf{y}}) \) is a set of all possible joint distributions \( \gamma \) whose marginals are \( p_{\mathbf{h}} \) and \( p_{\mathbf{y}} \).

WD is a metric used to quantify the dissimilarity between two probability distributions. Intuitively, it measures the minimum cost required to transform one distribution into the other. This cost is conceptualized as the amount of probability mass that must be moved multiplied by the distance that would be moved.

The advantage of WD over LA is that it can assign multiple reference segments to a single source sentences. In the case of "I like cats and dogs.", one can distribute its weight to both '"I like cats." and "I like dogs."', resulting in high utility score.

**Entropic regularized WD (EWD).** Entropic regularized WD (EWD) is an extension of WD where the KL regularization is enforced to the joint probability distribution \( \gamma \) (Peyré and Cuturi, 2020; Villani, 2021):

\[
\mathrm{WD}_C^{\epsilon}[p_{\mathbf{h}} \| p_{\mathbf{y}}]
= \inf_{\gamma \in \Gamma(p_{\mathbf{h}}, p_{\mathbf{y}})} \sum_{(i,j) \in m \times n} \gamma(h_i, y_j) C(h_i, y_j)
+ \epsilon \mathrm{KL}[\gamma \| p_{\mathbf{h}} \oplus p_{\mathbf{y}}],
\]

where KL represents the KL-divergence between the two probability distributions and \( \epsilon \) is a parameter to choose the weight on the KL-divergence term.

Intuitively, EWD is a WD plus the cost of every sentence pair is considered. The KL-divergence term requires the joint distribution \( \gamma \) to be spread across \( \mathbf{y} \). Thus, the KL term is smaller if the two documents are similar to each other overall in addition to the cost of the optimal assignment between pairs of sentences.

EWD is known to be robust under the model uncertainty (Azizzian et al., 2023) and is fast to compute using the Sinkhorn algorithm (Cuturi, 2013).

3 **MBR-OT: MBR Decoding using OT**

The performance of MBR decoding relies on the utility function. However, many of the state-of-the-art utility functions are developed for sentence-level evaluation and are not trained to predict document-level utility.

To this end, we propose **MBR-OT**, a variant of MBR decoding that uses WD as the utility function for MBR decoding. Let the utility function between two sentences be \( u_s \) where we assume its range to be [0, 1].

Let \( p_{\mathbf{h}} \) be a discrete probability distribution over \( \mathbf{h} = \{ h_1, h_2, ..., h_m \} \) and so is \( p_{\mathbf{y}} \) for \( \mathbf{y} = \{ y_1, y_2, ..., y_n \} \). We propose the WD between the two probability distributions to be the utility function of MBR-OT. Formally, the utility function is defined as follows:

\[
u(\mathbf{h}, \mathbf{y}) = 1 - \mathrm{OT}[p_{\mathbf{h}} \| p_{\mathbf{y}}],
\]

where

\[
C(h_i, y_j) = 1 - u_s(h_i, y_j),
\]

where \( u_s \) is a utility function to be used for computing the utility of two segments. In the following experiments, we use sentence-level utility functions such as BLEU and MetricX for \( u_s \). Figure 1 shows the illustrative example of how it is computed.
We use a uniform distribution for constructing \( p(\mathbf{h}) \) and \( p(y) \) so that each sentence is weighted equally:

\[
p(\mathbf{h}_i) = \frac{1}{|\mathbf{h}|}.
\] (9)

An alternative is to weight a probability proportional to its length so that each token is weighted equally:

\[
p_L(\mathbf{h}_i) = \frac{|\mathbf{h}_i|}{\sum_j |\mathbf{h}_j|}.
\] (10)

Incorporating segment informativeness may enhance the performance of our proposed method. This approach may be particularly beneficial in evaluating informal texts where there are large variations in segment informativeness.

The choice of the optimal transport formulation depends on the structure of the language and the objective of the task. We call the MBR-OT using AL, WD, and EWD for the OT in Eq. (7) as MBR-AL, MBR-WD, and MBR-WD\(^e\). In particular, we denote these algorithms using Eq. (10) as MBR-AL\(_L\), MBR-WD\(_L\), and MBR-WD\(_L^e\).

The strength of the method is that it can make the best use of the sentence-level utility functions. Sentence-level utility functions are meticulously developed by researchers and engineers and they are shown to have a very high correlation with human evaluation (Rei et al., 2022; Guerreiro et al., 2024). Compared to the effort on sentence-level utility functions, document-level utility functions are less investigated. In fact, many of them are based on using sentence-level utility functions efficiently (Liu et al., 2020; Vernikos et al., 2022).

Optimizations for MBR-OT using WD. There are several optimizations applicable to speed up MBR-OT when using WD. First, because WD (and EWD) are metrics, they are guaranteed to be symmetric:

\[
\forall \mathbf{h} \forall y : u(\mathbf{h}, y) = u(y, \mathbf{h}).
\] (11)

Also, the value of WD is 1 when the two distributions are the same. Thus,

\[
\forall \mathbf{h} : u(\mathbf{h}, \mathbf{h}) = 0.
\] (12)

Therefore, for \( N \) documents, one only needs to compute \( \frac{N(N-1)}{2} \) pairs of documents instead of \( N^2 \) pairs.

Second, because WD is a metric in a finite-dimensional space, it is likely that the matrix of utility scores can be well approximated by a low-rank matrix (Drineas and Mahoney, 2005; Holodnak and Ipsen, 2015). This enables the optimization by a low-rank approximation, significantly reducing the computational complexity (Trabelsi et al., 2024).

Third, we can train a document-level utility function by distilling the WD metric, or train the language model directly. There are several studies showing that the performance of LLM can be improved by distilling the output of MBR decoding (Yan et al., 2023; Ramos et al., 2024; Wu et al., 2024; Yang et al., 2024; Finkelstein and Freitag, 2024; Guttmann et al., 2024). These approaches require additional training but solve the computational overhead at decoding time.

In the experiments of the paper, we only use the first optimization so that the values are computed exactly.

4 Experiments

We first evaluate the accuracy of the WD metric for machine translation to assess if it is a metric suitable for a document-level evaluation. We evaluate the performance of MBR decoding and MBR-OT on machine translation, document simplification, and dense image captioning tasks.

In all the experiments, we divide the output into a set of sentences for MBR-OT. The default sentencizer in spaPy\(^1\) is used for English and German. GiNZA NLP Library\(^2\) is used for Japanese as the sentencizer in spaPy is incompatible with Japanese. See Appendix E for the hyperparameters used for the experiments and Appendix F for the implementation details.

4.1 Evaluation of WD Metrics for Machine Translation

We evaluate the accuracy of the WD metric using the Metric Shared Task on WMT22 and WMT23 (Freitag et al., 2022b, 2023b). We compare the correlation of the metrics with the human evaluation. As a baseline, we compare 1. using the metric to evaluate each segment (Base), and 2. using the document-level evaluation method by Vernikos et al. (2022) to evaluate each segment but with a context of the document (Doc). On computing the score using the WD, we compute the utility of the entire document without using the segmentation

\footnotetext{1https://github.com/explosion/spaCy}
\footnotetext{2https://github.com/megagonlabs/ginza}
<table>
  <tr>
    <th rowspan="2">data</th>
    <th rowspan="2">lp</th>
    <th rowspan="2">Utility</th>
    <th colspan="3">base</th>
    <th>doc</th>
    <th>ot (Ours)</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td>wmt22</td>
    <td>en-de</td>
    <td>sacrebleu</td>
    <td>0.6420</td>
    <td>0.6783</td>
    <td>0.7090</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>BERTScore</td>
    <td>0.7843</td>
    <td>0.7670</td>
    <td>0.8060</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>SentBERT</td>
    <td>0.8630</td>
    <td>0.8313</td>
    <td>0.8681</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>en-ru</td>
    <td>sacrebleu</td>
    <td>0.7861</td>
    <td>0.7480</td>
    <td>0.7919</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>BERTScore</td>
    <td>0.8106</td>
    <td>0.7864</td>
    <td>0.8061</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>SentBERT</td>
    <td>0.8030</td>
    <td>0.8437</td>
    <td>0.8114</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>wmt23</td>
    <td>en-de</td>
    <td>sacrebleu</td>
    <td>0.8911</td>
    <td>0.9165</td>
    <td>0.9384</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>BERTScore</td>
    <td>0.8906</td>
    <td>0.9212</td>
    <td>0.9553</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>MetricX-23</td>
    <td>0.9876</td>
    <td>0.9874</td>
    <td>0.9753</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>he-en</td>
    <td>sacrebleu</td>
    <td>0.8751</td>
    <td>0.8853</td>
    <td>0.8716</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>BERTScore</td>
    <td>0.8942</td>
    <td>0.9449</td>
    <td>0.7138</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>MetricX-23</td>
    <td>0.9357</td>
    <td>0.9600</td>
    <td>0.8072</td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 1: System level correlation of the metrics with the human evaluation. Base: sentence-level metric. Doc: a document-level metric by Vernikos et al. (2022), WD: a document-level metric using WD.

provided. Then, we use the average score over a set of documents as the system score. We set \( \epsilon = 0 \) and use a uniform weight (Eq. 9) for the probability distribution of WD.

We evaluate BLEU score (Papineni et al., 2002) using sacrebleu library (Post, 2018), BERTScore (Zhang et al., 2020) (bert-base-multilingual-cased).3 and a cosine distance of the sentence embedding model (SentBERT; sentence-transformers/all-mpnet-base-v2), and MetricX-23 (google/metricx-23-xl-v2p0; Juraska et al., 2023).

Table 1 shows the correlation of the metrics with human evaluation. Because MetricX-23 is trained with the WMT22 dataset, we evaluate SentBERT instead. Overall, we observe the WD metric to be on par with the accuracy of segment-level and document-level evaluation metrics, except for the WMT23 he-en. The authors are not familiar with Hebrew so are not aware of the reason.

Note that the WD metric does not exploit the fact that the sentences of the documents are aligned in the same order. Thus, it uses less information than the baselines. The result shows that the proposed method is on par with the state-of-the-art metric, while it is a robust metric that is applicable to a wide range of tasks where the disclosure structures can vary.

4.2 Document-Level Translation

The use of LLMs for document-level translation is shown to be effective due to their ability to comprehend long-context texts (Wang et al., 2023).

We use WMT24 En-De and En-Ja language pairs for the evaluation (Kocmi et al., 2024). We generate 32 samples using Llama-3.1 (meta-llama/Llama-3.1-8B-Instruct) as a language model (Dubey et al., 2024). See Appendix D for the prompt used.

We use MetricX-23 (google/metricx-23-xl-v2p0) as the utility function (Juraska et al., 2023). Because the output from the MetricX-23 models is a score in the range [0, 25] where lower is better, we inverted and rescaled it to [0, 1]. For an evaluation, we use MetricX-23 with EWD (\( \epsilon = 0.1 \)) to evaluate document-level text generation. Given that MetricX-23 with EWD shows a high correlation with human evaluation in Table 1, we foresee it to be a valid metric for the task. In addition to using MetricX as a utility function, we also evaluate using SentBERT, COMET-22, and SFR-2 (Salesforce/SFR-Embedding-2_R) as a reference. We use a bigger MetricX-23 model

<table>
  <tr>
    <th>MetricX-23</th>
    <th>En-Ja</th>
    <th>En-De</th>
  </tr>
  <tr>
    <td>Beam</td>
    <td>61.57</td>
    <td>79.07</td>
  </tr>
  <tr>
    <td>MBR (SentBERT)</td>
    <td>56.26</td>
    <td>78.66</td>
  </tr>
  <tr>
    <td>MBR (COMET-22)</td>
    <td>60.55</td>
    <td>80.92</td>
  </tr>
  <tr>
    <td>MBR (SFR-2)</td>
    <td>57.38</td>
    <td>76.88</td>
  </tr>
  <tr>
    <td>MBR (MetricX)</td>
    <td>68.81</td>
    <td>82.02</td>
  </tr>
  <tr>
    <td>MBR-LA (MetricX)</td>
    <td>70.01</td>
    <td>80.77</td>
  </tr>
  <tr>
    <td>MBR-LAL (MetricX)</td>
    <td>71.01</td>
    <td>83.13</td>
  </tr>
  <tr>
    <td>MBR-WD\(^c\) (MetricX)</td>
    <td>68.07</td>
    <td>83.77</td>
  </tr>
  <tr>
    <td>MBR-WD\(_L^c\) (MetricX)</td>
    <td>70.67</td>
    <td>83.24</td>
  </tr>
  <tr>
    <td>MBR-WD (MetricX)</td>
    <td><b>75.29</b></td>
    <td>83.40</td>
  </tr>
  <tr>
    <td>MBR-WD\(_L\) (MetricX)</td>
    <td>72.38</td>
    <td>83.24</td>
  </tr>
</table>

Table 2: Comparison of MBR-OT methods with LA, WD, and EWD. The evaluation metric is MetricX-23-XXL with EWD.

<table>
  <tr>
    <th>COMET-22</th>
    <th>En-Ja</th>
    <th>En-De</th>
  </tr>
  <tr>
    <td>CALM2-DPO</td>
    <td>EuroLLM</td>
    <td></td>
  </tr>
  <tr>
    <td>MBR</td>
    <td>79.65</td>
    <td>83.03</td>
  </tr>
  <tr>
    <td>MBR-WD\(^c\)</td>
    <td>83.54</td>
    <td><b>85.13</b></td>
  </tr>
  <tr>
    <td>MBR-WD\(_L^c\)</td>
    <td><b>84.08</b></td>
    <td>84.25</td>
  </tr>
</table>

Table 3: Evaluation of MBR-WD\(^c\) on WMT24 with 32 samples on LLMs specifically trained for the languages (cyberagent/calm2-7b-chat-dpo-experimental and utter-project/EuroLLM-1.7B-Instruct). The evaluation metric is MetricX-23-XXL with EWD.
Figure 2: Evaluation of MBR-OT on document-level machine translation tasks. WD metric with MetricX-23 as a sentence-level utility function is used as the evaluation metric.

(google/metricx-23-xxl-v2p0) to alleviate the overfitting problem of MBR decoding (Kovacs et al., 2024). We additionally evaluate with COMET-22 (Unbabel/wmt22-comet-da; Rei et al. 2022) in Appendix A.

Comparison of LA, WD, and EWD. Table 2 shows the performance of MBR-OT using LA, WD, and EWD as the formulation of the optimal transport. We observe that MBR-WD\( ^\epsilon \) and MBR-WD outperform the baselines.

We observe the BLEU scores of En-Ja are low compared to the state-of-the-art models (Table 4). Llama-3 tends to generate a shorter summary of the original English document rather than a precise translation. The average length of the generated texts is 642.34 characters, whereas the reference texts average 860.61 characters. In several cases, the generated outputs are less than one-third the length of the reference translations. Table 5 summarizes the number of characters, sentences, and the average number of characters per sentences generated by MBR and MBR-OT.

For the rest of the paper, we conduct experiments using EWD with \( \epsilon = 0.10 \), as its performance is only marginal below WD, and EWD is known to be robust to model noises.

Evaluation of MBR-OT. Figure 2 shows the performance of the algorithms on Llama-3.1 and Table 3 on LLMs specifically trained for the target languages. The result shows that the performance of MBR-OT is outperforming the baselines in both language pairs.

<table>
  <tr>
    <th>BLEU</th>
    <th>En-Ja</th>
    <th>En-De</th>
  </tr>
  <tr>
    <td>Beam</td>
    <td>7.75</td>
    <td>17.44</td>
  </tr>
  <tr>
    <td>MBR</td>
    <td>10.01</td>
    <td>18.42</td>
  </tr>
  <tr>
    <td>MBR-WD\( ^\epsilon \)</td>
    <td>9.66</td>
    <td>18.26</td>
  </tr>
  <tr>
    <td>MBR-WD\( ^\epsilon_L \)</td>
    <td>9.82</td>
    <td>18.42</td>
  </tr>
</table>

Table 4: BLEU scores of the MBR-WD\( ^\epsilon \) and MBR on WMT24 with 32 samples. mecab-python3 library (Kudo et al., 2004) is used for tokenizing Japanese texts. Note that the lexical metric is shown to have little correlation with human evaluation. The values are reported for reference.

4.3 Document-Level Simplification

Document-level simplification task is a combination of document summarization and a text simplification task (Sun et al., 2021; Blinova et al., 2023). The goal is to generate a short and easily readable summary of the given document so that the information is accessible to children and those who are learning the language. We use the first 300 entries of the JADOS dataset for the task (Nagai et al., 2024). The dataset contains articles in Japanese and their summaries, which were manually written by native Japanese speakers. We use the Wikipedia subset of the JADOS dataset as it is open-sourced.

We generate 32 samples using Llama-3.1 as a language model. We use the SentBERT as a utility function. D-SARI (Sun et al., 2021; Blinova et al., 2023) is used as the document-level evaluation metric following Nagai et al. (2024).

Figure 3a shows the D-SARI scores of the decoding algorithms with varying number of samples. Overall, we observe MBR-OT outperforming the baselines. We additionally evaluate the readability of the texts using JReadability (Hasebe and Lee, 2015) in Table 6 which shows that the readability
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="3">MBR</th>
    <th colspan="3">MBR-WD<sup>c</sup></th>
    <th colspan="3">MBR-WD<sub>L</sub><sup>c</sup></th>
  </tr>
  <tr>
    <th>|CI|</th>
    <th>|SI|</th>
    <th>|CI/ISI|</th>
    <th>|CI|</th>
    <th>|SI|</th>
    <th>|CI/ISI|</th>
    <th>|CI|</th>
    <th>|SI|</th>
    <th>|CI/ISI|</th>
  </tr>
  <tr>
    <td>WMT24 En-Ja</td>
    <td>713.6</td>
    <td>28.4</td>
    <td>25.2</td>
    <td>633.9</td>
    <td>29.4</td>
    <td>21.6</td>
    <td>625.0</td>
    <td>29.6</td>
    <td>21.1</td>
  </tr>
  <tr>
    <td>WMT24 En-De</td>
    <td>1435.0</td>
    <td>12.3</td>
    <td>116.4</td>
    <td>1366.9</td>
    <td>11.8</td>
    <td>115.3</td>
    <td>1366.0</td>
    <td>11.6</td>
    <td>117.4</td>
  </tr>
  <tr>
    <td>JADOS</td>
    <td>338.7</td>
    <td>7.8</td>
    <td>43.6</td>
    <td>317.9</td>
    <td>7.3</td>
    <td>43.5</td>
    <td>319.3</td>
    <td>7.2</td>
    <td>44.2</td>
  </tr>
  <tr>
    <td>CNNDM</td>
    <td>488.5</td>
    <td>3.0</td>
    <td>162.8</td>
    <td>489.7</td>
    <td>2.9</td>
    <td>168.9</td>
    <td>471.1</td>
    <td>2.9</td>
    <td>165.3</td>
  </tr>
  <tr>
    <td>PP-cc12m</td>
    <td>660.5</td>
    <td>5.9</td>
    <td>111.4</td>
    <td>589.0</td>
    <td>6.9</td>
    <td>85.5</td>
    <td>625.9</td>
    <td>7.2</td>
    <td>86.7</td>
  </tr>
  <tr>
    <td>PP-commonpool</td>
    <td>565.9</td>
    <td>5.1</td>
    <td>110.3</td>
    <td>517.8</td>
    <td>6.1</td>
    <td>84.9</td>
    <td>508.0</td>
    <td>5.9</td>
    <td>86.5</td>
  </tr>
  <tr>
    <td>PP-redcaps</td>
    <td>710.5</td>
    <td>6.5</td>
    <td>109.3</td>
    <td>572.7</td>
    <td>6.8</td>
    <td>84.5</td>
    <td>586.7</td>
    <td>6.5</td>
    <td>90.1</td>
  </tr>
</table>

Table 5: The average number of characters (|CI|), sentences (|SI|), and the average number of characters per sentence (|CI/ISI|).

<table>
  <tr>
    <th>jReadability</th>
    <th></th>
  </tr>
  <tr>
    <td>Beam</td>
    <td>4.46</td>
  </tr>
  <tr>
    <td>MBR</td>
    <td>3.30</td>
  </tr>
  <tr>
    <td>MBR-WD<sup>c</sup></td>
    <td>3.25</td>
  </tr>
  <tr>
    <td>MBR-WD<sub>L</sub><sup>c</sup></td>
    <td>3.25</td>
  </tr>
</table>

Table 6: Evaluation of readability of the generated text using jReadability (Hasebe and Lee, 2015). The lower score shows better readability.

<table>
  <tr>
    <th></th>
    <th>CNNDM</th>
    <th>JADOS</th>
  </tr>
  <tr>
    <td>Beam</td>
    <td>16.32</td>
    <td>16.39</td>
  </tr>
  <tr>
    <td>MBR</td>
    <td>17.58</td>
    <td>25.39</td>
  </tr>
  <tr>
    <td>MBR-WD<sup>c</sup></td>
    <td>18.23</td>
    <td>26.63</td>
  </tr>
  <tr>
    <td>MBR-WD<sub>L</sub><sup>c</sup></td>
    <td>18.88</td>
    <td>26.56</td>
  </tr>
</table>

Table 7: ROUGE scores of the MBR-WD<sup>c</sup> and MBR on CNNDM and JADOS with 32 samples. Note that the lexical metric is shown to have little correlation with human evaluation. The values are reported for reference.

<table>
  <tr>
    <th></th>
    <th>cc12m</th>
    <th>commonpool</th>
    <th>redcaps</th>
  </tr>
  <tr>
    <td>Beam</td>
    <td>28.26</td>
    <td>24.77</td>
    <td>27.57</td>
  </tr>
  <tr>
    <td>MBR</td>
    <td>28.38</td>
    <td>25.53</td>
    <td>29.27</td>
  </tr>
  <tr>
    <td>MBR-WD<sup>c</sup></td>
    <td>27.99</td>
    <td>25.20</td>
    <td>29.03</td>
  </tr>
  <tr>
    <td>MBR-WD<sub>L</sub><sup>c</sup></td>
    <td>28.28</td>
    <td>24.33</td>
    <td>28.70</td>
  </tr>
</table>

Table 8: Evaluation of the MBR-WD<sup>c</sup> and MBR using METEOR with 32 samples. Note that the lexical metric is shown to have little correlation with human evaluation. The values are reported for reference.

of MBR-OT is on par with MBR. Table 7 shows the ROUGE scores as a reference.

4.4 Document-Level Summarization

Although the proposed method is targeted for document-level text generation tasks, it is applicable to generating short paragraphs with a couple of sentences. We evaluate the performance of MBR-OT on the first 300 entries of the CNNDM dataset where the output is around 2 to 4 sentences. CNNDM is a summarization task where the goal is to generate a short summary of the given news article. We use Llama-3.1 as a language model and SentBERT as a utility function. The result shows that the present approach has a positive impact even when the number of segments in the output is relatively small.

4.5 Dense Image Captioning

The task of dense image captioning is to generate a caption of an image containing types, attributes, and counts of objects in an image, in addition to spatial relations between objects, the presence of text, various broad image categorizations, etc (Johnson et al., 2016; Krishna et al., 2017; Urbanek et al., 2024; Singla et al., 2024).

Unlike a traditional image captioning task, the caption wants to contain as much information about the image as possible.

We use the PixelProse dataset for the evaluation (Singla et al., 2024). PixelProse is a collection of images crawled web pages with curation to filter out harmful images and images that can violate privacy. We use 200 images from each of the three subsets of the dataset (cc12m, commonpool, and redcaps). We use PaliGemma-2 10B (Steiner et al., 2024b) fine-tuned on DOCCI dataset (Onoe et al., 2024) to generate 32 captions for each image (google/paligemma2-10b-ft-docci-448).
We use CLIPText as a utility function (openai/clip-vit-large-patch14; Qin et al., 2023). CLAIR (Chan et al., 2023a) is used as the
metric to evaluate the outputs. CLAIR uses GPT-4 as a judge to compute the utility of the caption.

Figure 4 shows the performance of the algorithms. Overall, the performance of MBR-WD outperforms MBR decoding in all three subsets, showing that it consistently achieves high quality captioning in a wide range of images. We observe little differences between the algorithms on METEOR (Banerjee and Lavie, 2005) scores (Table 8).

5 Related Work

MBR decoding. The performance of MBR decoding is known to be dependent on the quality of the utility function (Fernandes et al., 2022; Freitag et al., 2022a; Kovacs et al., 2024). One of the problems of MBR decoding is its computational complexity. Several methods have been proposed to reduce the computational complexity of MBR decoding (Cheng and Vlachos, 2023; Jinnai and Ariu, 2024; Trabelsi et al., 2024), many of which are also applicable to MBR-OT.

Figure 3: Evaluation of MBR-OT on document-level summarization and simplification tasks.

Figure 4: Evaluation of MBR-OT on dense image captioning tasks.

The sampling strategy is an important aspect of MBR decoding. Epsilon sampling (Hewitt et al., 2022) is known to be an effective choice for MBR (Freitag et al., 2023a; Jinnai et al., 2024). We use epsilon sampling in this paper following their work.

Document-level text generation tasks. Improvement to the decoding algorithm is not the only solution to solve document-level text generation tasks.

Hendy et al. (2023) show that few-shot learning is an effective strategy to generate high quality texts on document-level translation tasks using GPT. Briakou et al. (2024) propose a method to translate a document by decomposing the translation process into steps consisting of pre-translation research,
drafting, refining, and proofreading. Jiang et al. (2022) proposes a metric specific to the evaluation of document-level machine translation tasks.

Blinova et al. (2023) proposes to use both the explicit summarization model and the simplification model to generate the final output. Cripwell et al. (2023) proposes a method to plan a simplification as a sequence of four simplification operations (copy, rephrase, split, or delete).

Because the objective of dense image captioning is to extract as much information from the given image, many of the studies have proposed methods to extract information from the image effectively. The importance of dense image captioning is recognized by the study by Krishna et al. (2017) is one of the first studies to show the use of rich annotation data for images. Hao et al. (2024) combines a detection model, a vision-language model, an OCR model, and other CV tools to generate dense image captions.

While these methods achieve domain-specific optimization to improve the performance of the given task, our contribution is to present a general-purpose decoding algorithm that is independent of the domain, except for its length.

6 Conclusions

We evaluate the performance of MBR-OT, a variant of MBR decoding using optimal transport distance to compute the document-level utility using a segment-level utility function. Compared to the prior work on document-level metric for machine translation (Vernikos et al., 2022), the advantage of MBR-OT is that it is naturally applicable to tasks where there are many options for disclosure structures in the output text. WD can adapt to the reordering, merging, and separation of disclosures, making it applicable to many tasks without engineering task-dependent optimization.

The empirical result shows that MBR-OT outperforms MBR decoding in document-level text generation tasks including machine translation, text simplification, and dense image captioning.

7 Limitations

MBR-OT requires additional computational overhead on MBR decoding which is already known to be computationally intensive. In our experiments using NVIDIA A100 GPUs with 80 GB VRAM, MBR-OT is roughly more than four times slower than MBR with the same number of samples. We foresee it to be used combined with efficient MBR methods (Cheng and Vlachos, 2023; Jinnai and Ariu, 2024) in practice. Also note that approximation algorithms to compute optimal transport distance faster have been proposed (Altschuler et al., 2017; Dvurechensky et al., 2018; Sommerfeld et al., 2019), which may enable us to compute MBR-OT faster.

Our study is limited to simple formulations of OT where the ordering of the segments are ignored. This may result in cases where the ordering of the segments matters. For example, the meaning of the pronoun is dependent on the ordering:

(1) Bob likes cats. Charles likes dogs.
*He* always takes pictures of them.

(2) Charles likes dogs. Bob likes cats.
*He* always takes pictures of them.

In the first text, “He” refers to Charles and “them” refers to dogs, but in the second text, they refers to Bob and cats. Therefore, the third sentence has semantically different meanings in the context. However, the formulations we present in the paper cannot distinguish such a difference. This suggests that the use of additional context proposed by Vernikos et al. (2022) may be also beneficial to the OT utility functions. Alternative approach is to use a more sophisticated OT formulation such as Fused Gromov-Wasserstein (Titouan et al., 2019) to take into account the structures of the segments.

We consider the weight of the segments to be uniform or propotional to its length (Eq. 9 and 10). Evaluation of other approaches such as using probability mass or entropy would be future work.

We focus on directed text generation tasks. Open-ended text generation tasks such as story generation is an interesting future work.

The evaluation is limited to moderately long documents with a couple of paragraphs. The evaluation of the method for much longer text generation tasks (Liang et al., 2023; Zhang et al., 2024; Hsieh et al., 2024) is future work. For generating much longer texts, we require the segmentation of paragraphs instead of sentences to align larger semantic blocks. How to build a hierarchy of segmented texts from a long text is not a trivial problem that we need to investigate for applying MBR-OT to these tasks.

As mentioned in Section 5, there are many methods proposed for each specific document-level text generation task. For example, several methods have
been proposed for document-level machine translation (Voita et al., 2019; Kudo et al., 2024). Evaluation of MBR-OT combined with task-dependent optimizations is future work.

The study depends on automated metrics. Although the metrics we used in the study are known to have a high correlation with human evaluation, they are not flawless. Human evaluation is desirable for evaluation.

8 Impact Statements

This work was conducted using existing, publicly available, including WMT datasets, JADOS, Pixel-Prose, and CNNNDM. The datasets are constructed as benchmarks for research use in the research community (Table 14).

As for MBR, MBR-OT is not designed to prevent the system from generating toxic texts. Thus, it requires a countermeasure besides MBR-OT, such as language model alignment (Ouyang et al., 2022; Eisenstein et al., 2023; Rafailov et al., 2023).

Acknowledgments

We thank the anonymous reviewers for their insightful comments on the manuscript, which enabled us to conduct a more fine-grained analysis of the experiments. We would like to thank our colleagues at CyberAgent AI Lab, Kenshi Abe, Kaito Aritu, Tetsuro Morimura, Mitsuki Sakamoto, Yuma Fujimoto, and Ryota Mitsuhashi for their encouragement and for the helpful discussions during the early stages of this work.

References

Jason Altschuler, Jonathan Niles-Weed, and Philippe Rigollet. 2017. Near-linear time approximation algorithms for optimal transport via sinkhorn iteration. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.

Waïss Azizian, Franck Iutzeler, and Jérôme Malick. 2023. Regularization for wasserstein distributionally robust optimization. ESAIM: Control, Optimisation and Calculus of Variations, 29:33.

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72. Ann Arbor, Michigan. Association for Computational Linguistics.

Sofia Blinova, Xinyu Zhou, Martin Jaggi, Carsten Eickhoff, and Seyed Ali Bahrainian. 2023. SIMSUM: Document-level text simplification via simultaneous summarization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9927–9944, Toronto, Canada. Association for Computational Linguistics.

Eleftheria Briakou, Jiaming Luo, Colin Cherry, and Markus Freitag. 2024. Translating step-by-step: Decomposing the translation process for improved translation quality of long-form texts. In Proceedings of the Ninth Conference on Machine Translation, pages 1301–1317, Miami, Florida, USA. Association for Computational Linguistics.

David Chan, Suzanne Petryk, Joseph Gonzalez, Trevor Darrell, and John Canny. 2023a. CLAIR: Evaluating image captions with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13638–13646, Singapore. Association for Computational Linguistics.

David M Chan, Suzanne Petryk, Joseph E Gonzalez, Trevor Darrell, and John Canny. 2023b. Clair: Evaluating image captions with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore. Association for Computational Linguistics.

Julius Cheng and Andreas Vlachos. 2023. Faster minimum Bayes risk decoding with confidence-based pruning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12473–12480, Singapore. Association for Computational Linguistics.

Liam Cripwell, Joël Legrand, and Claire Gardent. 2023. Document-level planning for text simplification. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 993–1006, Dubrovnik, Croatia. Association for Computational Linguistics.

Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.

Petros Drineas and Michael W. Mahoney. 2005. On the nystrom method for approximating a gram matrix for improved kernel-based learning. Journal of Machine Learning Research, 6(72):2153–2175.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Pavel Dvurechensky, Alexander Gasnikov, and Alexey Kroshnin. 2018. Computational optimal transport: Complexity by accelerated gradient descent is better
than by sinkhorn’s algorithm. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1367–1376. PMLR.

Bryan Eikema and Wilker Aziz. 2020. Is MAP decoding all you need? the inadequacy of the mode in neural machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4506–4520. Barcelona, Spain (Online). International Committee on Computational Linguistics.

Bryan Eikema and Wilker Aziz. 2022. Sampling-based approximations to minimum Bayes risk decoding for neural machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10978–10993, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. 2023. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244.

António Farinhas, José G. C. de Souza, and André F. T. Martins. 2023. An empirical study of translation hypothesis ensembling with large language models. arXiv.

Patrick Fernandes, António Farinhas, Ricardo Rei, José De Souza, Perez Ogayo, Graham Neubig, and Andre Martins. 2022. Quality-aware decoding for neural machine translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1396–1412, Seattle, United States. Association for Computational Linguistics.

Mara Finkelstein and Markus Freitag. 2024. MBR and QE finetuning: Training-time distillation of the best and most expensive decoding methods. In The Twelfth International Conference on Learning Representations.

Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. 2021. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1–8.

Markus Freitag, Behrooz Ghorbani, and Patrick Fernandes. 2023a. Epsilon sampling rocks: Investigating sampling strategies for minimum Bayes risk decoding for machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9198–9209, Singapore. Association for Computational Linguistics.

Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022a. High quality rather than high model probability: Minimum Bayes risk decoding with neural metrics. Transactions of the Association for Computational Linguistics, 10:811–825.

Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Marianna Buchichio, Chrysoula Zerva, and Alon Lavie. 2024. Are LLMs breaking MT metrics? results of the WMT24 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pages 47–81, Miami, Florida, USA. Association for Computational Linguistics.

Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023b. Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent. In Proceedings of the Eighth Conference on Machine Translation, pages 578–628, Singapore. Association for Computational Linguistics.

Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022b. Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46–68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Vaibhava Goel and William J Byrne. 2000. Minimum bayes-risk automatic speech recognition. Computer Speech & Language, 14(2):115–135.

Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2024. xcomet: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995.

Kamil Gutmann, Mikolaj Pokrywka, Adrian Charkiewicz, and Artur Nowakowski. 2024. Chasing COMET: Leveraging minimum Bayes risk decoding for self-improving machine translation. In Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), pages 80–99, Sheffield, UK. European Association for Machine Translation (EAMT).

Jing Hao, Yuxiang Zhao, Song Chen, Yanpeng Sun, Qiang Chen, Gang Zhang, Kun Yao, Errui Ding, and Jingdong Wang. 2024. Fullanno: A data engine for enhancing image comprehension of mllms. arXiv preprint arXiv:2409.13540.
Yoichiro Hasebe and Jae-Ho Lee. 2015. Introducing a readability evaluation system for japanese language education. In Proceedings of the 6th international conference on computer assisted systems for teaching & learning Japanese, pages 19–22.

Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at machine translation? a comprehensive evaluation. arXiv preprint arXiv:2302.09210.

Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, volume 28.

John Hewitt, Christopher Manning, and Percy Liang. 2022. Truncation sampling as language model desmoothing. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3414–3427, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

John T. Holodnak and Ilse C. F. Ipsen. 2015. Randomized approximation of the gram matrix: Exact computation and probabilistic bounds. SIAM Journal on Matrix Analysis and Applications, 36(1):110–137.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.

Eduard Hovy and Laurie Gerber. 1997. MT at the paragraph level: improving English synthesis in SYS-TRAN. In Proceedings of the 7th Conference on Theoretical and Methodological Issues in Machine Translation of Natural Languages, St John’s College, Santa Fe. ?? Not mentioned on TOC.

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. RULER: What’s the real context size of your long-context language models? In First Conference on Language Modeling.

Yuchen Jiang, Tianyu Liu, Shuming Ma, Dongdong Zhang, Jian Yang, Haoyang Huang, Rico Sennrich, Ryan Cotterell, Mrinmayya Sachan, and Ming Zhou. 2022. BlonDe: An automatic evaluation metric for document-level machine translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1550–1565, Seattle, United States. Association for Computational Linguistics.

Yuu Jinnai and Kaito Ariu. 2024. Hyperparameter-free approach for faster minimum Bayes risk decoding. In Findings of the Association for Computational Linguistics ACL 2024, pages 8547–8566, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Yuu Jinnai, Tetsuro Morimura, Ukyo Honda, Kaito Ariu, and Kenshi Abe. 2024. Model-based minimum Bayes risk decoding for text generation. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 22326–22347. PMLR.

Justin Johnson, Andrej Karpathy, and Li Fei-Fei. 2016. Densecap: Fully convolutional localization networks for dense captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzaadeh, and Markus Freitag. 2023. MetricX-23: The Google submission to the WMT 2023 metrics shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 756–767, Singapore. Association for Computational Linguistics.

Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masasaki Nagata, Martin Popel, Maja Popović, Mariya Shmatova, Steinhoer Steingrimsson, and Vilém Zouhar. 2024. Findings of the WMT24 general machine translation shared task: The LLM era is here but MT is not solved yet. In Proceedings of the Ninth Conference on Machine Translation, pages 1–46, Miami, Florida, USA. Association for Computational Linguistics.

Geza Kovacs, Daniel Deutsch, and Markus Freitag. 2024. Mitigating metric bias in minimum Bayes risk decoding. In Proceedings of the Ninth Conference on Machine Translation, pages 1063–1094, Miami, Florida, USA. Association for Computational Linguistics.

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shanma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32–73.

Keito Kudo, Hiroyuki Deguchi, Makoto Morishita, Ryo Fujii, Takumi Ito, Shintaro Ozaki, Koki Natsumi, Kai Sato, Kazuki Yano, Ryosuke Takahashi, Subaru Kimura, Tomomasa Hara, Yusuke Sakai, and Jun Suzuki. 2024. Document-level translation with LLM reranking: Team-J at WMT 2024 general translation task. In Proceedings of the Ninth Conference on Machine Translation, pages 210–226, Miami, Florida, USA. Association for Computational Linguistics.

Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto. 2004. Applying conditional random fields to Japanese morphological analysis. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 230–237, Barcelona, Spain. Association for Computational Linguistics.
Shankar Kumar and William Byrne. 2002. Minimum Bayes-risk word alignments of bilingual texts. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002), pages 140–147. Association for Computational Linguistics.

Shankar Kumar and William Byrne. 2004. Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 169–176, Boston, Massachusetts, USA. Association for Computational Linguistics.

Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 957–966, Lille, France. PMLR.

Xiaobo Liang, Zecheng Tang, Juntao Li, and Min Zhang. 2023. Open-ended long text generation via masked language modeling. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 223–241, Toronto, Canada. Association for Computational Linguistics.

Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742.

Daniel Marcu, Lynn Carlson, and Maki Watanabe. 2000. The automatic translation of discourse structures. In 1st Meeting of the North American Chapter of the Association for Computational Linguistics.

Yoshinari Nagai, Teruaki Oka, and Mamoru Komachi. 2024. A document-level text simplification dataset for Japanese. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 459–476, Torino, Italia. ELRA and ICCL.

Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, Su Wang, and Jason Baldridge. 2024. Docci: Descriptions of connected and contrasting images. In Computer Vision – ECCV 2024, pages 291–309, Cham. Springer Nature Switzerland.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Alamed, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgium, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Justin Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Hee woo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondratiuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emry Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Fe-
Ipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillett, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayaragya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024. GPT-4 technical report. arXiv preprint arXiv:2303.08774.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Curran Associates, Inc.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA. Association for Computational Linguistics.

Gabriel Peyr´e and Marco Cuturi. 2020. Computational optimal transport. arXiv preprint arXiv:1803.00567.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.

Libo Qin, Weiyun Wang, Qiguang Chen, and Wanxiang Che. 2023. CLIPText: A new paradigm for zero-shot text classification. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1077–1088, Toronto, Canada. Association for Computational Linguistics.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems.

Miguel Ramos, Patrick Fernandes, Ant´onio Farinhas, and Andre Martins. 2024. Aligning neural machine translation models: Human feedback in training and inference. In Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), pages 258–274. Sheffield, UK. European Association for Machine Translation (EAMT).

Ricardo Rei, Jos´e G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andr´e F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Vasu Singla, Kaiyu Yue, Sukriti Paul, Reza Shirikvand, Mayuka Jayawardhana, Alireza Ganjdanesh, Heng Huang, Abhinav Bhatele, Gowthami Somapalli, and Tom Goldstein. 2024. From pixels to prose: A large dataset of dense image captions. arXiv preprint arXiv:2406.10328.

Max Sommerfeld, Jörn Schrieber, Yoav Zemel, and Axel Munk. 2019. Optimal transport: Fast probabilistic approximation with exact solvers. Journal of Machine Learning Research, 20(105):1–23.

Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. MPNet: Masked and permuted pre-training for language understanding. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, et al. 2024a. Paligemma 2: A family of versatile vlms for transfer. arXiv preprint arXiv:2412.03555.

Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, Siyang Qin, Reeve Ingle, Emanuele Bugliarello, Sahar Kazemzadeh, Thomas Mesnard, Ibrahim Alabdmulhoins, Lucas Beyer, and Xiaohua Zhai. 2024b. Paligemma 2: A family of versatile vlms for transfer. arXiv preprint arXiv:2412.03555.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, volume 33, pages 3008–3021. Curran Associates, Inc.

Renliang Sun, Hanqi Jin, and Xiaojun Wan. 2021. Document-level text simplification: Dataset, crite-
ria and baseline. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7997–8013, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Vayer Titouan, Nicolas Courty, Romain Tavenard, Chapel Laetitia, and Rémi Flamary. 2019. Optimal transport for structured data with application on graphs. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6275–6284. PMLR.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esioubi, Jude Fernandes, Jeremy Fu, Wenjin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Firas Trabelsi, David Vilar, Mara Finkelstein, and Markus Freitag. 2024. Efficient minimum bayes risk decoding using low-rank matrix completion algorithms. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero-Soriano. 2024. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26700–26709.

Giorgos Vernikos, Brian Thompson, Prashant Mathur, and Marcello Federico. 2022. Embarrassingly easy document-level mt metrics: How to convert any pretrained metric into a document-level metric. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Cédric Villani. 2021. Topics in optimal transportation, volume 58. American Mathematical Soc.

Elena Voita, Rico Sennrich, and Ivan Titov. 2019. Context-aware monolingual repair for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 877–886, Hong Kong, China. Association for Computational Linguistics.

Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023. Document-level machine translation with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16646–16661, Singapore. Association for Computational Linguistics.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierre Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.

Ian Wu, Patrick Fernandes, Amanda Bertsch, Seungone Kim, Sina Pakazad, and Graham Neubig. 2024. Better instruction-following through minimum bayes risk. arXiv preprint arXiv:2410.02902.

Renqiu Xia, Song Mao, Xiangchao Yan, Hongbin Zhou, Bo Zhang, Haoyang Peng, Jiahao Pi, Daocheng Fu, Wenjie Wu, Hancheng Ye, et al. 2024. Docgenome: An open large-scale scientific document benchmark for training and testing multi-modal large language models. arXiv preprint arXiv:2406.11633.

Yiming Yan, Tao Wang, Chengqi Zhao, Shujian Huang, Jiajun Chen, and Mingxuan Wang. 2023. BLEURT has universal translations: An analysis of automatic metrics by minimum risk training. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5428–5443, Toronto, Canada. Association for Computational Linguistics.

Guangyu Yang, Jinghong Chen, Weizhe Lin, and Bill Byrne. 2024. Direct preference optimization for neural machine translation with minimum Bayes risk decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 391–398, Mexico City, Mexico. Association for Computational Linguistics.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.

Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai,
Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024. xBench: Extending long context evaluation beyond 100K tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15262–15277, Bangkok, Thailand. Association for Computational Linguistics.

Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 563–578, Hong Kong, China. Association for Computational Linguistics.

A Additional Evaluation using COMET-22

Using the same evaluation metric as the utility function that MBR decoding uses is known to cause a bias in the evaluation. Prior work has shown that the metric overestimates the performance of MBR decoding using the evaluation metric (Kovacs et al., 2024).

Thus, the evaluation on Section 4.2 may exhibit bias. To this end, we additionally evaluate the outputs using COMET-22 (Unbabel/wmt22-comet-da; Rei et al. 2022) as a verification of the evaluation.

Tables 9 and 10 show the comparison of MBR-OT methods using varying OT formulations. Overall, we observe that MBR-WD outperforms the baselines.

Figure 5 shows the performance of MBR-WD with varying numbers of hypotheses. Overall, the same qualitative results are observed as in Figure 2.

B Analysis of MBR-OT on WMT24 En-Ja

To further investigate the behavior of MBR-OT, we computed the average scores for each domain in the WMT24 En-Ja task (Table 11). Note that the number of documents is uneven (news = 17, speech = 111, literary = 8, and social = 34). The performance of MBR-WD (MetricX) is better than MBR (MetricX) in all four categories. Interestingly, in the social domain, where MBR drops the score compared to the other domains, MBR-WD achieves a score on par with the other domains. We speculate that this is because documents in social domains are sourced from SNS (Mastodon) (Kocmi et al., 2024) and are less structured than those in news domains sourced from online news sites. OT may not offer a strong advantage in the news domain as the document structure is relatively fixed.

Note that the analysis is based on a small number of samples, so further investigation is needed for more fine-grained analysis.

<table>
  <tr>
    <th>COMET-22</th>
    <th>En-Ja</th>
    <th>En-De</th>
  </tr>
  <tr>
    <td>Beam</td>
    <td>58.14</td>
    <td>67.19</td>
  </tr>
  <tr>
    <td>MBR (SFR2)</td>
    <td>53.98</td>
    <td>61.72</td>
  </tr>
  <tr>
    <td>MBR (MetricX)</td>
    <td>67.76</td>
    <td>70.77</td>
  </tr>
  <tr>
    <td>MBR-LA (MetricX)</td>
    <td>64.12</td>
    <td>72.56</td>
  </tr>
  <tr>
    <td>MBR-LA_L (MetricX)</td>
    <td>64.56</td>
    <td>73.17</td>
  </tr>
  <tr>
    <td>MBR-WD (MetricX)</td>
    <td>66.82</td>
    <td>72.70</td>
  </tr>
  <tr>
    <td>MBR-WD_L (MetricX)</td>
    <td><b>70.46</b></td>
    <td>73.13</td>
  </tr>
  <tr>
    <td>MBR-WD^e (MetricX)</td>
    <td>66.82</td>
    <td>73.62</td>
  </tr>
  <tr>
    <td>MBR-WD^e_L (MetricX)</td>
    <td>70.23</td>
    <td><b>73.86</b></td>
  </tr>
</table>

Table 9: Comparison of MBR-OT methods with LA, WD, and EWD using COMET-22. Llama-3.1 is used as the text generation model.

<table>
  <tr>
    <th>COMET-22</th>
    <th>En-Ja</th>
    <th>En-De</th>
  </tr>
  <tr>
    <td>MBR</td>
    <td>83.26</td>
    <td>75.44</td>
  </tr>
  <tr>
    <td>MBR-WD^e</td>
    <td>83.06</td>
    <td>76.27</td>
  </tr>
  <tr>
    <td>MBR-WD^e_L</td>
    <td><b>83.55</b></td>
    <td><b>76.35</b></td>
  </tr>
</table>

Table 10: COMET-22 scores of the MBR-OT and MBR on WMT24 with 32 samples on LLMs specifically trained for the languages.

C Generation Examples on WMT24 En-Ja

Below are the generation examples of the Llama-3.1 model by random sampling. The source English document is the following.

Source document (Abbreviated)

Siso’s depictions of land, water center new gallery exhibition "People Swimming in the Swimming Pool" from 2022 is one Vicente Siso artwork that will display at Tierra del Sol Gallery beginning Jan. 13. (photo courtesy of Vicente Siso) Tierra del Sol is pleased to present "Vicente Siso: Memories of the Land and Water" at the new gallery location in West Hollywood. Siso has been an artist in the Studio Arts Program since 2012, this marks his debut solo exhibi-
<table>
  <tr>
    <th>Category</th>
    <th>Overall</th>
    <th>News</th>
    <th>Speech</th>
    <th>Literary</th>
    <th>Social</th>
  </tr>
  <tr>
    <td>Beam</td>
    <td>61.57</td>
    <td>63.22</td>
    <td>60.55</td>
    <td>56.01</td>
    <td>65.37</td>
  </tr>
  <tr>
    <td>MBR (SentBERT)</td>
    <td>56.26</td>
    <td>72.46</td>
    <td>54.38</td>
    <td>52.51</td>
    <td>60.35</td>
  </tr>
  <tr>
    <td>MBR (COMET-22)</td>
    <td>60.55</td>
    <td><b>77.06</b></td>
    <td>58.15</td>
    <td>56.17</td>
    <td>61.16</td>
  </tr>
  <tr>
    <td>MBR (SFR-2)</td>
    <td>57.38</td>
    <td>70.40</td>
    <td>54.35</td>
    <td>61.23</td>
    <td>59.86</td>
  </tr>
  <tr>
    <td>MBR (MetricX)</td>
    <td>68.81</td>
    <td>73.32</td>
    <td>71.14</td>
    <td>68.48</td>
    <td>59.03</td>
  </tr>
  <tr>
    <td>MBR-LA (MetricX)</td>
    <td>70.01</td>
    <td>77.34</td>
    <td>72.64</td>
    <td><b>73.62</b></td>
    <td>66.93</td>
  </tr>
  <tr>
    <td>MBR-WD<sup>c</sup> (MetricX)</td>
    <td>68.07</td>
    <td>74.64</td>
    <td>70.41</td>
    <td>58.77</td>
    <td>59.31</td>
  </tr>
  <tr>
    <td>MBR-WD (MetricX)</td>
    <td><b>75.29</b></td>
    <td>77.00</td>
    <td><b>75.94</b></td>
    <td>74.96</td>
    <td><b>72.40</b></td>
  </tr>
</table>

Table 11: Evaluation of the MetricX scores on the WMT’2024 En-Ja. The number of documents is news=17, speech=111, literary=8, and social=34.

tion. ... "Vicente Siso: Memories of the Land and Water" opens on Saturday, Jan. 13, with a reception from 6-8 p.m. The exhibition is on view through Sunday, March 3. The Tierra del Sol Gallery is located at 7414 Santa Monica Blvd. For information, visit tierradelsolgallery.org.

Generation example 1

Sisoの作品、「2022年に製作された「Swimming Poolの人間たち」」が、2023年1月13日からTierra del Sol Galleryで展示されることになります。この展示は、西ハリウッドの新しいギャラリーで開催される「Vicente Siso:ランドと水の記憶」というタイトルのグループ展です。...「Vicente Siso:ランドと水の記憶」は、2023年1月13日に開幕し、6-8 p.m.の受け入れを含み、3月3日まで展示します。ティエラデルソル・ギャラリーは、サンタモニカブリビュー7414号にあります。情報は、tierradelsolgallery.orgにアクセスしてください。

Generation example 2

2022年の作品「People Swimming in the Swimming Pool」が、西海岸のTierra del Sol ギャラリーで展示されるVICENTE SISOの作品です。Tierra del Sol ギャラリーは、ロサンゼルス西部の新しいギャラリーで、VICENTE SISOの個展「Memories of the Land and Water」を開催します。

...

VICENTE SISOの個展「Memories of the Land and Water」は、1月13日の土曜日6-8pmに開催され、3月3日の日曜日まで展示されます。Tierra del Sol ギャラリーは、サンタモニカ・ブールバード7414番地に所在しており、tierradelsolgallery.orgで情報を閲覧できます。

Generation example 3

シゾ・ソロ展「Vicente Siso：Land and Waterのメモリー」

新しいギャラリー、ティエラ・デル・ソル・ギャラリーにおいて、2022年の「People Swimming in the Swimming Pool」を始まり、1月13日から出展します。（写真：Vicente Siso）...「Vicente Siso：Land and Waterのメモリー」は、1月13日(土)6-8pmのオープン・リセプションに始まり、3月3日(日)まで開催されます。ティエラ・デル・ソル・ギャラリーは、サンタ・モニカ・ブールバード7414番地にあります。詳細はtierradelsolgallery.orgを参照してください。

The “。” character serves as the period in Japanese, so most texts are segmented based on this character. In the examples, the bolded text highlights segments where the structure differs from the rest of the generated outputs. In the second example, two sentences from the original English document are merged into a single sentence. In the
Figure 5: Evaluation of MBR-OT on document-level machine translation tasks using COMET-22. Llama-3.1 is used as the text generation model.

third example, the first sentence of the document appears to be interpreted as the title of the article.

Although the document-level MT is a directed text generation task, the generated texts from the LLM may have various structures.

D Prompts for Text Generation

We use the following prompt for the machine translation tasks in Section 4.2.

Translate the following paragraph from English to [[LANGUAGE]].

[[QUESTION]]

Translate this paragraph from English to [[LANGUAGE]].

We use the following prompt for the JADOS dataset (Section 4.3).

以下のWikipediaの記事を、小学生でも分かるやさしい日本語で要約してください。

記事: [[QUESTION]]
Q: この記事をやさしい日本語で要約してください。
A:

The text below is the above prompt translated into English.

Please summarize the following Wikipedia article in simple Japanese that even an elementary school student can understand.

Article: [[QUESTION]]
Q: Please summarize this article in simple Japanese.
A:

For the dense image captioning (Section 4.5), we use the same prompt as Singla et al. 2024:

Describe the image in detail. Please specify any objects within the image, backgrounds, scenery, interactions, and gestures or poses. If they are multiple of any object, please specify how many and where they are. If any text is present in the image, mention where it is, and the font. Describe the text in detail with quotation marks. For example, if the image has text, Merry Christmas, write it down as “Merry Christmas”. Describe the style of the image. If there are people or characters in the image, what emotions are they conveying? Identify the style of the image, and describe it as well. Please keep your descriptions factual and terse but complete. The description should be purely factual, with no subjective speculation. Make sure to include the style of the image, for example cartoon, photograph, 3d render etc.

E Hyperparameters

Table 12 shows the hyperparameters used for text generation. We use epsilon sampling (Hewitt et al., 2022) for all the experiments as it is shown to be effective for generating samples for MBR decoding (Freitag et al., 2023a; Jinnai et al., 2024).
<table>
  <tr>
    <th>Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>Temperature</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>top_p</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>epsilon_cutoff</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>max_new_tokens</td>
    <td>1024</td>
  </tr>
</table>

Table 12: Hyperparmeters for text generation

<table>
  <tr>
    <th>Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>Temperature</td>
    <td>0.3</td>
  </tr>
  <tr>
    <td>Version</td>
    <td>2024-05-13</td>
  </tr>
</table>

Table 13: Hyperparmeters for GPT-4

Table 13 shows the hyperparameters we use for GPT-4 evaluation and for CLAIR in Section 4.5.

F Reproducibility Statement

All datasets and models used in the experiments are publicly available. The code is implemented using Huggingface’s Transformers library (Wolf et al., 2020). The computation of Wasserstein distance is implemented by POT: Python Optimal Transport library (Flamary et al., 2021). For Section 4.1, we use the codebase of Vernikos et al. (2022)4 and use MT Metrics Eval V25 for the evaluation. Our code is available at https://github.com/jinnaiyuu/mbr-optimal-transport.

The experiments are conducted using NVIDIA A100 GPUs with 80 GB VRAM. The total amount of GPU time for the study is estimated to be 100-1000 GPU hours.

4https://github.com/amazon-science/doc-mt-metrics
5https://github.com/google-research/mt-metrics-eval
Table 14: List of datasets and models used in the experiments.

<table>
  <tr>
    <th>Name</th>
    <th>Reference</th>
  </tr>
  <tr>
    <td>WMT22 Metric Task</td>
    <td>(Freitag et al., 2022b) https://www.statmt.org/wmt22/metrics/index.html</td>
  </tr>
  <tr>
    <td>WMT23 Metric Task</td>
    <td>(Freitag et al., 2023b) https://wmt-metrics-task.github.io/</td>
  </tr>
  <tr>
    <td>WMT24 General Task</td>
    <td>(Kocmi et al., 2024) https://www2.statmt.org/wmt24/translation-task.html</td>
  </tr>
  <tr>
    <td>CNNDM</td>
    <td>(Hermann et al., 2015) https://github.com/google-deepmind/rc-data</td>
  </tr>
  <tr>
    <td>JADOS</td>
    <td>(Nagai et al., 2024) https://github.com/tmu-nlp/JADOS</td>
  </tr>
  <tr>
    <td>PixelProse</td>
    <td>(Singla et al., 2024) https://huggingface.co/datasets/tomg-group-umd/pixelprose</td>
  </tr>
  <tr>
    <td>Llama-3.1</td>
    <td>(Dubey et al., 2024) https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct</td>
  </tr>
  <tr>
    <td>PaliGemma-2</td>
    <td>(Steiner et al., 2024a) https://huggingface.co/google/paligemma2-10b-ft-docci-448</td>
  </tr>
  <tr>
    <td>MetricX-23</td>
    <td>(Juraska et al., 2023) https://huggingface.co/google/metricx-23-xxl-v2p0</td>
  </tr>
  <tr>
    <td>MPNet</td>
    <td>(Song et al., 2020) https://huggingface.co/sentence-transformers/all-mpnet-base-v2</td>
  </tr>
  <tr>
    <td>CLIP</td>
    <td>(Radford et al., 2021) https://huggingface.co/openai/clip-vit-large-patch14</td>
  </tr>
  <tr>
    <td>COMET-22</td>
    <td>(Rei et al., 2022) https://huggingface.co/Unbabel/wmt22-comet-da</td>
  </tr>
  <tr>
    <td>D-SARI</td>
    <td>(Sun et al., 2021) https://github.com/jinnaiyuu/mbr-optimal-transport Implemented by the authors.</td>
  </tr>
  <tr>
    <td>JReadability</td>
    <td>(Hasebe and Lee, 2015) https://github.com/joshdavham/jreadability</td>
  </tr>
  <tr>
    <td>CLAIR</td>
    <td>(Chan et al., 2023b) https://github.com/davidmchanclair</td>
  </tr>
</table>

Reviewer paper summaries: - They proposed document-level minimum Bayes risk (MBR) decoding using optimal transport (OT).
- They also proposed optimization methods to speed up MBR-OT.
- The proposed metric achieved higher correlations than sentence-level metric and the commonly used baseline (Vernikos et al., 2022). MBR-OT outperformed the sentence-level MBR in document-level translation, text simplification, summarization, and dense image captioning.

This paper presents a method that applies Minimum Bayes Risk Decoding (MBRD) for the document-level text generation task. The challenge in the approach lies in adapting the existing sentence-level metrics to the document-level task, given the sentence ordering and document structure changes between the generated text and reference. To address this, an optimal transport based aggregation function is proposed and evaluated. In particular, using the Wasserstein distance, the aggregation can attribute a source element to any number of target elements, which essentially allows flexible alignments between source and target. The empirical evaluation is conducted on several different document-level text generation tasks, including document-level machine translation, text simplification, and dense image captioning.
Summary of strengths: - They tackled document-level generation in terms of MBR decoding. The most previous MBR decoding only considered the sentence-level generation.
- The transportation cost between a hypothesis h and a reference y is formulated as 1-u(h,y), which is simple and can be used with any utility functions.
- The proposed MBR-OT outperformed the sentence-level MBR in document-level translation, text simplification, summarization, and dense image captioning.
- In addition, they also evaluated the accuracy of the WD metric and confirmed it achieved higher correlations than the sentence-level metric and the commonly used baseline (Vernikos et al., 2022). This suggests that the document-level evaluation extending the sentence-level evaluation metrics with OT is practical.

1) Document-level text generation is an important task, especially given the popularity of LLMs.
2) The proposed method is evaluated on a diverse set of document-level text generation tasks.
3) Overall the paper is well-motivated and clearly written.
Summary of weaknesses: - Why do you assume p(h_i) and p(y_j) as uniform distributions? If they are generated according to the P_model, their probability mass or entropy can be used. For example, intuitively, meaningless segments consisting only of fillers would not have the same weight as other informative segments.
- Equantion 11: Some evaluation metrics like BLEU, MetricX, and COMET-22 are asymmetry between a hypothesis and a reference. Did you discard either score? Could you show the difference from the results without this approximation?
- How much longer does MBR-OT take to choice the hypotheses compared with the standard MBR? I would also like to see the relationship between the execution time of the naive MBR-OT and the speed-quality trade-off due to the effects of the proposed optimizations.
- Table 4: Even if BLEU does not have enough correlation with the human judgement, 7.75% in the beam search is too low. Could you please check if there are any problems with the base translations or the evaluation? (In the evaluation of En-Ja translation, sacrebleu should tokenize target sentences with the `ja-mecab` tokenizer.)
- Lack of comparison with document-aware generation methods
    - DocRepair is one of the commonly used baselines in document-aware generation: [https://aclanthology.org/D19-1081](https://aclanthology.org/D19-1081/)
    - This paper [https://aclanthology.org/2024.wmt-1.14/](https://aclanthology.org/2024.wmt-1.14/) proposes two algorithms of document-level MBR decoding (Algorithm 1 and 2; (g) and (k) in Table 4), which are the baseline of MBR-OT.

It seems to me, the main motivation of the proposed OT-based metric that it adapts to text structure changes better (Line 125) is not well supported. It would better to show some in-depth analyses on this. 

In particular, based on Table 1, it seems the proposed method no longer outperforms baselines when the sentence-level metric is strong (MetricX-23). This seem to contradict this assumption.
Comments, suggestions and typos: ## Notation
- L084: \mathcal{Y} is undefined.
- L143: missing \ldots in y = {...}
- What is the domain of \bm{x}, \bm{h}, and \bm{y}? In L081, \bm{h} is defined as “a candidate output”, i.e., a sentence, but \bm{h} is a set of sentences in L143.
- L225: Is C(h_i, h_j) correct?
- L225: What is u_{s}()?

## Questions for clarification
- “sacrebleu” is a name of evaluation toolkit that contains BLEU, chrF, and TER. Did you use BLEU implemented on sacrebleu?
- Which model did you use for BERTScore?

1) Line 161: it seems Linear Assignment can also handle the example in Figure 1. Maybe it would be useful to give one example where Wasserstein distance is better than LA?
2) nit: There are several small mistakes in citation. e.g., Line 403: `Figure X` -> `Figure 3`. Table 5~7 are not cited in the text.
Overall assessment scores: 3.5

3.5