EMO: EARTH MOVER DISTANCE OPTIMIZATION FOR AUTO-REGRESSIVE LANGUAGE MODELING
Authors: Siyu Ren
Zhiyong Wu
Kenny Q. Zhu
Venue: ICLR 2024 poster (2024)
Keywords: language modeling
earth mover distance
language generation
language understanding
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Neural language models are probabilistic models of human text. They are predominantly trained using maximum likelihood estimation (MLE), which is equivalent to minimizing the forward cross-entropy between the empirical data distribution and the model distribution. However, various degeneration phenomena are still widely observed when decoding from the distributions learned by such models. We establish that the forward cross-entropy is suboptimal as a distance metric for aligning human and model distribution due to its (1) recall-prioritization (2) negative diversity ignorance and (3) train-test mismatch. In this paper, we propose Earth Mover Distance Optimization (EMO) for auto-regressive language modeling. EMO capitalizes on the inherent properties of earth mover distance to address the aforementioned challenges. Due to the high complexity of direct computation, we further introduce a feasible upper bound for EMO to ease end-to-end training. Upon extensive evaluation of language models trained using EMO and MLE. We find that EMO demonstrates a consistently better language modeling performance than MLE across domains. Moreover, EMO demonstrates noteworthy enhancements in downstream performance with minimal fine-tuning on merely 25,000 sentences. This highlights the tremendous potential of EMO as a lightweight calibration method for enhancing large-scale pre-trained language models.

Published as a conference paper at ICLR 2024
EMO: EARTH MOVER DISTANCE OPTIMIZATION FOR AUTO-REGRESSIVE LANGUAGE MODELING
Siyu Ren 1 ZZiyonng Wult, Kenny Q. Zhu
'Shanghai Jiao Tong University "Shanghai AI Laboratory 'University of Texas at Arlington roy0702 jtu. edu. en, wuzhiyong@p org. en, kenny. zhu@uta..... edu
ABSTRACT
Neural language models are predominantly trained using maximum likelihood es- timation (MLE), which is equivalent to minimizing the forward cross-entropy be- tween the empirical data distribution and the model distribution. However, various degeneration phenomena tributions learned by such models. We establish that the forward cross-entropy is suboptimal distance metric for aligning human and model distribution due its (1) recall-prioritizatiin
still widely observed when decoding from the dis-
(2) negative diversity ignorance and (3) train-test mis- match. In this paper, we propose Earth Mover Distance Optimization (EMO) for auto-regressive language modeling. EMO capitalizes on the inherent properties of earth mover distance address the aforementioned challenges. Due to the high complexity of direct computation, we further introduce a feasible upper bound for EMO to ease end-to-end training. Upon extensive evaluation, EMO demonstrates a consistently better language modeling performance than MLE across domains. Moreover, EMO shows noteworthy enhancements in downstream performance with minimal fine-tuning merely 25,000 sentences, highlighting its potential as a lightweight calibration method for enhancing large-scale pre-trained language models. Code available at
INTRODUCTION
The dominant paradigm of natural language generation systems hinges on probabilistic neural lan- guage models (Radford et al. 2019 Zhang et al.) 2022), which permit evaluating the probability of any given text sequence as well as generating novel ones using various decoding strategies upon learned distributions (Holtzman et al. 2019 Meister al. 2023b). Language modeling, the process of aligning model distribution with that of human language, is usually formulated as a sequence prediction task in which maximum likelihood estimation (MLE) is typically adopted the training objective owing to its simplicity and intuitiveness.
However, various text degeneration phenomena with incoherent and nonsensical (LeBrun et al. 2021 Holtzman et al. 2019) content are still widely observed in text generated from language models pre-trained on massive amounts of human data. This indicates that the model distribution Qe (parametrized by f) learned by MLE still differs substantially from the human language distri- bution P, despite having seemingly low training loss (Meister et al. 20236). From a distributional view, training with MLE is equivalent to minimizing the forward cross-entropy between P and Qo:
CE(P, Qa) = Ploog Qo(x))
(1)
We argue that the forward cross-entropy has inherent limitations as metric for matching model distribution and that of human language. Firstly, forward cross-entropy is recall-prioritized (Meister et al. 2023a). At each time step, it focuses exclusively increasing the model likelihood of the ground-truth next token. This can result in poor precision of the learned model distribution when training data noisy slow convergence even when sufficient amounts of high-quality text corpus are available. Secondly, when used in language model pre-training, forward cross-entropy faces
Work done during an internship at Shanghai AT Laboratory. 1 "Corresonndence to: Zhiyong Wu, Kenny Q. Zhu, partly supported by National Science Foundation (NSF) Grant No. 2349713, and University of Texas STARs program.

---

Published as a conference paper at ICLR 2024
the negative diversity ignorance issue (Li et al., 2019) where all non-ground-trine truth next tokens deemed as equally incorrect. However, some tokens might be less incorrect or even plausible al- ternatives to the ground truth than other tokens. Capturing these latent negative diversity can assist language models in enhancing their modeling of the human language distribution. Thirdly, the form of forward cross-entropy is inconsistent with how language models are evaluated (Pang & He 2020)_ Such train-test objective mismatch makes MLE a less reliable indicator of modeling quality.
To alleviate the aforementioned limitations of MLE, we direct our attention towards an alternative distance metric, namely the Earth Mover Distance (EMD) (Kantorovich, 1960). EMD is initially discussed in the context of optimal transport problem (Villami et al., 2009) and then incorporated as a distance metric for implicit generative modeling, e.g., WGAN (Arjovsky et al.] (017) and WAF (Tolstikhin et al., 2018). The appeal of EMD lies (1) takes into account both precision and recall during modeling; (2) it acknowledges the varying degrees of correctness in data samples, enabling more nuanced training signals. (3) its mathematical formulation permits better consistency between the training and testing phases. Given these properties, we incorporate EMD as better token-level probability distance measure into language modeling. However, computing the exact value of EMD requires external solvers that detached from the computation graph and block gradient back-propagan propaagtttt... We overcome this issue by developing an differentiable upper bound of EMD (DEMD) that can be optimized in end-to-end manner without resorting external specialized solvers. Combined with a semantically informed transport cost function, we present EMO (Earth Mover Distance Optimization) for training auto-regressive language models.
We first evaluate the effectiveness of the proposed method on the task of language modeling across diverse domains and show that EMO yields generations with significantly higher distributional close- ness (6.2 points on average measured by MAUVE) with human text. We further demonstrate that, by applying EMO in lightweight fine-tuning stage using several orders of magnitude fewer tokens than pre-training, pre-trained LLMs' performance on a range of downstream language understand- ing tasks can be significantly boosted, e-g., an average improvement of 4 points across 8 datasets. By progressively increasing the volume of data utilized for continual fine-tuning, EMO also demon- strates superior scaling properties compared to existing methods.
BACKGROUND AND MOTIVATION
2.1 AUTO-REGRESSIVE LANGUAGE MODELING
Current language generation systems are predominantly based on probabilistic neural auto- regressive language models (LMs) (Bengio et al. 2000). Denoting a language model parametrized by 0 as Qe. it essentially computes the probability of a given text sequence the product of each token's 's conditional probability given preceding tokens:
Qo(x) = II @0(2:/=21)
(2)
Prior works have adopted various neural architectures, e.g., LSTM (Hochreiter & Schmidhuber 1997). GRU (Chung et al., 2014), and now the most widely used Transformer (Vaswani et al, 2017 to transform natural language input to next token probability. To estimate 0, the most com- mon approach is to perform seff-supervised pre-training on enormous volume of text corpus using maximum likelihood estimation. Given the data distribution P, the training objective of MLE is equivalent to minimizing the forward cross-entropy between P and Qe:
LM CE(P, Qo)
- -En~ [log Qo(x))
(3)
[a|
= - E
log Q0
(4)
= E P
CE(P() Q+(1Iz2))
(5)
From Eq. to Eq. 5] the sentence-level cross-entropy is further decomposed into the sum of forward cross-entropy between token-level data distribution P(: ((())))))) and model distribution (=
2

---

Published as a conference paper at ICLR 2024
2.2 DEFICIENCY OF MAXIMUM LIKELIHOOD ESTIMATION
In this subsection, we will delve into certain properties of the forward cross-entropy employed in MLE training, and elucidate the impact of these properties on the learned model distribution.
2.2.1 RECALL-PPRIORTIZZTIIO
A series of recent works (ucicc et al. 2018 Sajjadi et al. 2018; Djolonga et al. 2020) have gen- eralized the classification metric recall to measure the quality of generative modeling- Specifically, recall here is defined as the model distribution Q3 s coverage of data distribution P. i.e., high recall means that high likelihood tokens under P' shall also have high likelihood under Qs. In contrast, the precision of Qe focuses measuring whether low-quality tokens (unlikely under P) have low probabilities under Qo. To elaborate further, we derive the gradient of forward cross-entropy w.r.t model parameters 9 as follows:
lani voQolailze)) Qo (xet))
COLLLE = - PE
(6)
Eq. [ clearly shows that, by minimizing LMLE via gradient descent, Qo is encouraged to only assign high probability to the ground-truth next token and therefore being recall-prioritized Conse- quently, the precision of Qa is not adequately incentivized in MLE because 64. not explicitly discourage learning of low-quality tokens. In short, recall-rrioritization results in insufficient opti- mization of Qa's 's precision and amplifies the need for enormous amounts of high-quality text corpus to overcome this limitation.
2.2.2 NEGATIVE DIVERSITY IGNORANCE
Another noteworthy property of MLE is its ignorance of diverse supervision signals of non-ground truth tokens during auto-regressive language modeling Zhang & Zhao (2018); Li et al. (2019). Specifically, MLE assumes token 2, observed in training sample x is the only ground-truth token at time step t and maximizes its log-likelioood under Qa- Concurrently, the remaining tokens other than 2, in the vocabulary in MLE. This can be demonstrated by analyzing the partial derivative of CE(P(-) Qo (-(2)=))) w.r the output logits before softmax:
treated equally incorrect, and their probabilities are implicitly penalized
aCE(P())rc)) Qo(-(ze1))
(Q((x)) - =f
Ozi
(Q((:) others
where denotes the i-th token in the vocabulary. To reach a local minimum during gradient-based optimization (gradient norm - 0), the model will try increase the probability of Iz (Qu(z;) - 1) and decrease the probability of all other tokens (Qu(2;) + 0). In practice, however, certain tokens can serve as plausible alternatives to Zt, e.g., synonyms of sr- The training objective should assign high probabilities to those tokens rather than penalize them as did in MLE. In essence, such an inability of MLE may inhibit building more powerful neural models of human language that can accurately distinguish the relative correctness of the next token.
2.2.3 TRAIN-TEST MISMATCH
When training is completed, language models are often evaluated against objectives that differ sig- nificantly from MLE. For example, ROUGE (Lin) 2004) for summarization and BLEU (Papineni et al. 2002) for machine translation. This creates a train-test mismatch for language modeling. In other words, we draw sample from Qe and then assess its quality using certain evaluation function i.c. maximizes Ez~qo [f(x)( where f(-) varies according to different downstream scenar- ios. This is inconsistent with MLE in which the expectation is taken w.r.t data distribution P, i.e., E,wr(log Qo(z)). Most prior works have attempted to address this issue by incorporating the eval- uation objective f(-) into training and adopting reward-augmi awwrrd-ugmmeted maximum likelihood Norouzi et al. (2016) Zhang & Zhao (2018); Liu et al. (2022) based on the policy gradient theorem (Sut- ton et al.] 1999) or contrastive learning. However, such changes incur non-trivial overhead, and the choices of evaluation function f(.) are usually task-specific and less applicable for general lan- guage modeling. In light of this, there is a critical need for objectives that exhibit better train-test consistency enhance the efficacy of language modeling.
3

---

Published as a conference paper at ICLR 2024
3 EMO: EARTH MOVER DISTANCE OPTIMIZATION
In pursuit of divergence measure that circumvents the adverse properties of forward cross-entropy, we draw our attention the Earth Mover's Distance (EMD), a distance function that was originally studied in the context of optimal transport planning of goods and materials (Kantorovich, 1960.
Villani 202T) and then borrowed for generative modeling by ML community (Arjovsky et al. 2017 Tolstikhin et al.) 2018). In Section 3.1 we provide the formal definition of EMD and elucidate its adaptation for auto-regressive language modeling with a semantically informed cost function. In Section 3.3 tackle the challenge posed by the intractable infimumm associated with EMD by developing its upper bound. Collectively, we introduce EMO, an approach dedicated to the training of auto-regressive language models through the optimization of the Earth Mover's Distance.
3.1 ADAPTING EARTH MOVER'S DISTANCE TO AUTO-REGRESSIVE LANGUAGE MODELING
Formally, given two probability distributions Pi and P2 over metric space X, the earth mover's distance between P1 and P2 is defined as the minimum accumulative cost moving all probability mass of P1 to Ps:
inf ((er,)))yy(((m) EMD(PI, PE)
(8)
where inf stands for infinitesimal, II(PI, P2) denotes the set of all joint distributions 7(2 x2) whose marginals are P1 and P2, respectly. r(21, x2) is interpreted as the amount of probability mass transported from Pi (111 to P2(222). C(21, 22) is non-negative function that measures the cost of transporting a unit mass from 21 to x2. In the context of auto-regressive language modeling, Pi refers to the model distribution to learned and P3 refers to the data distribution, both representing the locally factorized probability distribution over the next token at time step / given preceding tokens, i.., Pi == Qo(-Iazd and P3 == (() Thus, Eq. can be reformulated as:
inf (MD)( = (((e))
B(an, (C(21,
M M
inf (em(oo(F
ΣΣ
(16 Uy Clu
where V is the vocabulary of language model and v; indexs the i-th token in V. Once the cost function C is defined, computing the above earth mover's distance amounts solve the following constrained linear optimization problem:
M
min
(0, 23 U,)CCu vz)
(10)
8.1
(w;, Uz) = P(us(te)) Vi (l,...,.]
3
M
7(%,, = Qo(0y(z<)) Vj (1....//VB
Semantically-Inie Informed Transport Cost The next step is to establish a definition of C such that it reflects a meaningful distance between pairs of tokens v; and vy. Intuitively, tokens that are more likely to be used interchangeably should have smaller distances, e.g., glad and happy. Conversely, tokens that are improbable to within each other's context, e.g., cat and galaxy, should be farther away. One such measure of token distance is naturally provided by their cosine distance in the
contextual embedding space, i.e., C(v,, vy) = 1 e, where e, is the i-th column of the language modeling head E of LM Q pre-trained via MLE. Because during training e; is optimized to be close to the contextual representation of all prefixes of which the next token is v; the cosine distance between e; and e, therefore serves i and Uy. Once initialized, E will be jointly updated with other model parameters so that C' can capture the shift in the domain-specific distribution from the pre-training corpus.
effective proxy for quantifying the transport cost between
4

---

Published as a conference paper at ICLR 2024
3.2 A TRACTABLE UPPER BOUND
The complexity of traditional EMD solvers (Ling & Okada 2007 Shirdhonkar & Jacobs 2008) for computing Eq is O[[V
( log [V]). which becomes burdensome for recent LLMs whose vo- cabulary can contain several tens of thousands of tokens. Additionally, employing external solvers disrupts gradient propagation, making end-to-end training infeasible. To tackle these challenges, we present a tractable upper bound of EMD that allows for efficient gradient-based optimization.
We start by defining transport plan y that directly leverages the data distribution P(- and model distribution Qo(-|zes) meanwhile being valid by adhering to the constraints stated in Sec. 3.1
7(0,, v/) = (o(e:)
(11)
Here omit the prefix F<t for notational simplicity. Essentially; represents the probability of a data-dependent transport plan that moves the probability mass of v; under Qo to other tokens according the proportions specified by P. Since both Qe and P add up to 1, 5 therefore
a legitimate but not necessarily optimal plan. Denoting the unknown optimal plan with minimal transport cost as y", we have the following inequality holds:
V
IV
EMD(Qa, P)
F(us C)),
1=1
IV
M
(12)
i=1 j=1
= QJ
(13)
(11T- - É E)P
(BQQ)
(14)
where C' E RRVV VI is the matrix notation of C(oi, 0z) used to transform the summation (Eq. into quadratic form (Eq. 13). is all-one column vector, É is the row-wise normalized version of E, and is the one-hot next token distribution. Prior works on distribution matching using EMD/Arjovsky et al. (2017): Gulrajani et al. (2017) commonly resort to the Kantorovich- Rubinttten
12
duality (Villani (2008) or entropic regularization (Cuturi) 2013) |Frogner et al.) (015), which either conduct adversarial training of the generative model with additional I-Lipschitz critic network or adopt Sinkhorn-like iteration algorithm. In contrast, the upper bound we derived above only pertains to the training of Qg, therefore being more stable and efficient for optimization. We term Eq. 14 as
DEMD and incorporate it in conjunction with MLE A.3 for auto-regressive language modeling.
Generalized Form for Arbitrary P When P is dense, the optimal solution of Eq. 14 is a one- hot distribution with all probability mass placed on the token with the smallest expected transport cost, rather than P. To tackle this, we derive the following generalized form for arbitrary P, which minimizes the absolute difference between the surrogate transport cost of Qo and P:
DEMD(Qu. P) = QQ] - PTPC 2 CQTCP-PTCR CPI
(15)
3.3 BEHAVIORAL DIFFRRENCES COMPARED TO MILE
Next, we delve into some properties of the proposed DEMD and provide insights how it improves over MLE in terms of behavioral differences during optimization. To begin with, we first present DEMD's gradient with respect to model parameters (assuming a one-hot P):
|V|
V P)
(eQu(v))
U(),,))) =
j=1
(16)
Harmonizing Recall and Precision MLE is shown to be recall- eccoll-prioiitizing in the sense that its gradient update only ensures the target token is assigned with high probability. As result, MLE- induced model tends to be over-confident on the low-quality regions in human language. In con- trast, at each time step, DEMD also takes into account the precision of Qo by explicitly penalizing
5

---

Published as a conference paper at ICLR 2024
low-quality tokens, i.e, those tokens will have large transport costs and thus large penalties. By effectively alleviating the overestimation of degenerated text, EMO better operationalize the harmo- nization of recall and precision compared to MLE.
Negative Diversity Awareness The awareness of diverse supervisory signals of all tokens natu- rally arises from the recall-precision balancing property of DEMD. From Eq. [6]ee can see that, the update of model parameters 0 in DEMD comprises the sum of the gradients of the model's token probabilities across the entire vocabulary. weighted by their expected transport cost. Specifically, by employing gradient descent, tokens that deviate significantly from the data distribution (resulting in higher transport costs) will be down-weighted more severely than tokens that are contextually similar the data distribution. Thus, the model distribution Qo learns to allocate probability mass more accurately than MLE due to the availability of more informative training signals.
Better Train- Test Consistency One notable downside of forward cross-entropy its train-test disparity nature (Sec. 2.2.3.. Namely, during the training phase, its objective involves pectation that is computed with respect to the data distribution P, whereas during testing, sam- ples are drawn from the model distribution Qo and evaluated by humans. By rewriting [q[[2] as Busequ | P(v,)C(v,, vj]], we can see that DEMD explicitly involves the optimization of the expected transport cost computed with respect to Qo. Therefore, DEMD has a higher degree of train-test consistency compared to MLE.
4 EXPERIMENT
We demonstrate EMO's empirical performance as a continual fine-tuning method upon pre-trained LMs. In Sec. we compare EMO against MIE as well as other training criteria on a diverse range of language modeling datasets. In Sec. 4.2] we investigate the effectiveness of EMO on natural language understanding tasks under the few-shot in-context learning setting based on LLMs with various scales. The evaluation of EMO Instruction-tuning scenario is deferred to Appendix C
4.1 LANGUAGE MODELING
4.1.1 SETUP
Task Definition and Evaluation Metric To gauge the quality of the learned model distribution after fine-tuning to continue with a segment of text that should ideally be similar to the reference text. We adopt Mauve (Pillutla al. 2021) the main evaluation metric, which compares the generated continu- ation against human text by calculating the area under the KL divergence curve and has seen wide usage in open-ended text generation (Ji et al. 2022 Zhang et al. 2023 Meister al. 2023a).
domain-specific corpus, we provide the model with prefix and request
Pre-trained Language Models We utilize two representative decoder-only Transformer Vaswwani et al. 2017) language models, namely GPT-2 (Radford et al.] 2019) and OPT-125M (Zhang et al.) 2022 as Qo. and fine-tune them using distinct training criteria including EMO and several recently proposed methods discussed below.
Baselines Aside from MLE, we also compare with the following baselines, which aim to address the shortcomings of MLE by introducing novel divergence measures (or approximated variants) as their training objectives: (1) TaiLr (Ji et al. 2022) adopts the total variation distance (Van Handel] 2014] more robust measure between probability distributions and uses its token-level factor- ization as the training objective. (2) MixCE (Zhang et al., 2023) penalizes low-quality samples by leveraging approximated version of reverse cross-entropy. For TaiLr and MixCE, follow the implementations from their corresponding official codebase. More discussions regarding these baselines can be found in the Appendix A.I
Datasets We use 6 English textual corpora from 5 different domains for comprehensive evalua- tion:(1) WikiText-2 and WikiText- 103 (Merity et al. 2016) are two commonly used language mod- eling benchmarks consisting of high-quality Wikipedia articles. (2) WebText (Radford et al.,

---

Published as a conference paper at ICLR 2024
2018) is the test set of the official WebText dataset from OpenAI, that was used to train GPT-2. contains Wall Street Journal material in financial 2018 features text from the writing prompts forum of Reddit. (5) AG News (Zhang et al. 2015) is a collection of news articles from diverse domains, e.g., business, sports, and science. The statistics of each dataset deferred to the Appendix A.4
(3) Penn Tree Bank (PTB) (Mareus et al.
1993)
domain. (4) WritingPrompts (Fan et al.
Training Details We fine-tune GPT-2 and OPT-125M for 3 epochs on the training set of each dataset and save the model checkpoint with the lowest validation loss. We use the AdamW (Loshchilov & Hutter 2018) optimizer with a learning rate of 5e-5. The batch size is fixed as 32 for all experiments. The maximum input length during training is set to 256. For Tailr and MixCE that involve weighting coefficient, we conduct a hyperparameter sweep within {0.9, 0,8, 0.7}. EMO does not necessitate any hyperparameter tuning.
Decoding Algorithm To gauge the quality of the learned model distribution Qo in faithful way (Eikema & Aziz, 2020). we employ unbiased sampling (also known as ancestral sampling) the primary decoding algorithm throughout the experiments. The length of prefixing and gener- ated tokens for each dataset can be found Appendix A.4 We repeat the sampling process 5 times for each prefix and report the average Mauve score.
4.1.2 MAIN RESULTS
Table 1: Unbiased sampling results (Mauvet) of models fine-tuned by EMO as well as compared baselines. Numbers the mean of 5-run sampling, aggregated over different random seeds. Bold numbers indicate the results are significantly better than MLE with p-value < 0.001.
Model
Objective
Wiki'ext2
WikiText103
WebText
PTB
WritingPrompts
AG
MLE
77.5
77.1
75.5
76.1
83,6
75.0
TaiLr
79.6
78.0
76.5
73.8
84.1
75.8
GPT-2
MixCE
78.3
77.6
76.3
76.9
82.7
76.6
EMO
87.5
82.1
80.5
79.6
87.4
84,9
MLE
77.2
75.8
74.7
83.6
84.1
82.1
TaiLr
78.4
75.2
74.2
82.2
83.4
81.8
1PT
MixCE
78.6
75.4
75.3
81.5
83.5
83.2
EMO
82.9
81.0
80.7
86.1
87.9
84.8
[ the unbiased sampling results of GPT-2 and OPT-1255X fine-tuned with different training objectives on six datasets. We can clearly observe that EMO consistently outperforms MLE and other recently proposed training criteria across various domains. Although TaiLr and MixCE both leverage new distance measures that have theoretical advantages over forward cross-entropy, they suffer from either a mild assumption about the model's training dynamics or degeneration into regularized version of forward cross-entropy. Therefore, they still exhibit the same drawbacks of MI_E stated in Sec. 2.2 In contrast, EMO effectively manifests its theoretical advantages and leads to language models with more human-like distribution. For more quantitative results about the learned model distribution please refer to Appendix A.4.I
4.1.3 EXPERIMENT WITH ORACLE DATA GENERATOR
In addition to the setting where we only have access to training data sampled from unknown dis- tribution, in this subsection we seek to analyze more fine- fine-graines grained distributional properties of mod- els trained with different criteria. Specifically, we use training data sampled from an orcale GPT- 2-Large model whose distribution P is known. We use the GPT-2-output dataset consisting of 250,000/5,000/5.00 paragraphs in the training/validationneee set generated via unbiased sampling.
Setup Apart from Mauve for measuring the sequence-level similarity between texts sampled from the learned and oracle model, we also incorporate model's 's test set perplexity PPL the oracle model's 's perplexity PPLoracl (caleulated using oracle model model-generatee texts) and ROUGE- 1/L (Lin 2004) that evaluate the learned distributions from different perspectives. PPL is com- monly adopted as quantitative measure of model's recall. PPI onacle emphasizes more on precision

---

Published as a conference paper at ICLR 2024
by penalizing generations from Qo that unlikely to be produced by the oracle model P, i.e., P(x)) is low, While ROUGE score focuses on recall by rewarding high n-gram overlap. For each training method, fine-tune GPT-2 using the same experimental setting described in Sec. 4.1.1
Unbiased sampling results of GPT-2 fine-tuned with different training criteria. Numbers are the mean of 5-run sampling, aggregated over different random seeds. Bold numbers indicate the results significantly better with p-value < 0.001.
Table 2:
Methods
PPL +
PP 4
Mauvet
ROUGE-11
ROUGE-L
MLE
70.1
114.46
77.5
34.59
29.85
TaiLr
73.5
95.22
77.4
34.95
30.09
MixCE
74.4
79.46
78.4
35.31
30.26
EMO
74.9
55.85
83.4
37.37
31.17
Results We report the performance of EMO as well as baseline methods in Table 2 The results consistently reveal that EMO outperforms all baseline methods across all evaluation metries, except for PPLes. Notably, EMO exhibits a significantly reduced PPLonlle compared to the baseline meth- ods, demonstrating its effective mitigation of the overestimation issue associated with low-quality text in prior divergence measures. The awareness of diversity within the range of plausible tokens in addition to the gold token is naturally reflected in EMO's higher PPL indicated by the highest MAUVE score, EMO strikes the hest balance between recall and precision, suggesting that utilizing a well-structured probability distribution distance metric as the optimization objective enables the language model to effectively balance precision and recall.
4.2
LANGUAGE UNDERSTANDING
4.2.1
SETUP
Pre-trained LLMs We adopt ILLMM-77B and LLaMa-13B (Touvron al. 2023a the pre-
trained LLMs. More results using LLaMa2-7B/13B (Touvron et al. 2023b) are deferred to Ap- pendix B due space limits.
Continual Fine-tuning We perform continual fine-tuning WikiText-109 103 using EMO and all baseline methods compared in Sec. 4.1 The corpus used for fine-tuning substantially smaller (O.1B v.s. 1.4T tokens) than the corpus used for pre-training L.Ms,, therefore being much more efficient and resource-friendly. Using the same corpus for lightweight fine-tuning, our goal here is to explore the effect of different training objectives on downstream performance. For EMO, E is initialized from the pre-trained language modeling head and stay fixed during fine-tuning.
Downstream Tasks We evaluate the fine-tuned models across an array of NLU tasks using in- context learning. Specifically, we the following datasets: Tweet Emotion (Mohammad et al., 2018), TREC (Li & Roth 2002; Hovv et al., (001). : SST-2 (Socher et al.) 2013). Subj (Conneau Kiela 2018), Customer Review (Hu & Liu 2004). Rotten Tomatoes (Pang Lee 2005), AG
News (Zhang et al. 2015), and MMLU (Hendrycks et al. 2020). Following Wu et al. (2022) 2023),
pre-trained dense retriever is used find the 8 most similar samples as the in-context demon- strations for all datasets except for MMLU, where fixed 5-shot demonstrations are used following
common practice. Prompt templates and statisties for each task can be found in Appendix A.5
4.2.2 MAIN RESULTS
From Table. we observe that continual fine-tuning using MLE often only marginally outperforms the pre-trained one and sometimes even hurts performance. The optimal performance of TaiLr and MixCE is obtained via grid search over the weighting coefficient from {0.9, 0.8, 0.1}. Notably, with- out any tunable hyperparameter, EMO yields the most significant gains across all tasks compared to existing methods upon both LLaMa-7B and LLaMa- 13B, demonstrating the broader applicability of our method in terms of tasks, and model sizes.
8

---

Published as a conference paper at ICLR 2024
Table 3: Downstream performance of LLaMa-7B/113B fine-tuned with different training objectives.
Models
Methods
TE
SST-2
TREC
Subj
CR
RT
AG
MMLU
Pre- traitrained
54.1
94.7
77.8
74.7
91.4
90.0
85.6
31.4
MLE
53.5
94.8
79.0
74.5
92.0
91.8
85.5
31.9
LLaMa-7B
TaiLr
56.2
94.9
79.6
76.8
92.0
91.9
86.3
33.2
MixCE
60.0
95.0
81.2
78.5
92.0
91.8
87.5
33,9
EMO
65.6
95.2
83.4
79.2
92.0
89.4
34.8
Pre-trained
58.5
95.6
81.2
77.4
91.2
91.0
84.5
44.5
MI.E
58.6
95.5
79.8
76.9
92.0
91.3
84.3
44.9
LLaMa-13B
Tail.r
61.9
95.5
81.0
78.5
92.3
91.4
85.6
45.9
MixCE
65.7
95.6
82.8
80.6
92.0
91.3
85.9
46.7
EMO
70.4
95.9
85.2
81.1
92.6
92.2
88.4
47.5
4.2.3 SCALING LAW OF EMO
Average accuracy
Average accuracy
MaCZ
MixCE
TVlr
"Ml"
MLE Pre-trained
VLE Fre-rannd
3.30
4M
16M
32M
100M
of tocenns seen during finestuning
Figure 1: Scaling law of EMO with respect model scale and data size.
Model Scaling To comprehensively quantify the effectiveness of EMO, we perform the previously described experiment upon OPT-1.3B/2.7B (Zhang al., 2022) in addition to LLaMa-7B/133 7B/13B and visualize the scaling curve of the task accuracy averaged over the collection of 8 datasets with respect model scale in [ig[[ (left). While MLE fails consistently improve over pre-trained models, TaiLr and MixCE both bring positive impacts when their weighting coefficients are carefully tuned. Notably, EMO shows steady improvements over other methods across all model scales.
Data Scaling We further examine how performance changes by varying data volumes during fine- tuning. monitor the change average accuracy using LLaMa- 13B and display the results in Fig. (right). MLE-tuned models exhibit certain declines in accuracy fine-tuning progresses, which can be attributed to its thcoretical deficiencies described in Sec. 2.2 TailLr and MixCE mod- erately improve over MLE. EMO shows the most significant performance boost and even matches the performance (00M-tokens-traine) MixCE with merely 4M tokens. This highlights the poten- tial of employing EMO in post-training phase refine the distribution of pre-trained LLMSs for improved downstream performance in an effective and sample-efficient manner.
5 CONCLUSION
In this work, we introduce EMO, a novel approach for training auto-regressive language models by optimizing a differentiable upper bound of the earth mover distance between the model distribution and human text distribution. Experiments on open-ended text generation demonstrate that EMO consistently outperforms MLE and its robust baseline methods across diverse domains in terms of how human-like the texts generated from fine-tuned models are. Through highly lightweight con- tinual fine-tuning phase unsupervised corpora, EMO significantly enhance downstream per- formance compared to pre- traitrd models and exhibits commendable scaling properties regarding the amount of training data, rendering it favorable for general-purpose continual fine-tuning.

---

Published as a conference paper at ICLR 2024
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial net- works. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Con- ference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp- 214-223. PMLR, 06-11 Aug 2017. URL / http:///prreeecinss.......p
m
arjovsky17a.. html
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in neural information processing systems, 13, 2000.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Au- toregressive Language Modeling with Mesh-Tensorflow, March 2021. URL https://doi,
orq/ 52811/ zenodo. 5297715
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Alexis Conneau and Douwe Kiela, Senteval: An evaluation toolkit for universal sentence represen- tations. arXiv preprint arXiv:1803.05444 2018.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. 2023.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Josip Djolonga, Mario Lucic, Marco Cuturi, Olivier Bachem, Olivier Bousquet, and Sylvain Gelly. Precision-recall curves using information divergence frontiers. In International Conference on Artificial Intelligence and Statistics, pp. 2550-5559, PMLR, 2020.
Bryan Eikema and Wilker Aziz. Is map decoding all you need? the inadequacy of the mode in neu- ral machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 4506-4520, 2020.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pp. 889-898, 2018
Charlie Frogner, Chiyuan Zhang. Hossein Mobahi, Mauricio Araya, and Tomaso A Poggio. Learn- ing with a wasserstein loss. Advances in neural information processing systems, 28, 2015.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference Learning Representations, 2019.
10

---

Published as a conference paper at ICLR 2024
Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin Yew Lin, and Deepak Ravichandran. To- ward sreantics-based answer pinpointing. In Proceedings of the First Interattinall Confer- on Human Language Technology Research, 2001. URL https:/ /wtp://wwwww.o// www.wb.rrbborg// org/
anthol Mathooooy/HOO 1069
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '04, pp. 168-177, New York, NY, USA, 2004. Association for Computing Machinery. doi: 10.1145/101405221011 1014033 URL httlh:ts///ee /ot...oo//...11.555551
ISBN 138881.
101405 1014073
Haozhe Ji, Pei Ke, Zhipeng Hu, Rongsheng Zhang, and Minlie Huang. Tailoring language genera- tion models under total variation distance. In The Eleventh International Conference on Learning Representations, 2022.
Leonid V Kantorovich. Mathematical methods of organizing and planning production. Management science, 6(4):366-422, 1960.
Benjamin LeBrun, Alessandro Sordoni, and Timothy J O'Donnell. Evaluating distributional dis- tortion in neural language modeling. In International Conference on Learning Representations, 2021.
Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of Ilms. 2023.
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment. arXiv preprint arXiv:2310.054700 2023a.
Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Heng Huang, Jiuxiang Gu, and Tianyi Zhou. Reflection-tuning Data recycling improves Ilm instruction-tuning, 2023b.
Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th Interna- tional Conference Computational Linguistics, 2002. URL httlhttp:://wer www.wa.cweeb...
anthology/C02-1155
Xuechen Li, Tianyi Zhang. Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction- followoiinnn models. https:://iiii wtt.:au...oottttttaa/aaaaaaaaaea
paala 2023c.
Zuchao Li, Rui Wang, Kehai Chen, Masso Utiyama, Eiichiro Sumita, Zhuosheng Zhang, and Hai Zhao. Data-dependent gaussian prior objective for language generation. In International Confer- ence on Learning Representations, 2019.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. htt.atppseaaccaatooooo.ooo.oooo.
Haibin Ling and Kazunori Okada. An efficient earth mover's ' distance algorithm for robust histogram comparison. IEEE Transactions Pattern Analysis and Machine Intelligence, 29(5):840-88) 2007. doi: 10. 11110//TAM..000..00555
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. BRIO: Bringing order ab- stractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 2890-2903, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 1221 https: https://aclantioooory. org./oog.orr///// acw.o22.cl--ooo..20
URİ.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- on Learning Representations, 2018.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? large-scale study. Advances in neural information processing systems, 31, 2018.

---

Published as a conference paper at ICLR 2024
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-335 1993. URL
https:/ www wwwwwb.ww.......... org/ Sttthooogy/JJ332200
Clara Meister, Tiago Pimentel, Luca Malagutti, Ethan Wilcox, and Ryan Cotterell. On the effi- cacy of sampling adapters. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 1437-1455, Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10. 16653////0023.acloooog 80
URL
Clara Meister, Tigo Pimentel, Gian Wiher, and Ryan Cotterell. Locally typical sampling. Transac- tions of the Association for Computational Linguisties, 11:102-12,, 2023b.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. Semeval- 2018 task 1: Affect in tweets. Proceedings of the 12th international workshop on semantic evaluation, pp. 1-17,2018
Mohammad Norouzi, Samy Bengio, zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, and Dale Schuurmans. Reward augmented maximum likelihood for neural struc- tured prediction. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL htttps://prooceeding...ees / 205800000he2211111ec000333333330
wwwpsper..eeeeeeee http:/ppper.illeeppppereeeeee fft.ess/aapeee//11//////// pdf
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the ACL, 2005.
Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations. Interna-
fional Conference on Learning Representations, 2020.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic
evaluation of machine translation. In Proceedings of the 40th Anmual Meeting of the Associa- tion for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002.
Association for Computational Linguistics. doi: 10.3 0..111//1073083.1773333 URL https:
//aclaattholooy..rr//22------
Baolin Peng, Chunyuan Li, Pengeheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using diver- gence frontiers. In NeurIPS, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Lan- guage models are unsupervised multitask learners. 2018. URL / mode 1s/ wtw.//aaggggeemdelll........
c clouufront.com wwt.::eeeeeeeeeeeeriaaaaaaeaeeae
Alec Radford, Jeff
Wu, Rewon Child, D. Language
Luan, Dario Amodei, and models are unsupervised multitask leam-
1lya Sutskever.
ers. 2019.
URL
https:
Rafael Rafailov, Archit Sharma. Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint 2rXivv2305./822 2023.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucie, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in neural information processing systems, 31, 2018.
12

---

Published as a conference paper at ICLR 2024
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Sameer Shirdhonkar and David W. Jacobs. Approximate earth mover's distance in linear time. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8, 2008. doi: 10. 10.1109/CVPR.200 4587662.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Pruceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. URL https: //wp:::://www..... wwwwwb.wwwbb....... org/ httpa/annhooooy/DD// -13-1170
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto- encoders. In International Conference Learning Representations, 2018.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.008 09708288, 2023b.
Ramon Van Handel. Probability in high dimension. Lecture Notes (Princeton University), 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Cédric Villani. Topics in optimal transportation, volume 58. American Mathematical Soc., 2021.
Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2009.
Cédric Villani. Optimal transport Old and new, volume 338, pp. xxii+973. 01 2008. doi: 10. 1007/978-3-540-71055--
Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang. Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. Pandalm: An auto- matic evaluation benchmark for Ilm instruction tuning optimization, 2023.
Zhenyu Wu, Yaoxiang Wang, Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Jingjing Xu, and Yu Qiao. OpenICL: An open-source framework for in-context learning. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstra- tions), pp. 489-498, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10. 33255//w12022..allddeea
URL httpt:t////// /ttp:/acllee aatccaanthlllo ogy. lgy..org/.22033 acl -eeoo.
47
Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. Self-adaptive in-context learn- ing: An information compression perspective for in-context example selection and ordering.
In Annual Meeting of the Association for Computational Linguistics, 2022. URL https:
api www.smemmmmmmlllaat...o.oooo http://eattttttaaaaaarrroorrraaa hrt.aarrrrrccoooaaa:02555555555
Huan Zhang and Hai Zhao. Minimum divergence vs. maximum margin: an empirical comparison seq2seq models. In International Conference Learning Representations, 2018.
13

---

Published as a conference paper at ICLR 2024
Shiyue Zhang, Shijie Wu, Ozan Insoy, Steven Lu, Mohit Bansal, Mark Dredze, and David Rosen- berg. MixCE:
Training autoregressive language models by mixing forward and reverse cross- entropies. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9027-9050, Toronto, Canada, July 2023. Associ- ation for Computational Linguisties. doi: 10.18653/v1/20233 10.8665.1//00233alloon....1 //aclanthology
URL https:
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Opt: Open pre-trained transformer language models. arXiv preprint 22Xiv:2005.0066. 2022.
Xiang Zhang. Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08599, 2019.
14

---

Published as a conference paper at ICLR 2024
A DISCUSSION AND EXPERIMENTAL DETAILS
A.1 DISCUSSION OF BASELINE METHODS
In this paper, we mainly compare our method baselines that attempt improve MLE by optimiz- ing distance measures beyond forward cross-entropy.
TaiLr et al. 2022) proposes to leverage the total variation distance (TVD) as a more robust alternative to the forward cross-entropy for language generation model training. Specifically, they introduce token-level factorization of the original TVD and optimize its upper bound in addition to the MLE loss. The training objective of Tailr can be written as:
Q0 (0(x)) 7 (1 (2Q0(2)(<<)) log Q0 (x())
(17)
From the form of Eq. 17 we can see that Tail only alleviates the recall- praall--rioiiizaitonn issue of MLE while still confronting the negative diversity ignorance and train-test mismatch problems.
MixCE Zhangg et al. (023) is another modification to MLE which incorporates the reverse cross- entropy into the training objective. Due the one-hot encoded token-level data distribution, the author proposes approximation and uses an interpolation coefficient to combine it with forward
cross-entropy as follows:
Cmixce = (1-))=+(==) log Q0(x)(<<1)
(18)
Both Eq. 17 and Eq. be regarded as the original LMLE multiplied by coefficient determined by tunable byper-parameter "y and model's probability (confidence) of x₁. Though attractive in terms of their original formulation, TaiLr and MixCE both degenerate into certain regularized forms of MLE, hence demonstrating limited improvements.
A.2 DIFFERENCES BETWEEN EMO AND REINFORCEMENT LEARNING FROM HUMAN FEEDBACK (RLHF)
Prevailing methodologies impart the desired behaviors into base language model through metic- ulously crafted human preferences that represent the types responses that humans find helpful. This stage, dubbed supervised fine-tuning (SFT), often happens after the initial unsupervised pre- followoinn
training on a large text dataset. Although the STF models already exhibit good capabilities, the common practice is further align their behavior with human value, procedure known as Reinforcement Learning from Human Feedback (RLHF) (Christiano et al. 2017 Ziegler
al. 2019 Bai et al. 2022).
The differences between EMO and RLHF manifest in multiple dimensions, including motivation, gradient, and application scenario. In the following, we discuss these points in detail.
The motivation behind EMO is to explore effective means to adapt language model to given human text dataset through the lens of mover distance optimization. Evaluation is thus focused on quantifying how similar the model distribution Qo(z) ( is to human text distribution Par)). contrast, RLHF prioritizes steering the behavior of the language model based on the feedback provided by specific reward model (PPO (Schul- man et al. 2017 ) or directly from existing human preference dataset (DPO (Rafailov et al,, 2023)) The evaluation is often based on human- subjective metrics such as helpful- ness and safety.
Motivation:
Gradient: The per time step gradient of EMO is the combination of gradient of probability of each token in the vocabulary, weighted by their respective expected transport costs, i.e., D.V VoQo(u; (ung m;)]. For PPO, the per time step gradient is the gradi- ent of current token's 's log probability, weighted by the reward r(x, y) and the deviation from a reference model D (Qo (e) i.c., V log Qe (x, (<c) y) log Qn(v.) ). For DPO, the per time step gradient is the gradient of the current token the preferred or dispreferred response)'s 's log probability, weighted by the incorrectness value of an implicit reward model, i.c., V log Qo (v x Ber) (03 log Qo(x)) 3 log ) Queln))
IV
Qua((s))
15

---

Published as a conference paper at ICLR 2024
Application Scenario: As general-purposs. objective for auto-regressive language mod- eling. EMO is applicable to domain-spenn -speciii mne-tuning//dappation,, instruction-tumm and continual pre-training. Currently, the use of RLHF predominantly occurs in the alignment stage after supervised fine-tuning.
A.3 DYNAMIC WEIGHTING
In situations where the language model is relatively weak (having high perplexity). DEMD may converge at slow pace due to bounded gradient scaling induced by cosine-based transport cost. To overcome this potential issue, the final loss function in EMO implemented as dynamically weighted combination of MLE and DEMD:
L = 0.5 * (LMLE + PMLE deetach() * LDMDD)
(19)
LDEED
A.4 OPEN-ENDED GENERATION
Datasets
WikiText2
Wikitext103
WritingPrompts
train samples samples
36,700
1,800,000
20,000
4,210
10,000
112,000
3,760
3,370
# of test samples
4,360
4.360
5,000
3,760
1,047
7,600
pretix length generation length
20
20
5
35
10
80
80
25
80
Table 4: Length of the provided prefix and model generations for each dataset employed in the open- ended generation experiments.
We provide the detailed statistics and settings in the open-ended generation experiment in Table. For WikiText2, Wikitext103, PTB, and AG News, we download the datasets from the HuggingFace Datasets hub. For WritingPrompts and WebText, utilize the official split provided by Zyhaang al.
2023
A.4.1 QUANTIFYING THE PRECISION-RECAL TRADEOFF
To have quantitative understanding of how MILE biased towards recall, we visualize the aver- aged token-level forward and reverse cross-entropy between Qo of GPT-2 model fine-tuned with different objectives and that of pre-trained GPT-Neo-
(Black et al. 202I) model (which serves as surrogate target distribution) in Fig. 2 TaiLr and MixCE demonstrate an improved balance be- tween precision and recall, while our proposed EMO further outperforms these two methods with significant margins.
A.5 PROMPT TEMPLATES FOR LANGUAGE UNDERSTANDING TASKS
The specific prompt templates used for each task Sec. 4.2.3 are presented in Table For MMLU, follow the prompt design in Contributors 2023)
implementation, each in- put is prepended with 8 demonstrations that are retrieved using pre-trained dense retriever based on semantic similarity. One exception is MMLU, where we adopt fixed 5-shot demonstration following previous works. We compute the perplexity for the constructed prompt corresponding each candidate answer and choose the one with the smallest perplexity as the final prediction. Evaluations are implemented based on the OpenICI. (Wu et al. 2023) library.
B ADDITIONAL RESULTS ON LANGUAGE UNDERSTANDING
The additional results of LLaMa2-7B, LLaMa2-13B and OPT-2.7B on downstream natural lan- guage understanding tasks evaluated in Sec. 4.2.3 summarized in Table. Table. 7 and Table. respectively. LLMs fine-tuned with our proposed method display notable improvements over MLE and strong baselines in most tasks.
16

---

Published as a conference paper at ICLR 2024
Mean of ForwardReverse Cross Entropy
EMO
MLE
TaiLr
MiXCE
4.0
WkkTTet22
WikiText100 Dataset
Promits
Figure 2: The average of token-level forward and reverse cross-entropy between distribution Qe of GPT-2 fine-tuned with different objectives and that of GPT-Neo-1.31 on the validation of three different datasets. The lower the value, the better the learned Qo balance precision and recall.
Tasks
Prompts
Label
Review: "<X>" positive.
Positive
Review: <<>"" It negative.
Negative
Tweet: "<X">" is anger.
Anger
Tweet: "rextlessX">" it joy.
Joy
Tweet Emotion
Tweet: "<X>" it oplimism.
Optimiso
Tweet: "<X Xx sadness.
Sadness
Question: "<Xs" It about abbreviation.
Abbreviation
Question: "<X>" about entity.
Fintiyy
Question: "<X>" It about description and abstract concept.
Description and abstract concept
TREC
Question: "<X>" It is about human being.
Human being
Question: " "x>"
about location.
Location
Question: X> It about numerical value.
Nomerical value
x> is objective.
Objective
subjective.
Subjective
Review: "<X>" positive.
Positive
CR
Reviow; xx> negative.
Negative
Review: It is nogative.
posirive.
Rotten Tomatoes
Review:
Negative
"XX> about world.
World
It about sports.
Sport
AG News
"x about business.
X It about science and technology.
Science and Technology
Table 5: Prompt templates for natural language understanding tasks used in Sec 4.2.3
is
placeholder that indicates the real input context/question.
Methods
TE
SST-2
TREC
Subj
CR
RT
AG
MMLUU
Pre-trained
56.7
95.4
74.6
76.2
93.6
91.6
86,2
43.6
MLE
58.4
95.8
72.4
74.6
93.4
92.1
86.0
43.3
TaiLr
62.4
95.9
74.6
81.0
92.8
92.5
87.4
44.4
MixCE
66.5
95.9
77.6
82.5
93.1
92.1
88.0
45.1
EMO
69.0
95.8
78.0
83.1
93.4
92.8
88.1
45.5
Table 6: Downstream task performance of LLaMa2-7B fine-tuned with different training objectives on WikiText-103
17

---

Published as a conference paper at ICLR 2024
Methods
TE
SST-2
TREC
Subj
CR
RT
AG
MMLU
Pre-trained
60.4
95.5
80.4
81.7
91.2
90.1
86.5
54.7
MLE
60.9
95.8
81.0
81.8
90.9
89.6
86.2
54.3
TaiLr
61.7
96.2
80.4
81.4
91.5
90.5
87.2
54.5
MixCE
64.4
95.7
83.6
84.7
91.2
90.4
87.4
55.0
EMO
70.7
96.2
85.8
89.9
91.2
91.7
89.6
55.2
Table 7: Downstream task performance of LLLMM22-113B fine-tuned with different training objectives on WikiText- 103.
Methods
TE
SST-2
TREC
Subj
CR
RT
AG
MMLU
Pre-trained
65.7
92.9
68.6
85.5
92.5
87.6
84.3
24.4
MLE
67.3
88.7
66.8
83.4
91.5
81.3
84.4
24.9
TaiLr
67.1
92,9
70.4
88.2
92.6
87.1
84.0
24.7
MixCE
66.8
93.1
70.6
88.8
92.8
87.4
84.1
25.0
EMO
68.6
93.7
73.6
87.6
92.8
86.0
87.6
25.6
Table 8: Downstream task performance of OPT-2.7B fine-tuned with different training objectives on WikiText-103.
C INSTRUCTION-TUNIC
The effectiveness of Large Language Models (LLMs) heavily relies on their capacity to comprehend precise instructions. These generative language models undergo training using extensive raw web data and are further refined through meticulous selection of instruction data, albeit in relatively limited amount. The process of fine-tuning with instructions plays pivotal role in harnessing the potential of LLMs. Consequently, the utility of such models is predominantly shaped by proficiency in maximizing their performance using compact instruction datasets.
To this end, we also apply EMO the instruction-tume tuminng stage of LLaMa-7B/13BB using the Alpaca- GPT4 dataset (Peng et al. 2023). In addition, we perform experiments using a more advanced instruction-tuniin dataset, i.e., Recycled Evol-Instruct-70K proposed by Li et al. (2023b), well as OpenPlatypus (Lee et al. 2023). curated dataset derived from 11 open-source datasets, primar- ily focusing on enhancing LLMs STEM and logic proficiency. We follow the standard training recipe (3 training epochs, 128 global batch size, 2e-5/1e-5 learning rate for 7B/13B models respec- tively) adopted in the original Stanford Alpaca repository Afterwards, we assess the instruction- adherence efficacy of the resulting instruction-tur tuned models by incorporating the following recent L.LM-based evaluation methods:
AlpacaEval (Li et al. 2023c). which is LLLM-based automatic evaluation that is fast, cheap, and replicable. We adopt GPT-4 as the evaluator and report the win rate against responses generated by text-davinci-003.
Auto-J (Li al. 2023a). which is 13B parameter open-source generative judge that can effectively evaluate different LLMs on how they align to human preference. We report the win and tie counts of models fine-tuning using MLE and EMO.
PandaLM (Wang et al., 2023). 7B parameter instruction LLM that aims provide reproducible and automated comparisons between different large language models.
We empirically found that both Auto-J and PandaLM fail to distinguish the differences between lengthy responses. Therefore, only apply them to evaluate models trained on Alpaca-GPT4, in which the reference responses are much shorter. As indicated in Table. C EMO-tuned LLaMa at- tains superior success rates in comparison MLE-tuned counterparts across various model sizes. The average response length (measured by the number of tokens following tokenization) for MLE and EMO are 233 and 226. respectively. This shows that EMO-tuned models are able to produce higher-quality responses without relying on GPT-4's bias towards length and verbosity. In pairwise
httpp:://gittub..cmmm
18

---

Published as a conference paper at ICLR 2024
Training Objective
AlpacaEval Win Rate( %
MLE
59.3
LLaMa-7B
EMO
68.4
MLE
70.3
LLaMa-13B
EMO
74.2
MLE
59.3
LLaMa2-7B
EMO
70.3
MLE
67.3
LLaMa2-13B
EMO
79.1
Table 9: AlpacaEval win rate of LLaMa-7B/13B and 1LaMa2-7B/133 fine-tuned with MLE and EMO on Alpaca-GPT4 against text-davinci-O00 on 805 test instructions.
Training Objective
AlpacaEval Win Rate( %)
MLE
76.2
Evol-Instruct-7KK
EMO
78.8
MLE
58.9
OpenPlatypus
EMO
63.0
Table 10: AlpacaEval win rate LLaMa2-7B fine-tuned with MLE and EMO on Recycled Evol- Instruct-70K and OpenPlatypus against text-davinci-003 805 test instructions.
evaluation performed by Auto-J and PandaLM (Fig 3 5 EMO-tuned models also achieve higher win rates over MLE, further verifying the superiority of EMO when applied to instruction- tuning. Due to higher model capacity and more comprehensive rationale for decision making, Auto-J more capable of differentiating the quality between different responses, while PandaLM consis- tently produces more "tie".
In light of the efficiency and commendable performance of Auto-J, further adopt it to compare the respones produced by MLE/EMO LLaMa2-7B against the publically available respones from wide variety fnsluwiion-followinng models the AlpacaEvval leaderboard. The win rates are shown in the table below.
Competetor Model
Evol-Instruct-7K
OpenPlaytpus
MLE
EMO
MLE
EMO
Davinci-003
91 %
93%
77%
78%
Baize-v2-7B
82%
86%
57%
59%
LLaMa2-7B-Chi
63%
65%
37%
39%
Vicuna-7B-vl.3 76%
80%
48%
52%
Zephyr-7B-alpha 70%
72%
40%
44%
Table 11: Auto-J judged win rates of MLE/EMO-tuned LLaMM2-7BB on Evol-Instruct--05 and OpenPlatypus against publically available responses from various slore-soureed and 7B-sized open- source models.
19

---

Published as a conference paper at ICLR 2024
Auto-judige Pairwise EvalustioniLLafke
Auto-Judge Pairwise Evaluation(L (MMM-3333
Wh
EMD
Te-
Figure 3: Auto-J pairwise response comparison results LLaMa-7B/133 fine-tuned with MLE and EMO on 805 test instructions from AlpacaEval.
Auto-judge (valustion(LaMM2-781
Auto-juudge Painwise (allatton/LLMaa-1111
EMC Mn
EX0O Wir
MLE Win
338
MLE Wir
* of Test Instructions
Test Instrucions
Figure 4: Auto-J pairwise response comparison results of LLaMa2-7B/13B fine-tuned with MLE and EMO on 805 test instructions from AlpacaEval.
PandaLM Pairwise Evaluation
PandaLM Painwise Eve LaMenn1
152
192
120
700
360
405
50
100
150
200
250
300
350
400
Text Inatructions
Test Instructions
Figure 5: PandaLM pairwise response comparison results of LLaMa-7B/133 fine-tuned with MLE and EMO on 805 test instructions from AlpacaEval.
20

---

Published as a conference paper at ICLR 2024
PandaLM Painwise EvaluationnL.aMa2-22
Evalusaion/LLaMs2-1133
EMO Win
END War
MLE 9in
214
MLE Win-
1.76
350
100
200
200
of Test Instructions
Test Instructions
Figure 6: PandaI. pairwise response comparison results of LLaMa2-7B/133 fine-tuned with MLE and EMO on 805 test instructions from AlpacaEval.
21


Reviews:
[{"review_id": "IUohXjvzQZ", "reviewer": "Reviewer_2RWM", "summary": "This paper proposes to train \"decoder-only\" language model with a different loss function EMO  derived from the EMD (Earth Moving Distance).  The motivation is threefold: Tradeoff between Recall and Precision, Negative Diversity, Train-Test Consistency. These three points are clearly defined and compared for the standard loss (maximum loglikelihood, MLE) and EMO. The authors propose to define the inner cost of this distance with a semantic similarity (cosine between word vectors obtained from a pretrained LM). They also use a more tractable upperbound on the EMD.  The experimental setup proposes different kind of evaluation to assess the impact of this new training strategy.", "questions": "The decoding process is unique and since your purpose is to improve diversity, it could be nice to have a discussion on decoding, ie generation.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is overall well written and describes an interesting idea. The experimental setup is well described and the results look reproducible.", "weaknesses": "A \"related work\" section is missing, and it could be nice to better discuss the introduction of EMD (and optimal transport in general) in NLP and this kind of task. \n\nThe experimental setup focuses on the improvement of MAUVE. This is a quite recent metric that makes a tradeoff between precision and recall. While it is interesting to use that metric, it could be nice also to provide also perplexity. I know MLE optimizes the perplexity so it is not fair for EMO, but it can provides a meaningful comparison point (I mean in table 1).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KTCGbptnN5", "reviewer": "Reviewer_TjsD", "summary": "The authors propose to train the language model using an upper bound of Earth Mover Distance. The cost function of EMO is established by the similarity of embeddings from pretrained language model. The authors provide theoretical analysis and argue that EMO is better at handling synonyms compared with MLE. Experiments on various tasks demonstrate EMO's superiority.", "questions": "What are the results of EMO-trained models and MLE-trained models when employing beam search instead of sampling?\nI am curious as sampling reflects the entire distribution, whereas beam search captures the distribution's mode.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors propose to apply the Earth Mover Distance to train the language model and establish an upper bound for practical backward propagation training.\n2. The experiments demonstrate promising performance of EMO on various tasks.", "weaknesses": "1. The authors argue that MLE exhibits a recall-prioritization behavior, which serves as the primary motivation for introducing their proposed approach, EMO. The claim appears to be confusing, as MLE is equivalent to minimizing the forward KL-divergence, i.e., $KL(p||q_{\\theta})$.\nIf the model $q_{\\theta}$ has sufficient capacity, the optimal $q_{\\theta}$ converges to $p$. Otherwise, $q_{\\theta}$ tends to exhibit a mean-seeking behavior. Therefore I have doubts about whether \"recall-prioritization\" is proper.\n\n2. As pointed out in Section 3.3, EMO exhibits a property of harmonizing recall and precision. A straightforward inference is that EMO is better at handling synonyms compared to MLE, potentially granting EMO-trained models the capability to generate more diverse texts than models trained with MLE. The authors did not conduct such experiments.\n\n3. I guess the distribution of model trained by EMO is very different from that by MLE. However, there's no experiments and analyses regarding that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ymzpfZWBLq", "reviewer": "Reviewer_JZqx", "summary": "This paper introduces a novel loss function, EMO. The main idea behind emo is to soften the loss function (vanilla cross entropy) based on the cosine similarity of pretrained lm head embeddings. EMO demonstrates consistently better than MLE across models, tasks, scales, and metrics.", "questions": "pls see weakness and the following questions:\n\n3) why EMO can make ppl better? ppl is directly related to MLE, in other words, there does not exist a training-test mismatch problem. Is that because the evaluation is conducted over the sampled sentences instead of the test set. If so, please also report the ppl in another table.\n\n4) please also compare with label smoothing, which is also a very useful loss function in the era of pre-llm.\n\n5) what does the accuracy mean in sec 4.2.3?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. This paper is well-organized and well-written, so it is generally easy to follow.\n2. The intuition of the idea makes sense, and it is good that the complex earth mover distance loss can be reformulated into the cosine similarity dot probability. \n3. The author(s) conducted extensive and detailed experiments, encompassing different model sizes, data scales, and evaluation metrics (which is very important). The thorough and detailed experiments show the advantages of EMO, adding to the soundness of the paper.", "weaknesses": "I have two concerns here.\n\n1) Originality of the method.\nIn my view, the final loss function is very similar to the d2gpo loss, the authors did cite the d2gpo paper, but they ignored the methodology comparison and they should add d2gpo as a baseline.\n\n2) Why Cosine Similarity?\nUsing cosine similarity is a choice, but may not be the best choice. The cosine similarity relies on the pre-trained llm head embedding, which makes it not unbiased. And through the 3.3 section, the gradient of the proposed EMO is very similar to the REINFORCE with cosine similarity as the reward, so maybe RLHF/RLAIF will be a better reward model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8wUoPIKgar", "reviewer": "Reviewer_XE26", "summary": "This paper presents a training objective called Earth Mover Distance Optimization (EMO) for autoregressive language models. In contrast to maximum likelihood estimation, which solely enhances the likelihood of the ground-truth token, EMO takes into account the embedding similarity between predicted tokens and the ground-truth token, promoting predictions of tokens with high embedding similarity. Experimental results demonstrate that EMO generates superior outputs compared to MLE when employing unbiased sampling.", "questions": "None", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "This paper is easy to follow. The presentation is mostly clear.", "weaknesses": "1. **There are significant concerns regarding the evaluation methodology in this paper. Comparing MLE and EMO based on outputs generated by unbiased sampling is unfair, as the two objectives result in considerably different model distributions.** The expected MLE loss is minimized when the model distribution Q matches the data distribution P. As mentioned in the paper, the expected EMO loss is $E_{v_i ~ Q}[\\sum_{j=1}^{|V|}P(v_j)C(v_i, v_j)]$, which is minimized when the model distribution Q is a one-hot distribution that only outputs the token i that maximizes $\\sum_{j=1}^{|V|}P(v_j)C(v_i, v_j)$. Consequently, models trained using EMO will predict a much sharper distribution, leading to higher-quality but lower-diversity outputs when sampling from EMO. The evaluation methodology in this paper, i.e., comparing the sampling results of EMO and MLE, is therefore like comparing the output quality of greedy decoding and unbiased sampling, which is inherently unfair. The fair way is to compare their decoding results of greedy/beam search, necessitating further experiments to establish the effectiveness of EMO.\n\n2. Based on the above analysis, the authors' motivation appears to be flawed. Under ideal circumstances, MLE trains the model to conform to the data distribution. In contrast, EMO results in a one-hot distribution, which is recall-prioritization and negative diversity ignorance. \n\n3. The proposed EMO loss is very similar to the word embedding-based loss [1]. The only difference lies the choices of distance mertics (Cosine distance in EMO, Euclidean distance in [1]).\n\n[1] https://arxiv.org/pdf/1807.11219.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)