Language Model Decoding as Direct Metrics Optimization
Authors: Haozhe Ji
Pei Ke
Hongning Wang
Minlie Huang
Venue: ICLR 2024 poster (2024)
Keywords: language model
decoding algorithm
energy-based model
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We introduce a novel decoding framework that treats language model decoding as an optimization problem, aiming to strictly match the expected performance of generations with human texts measured by metrics of desired aspects simultaneously.

Abstract: Despite the remarkable advances in language modeling, current mainstream decoding methods still struggle to generate texts that align with human texts across different aspects. In particular, sampling-based methods produce less-repetitive texts which are often disjunctive in discourse, while search-based methods maintain topic coherence at the cost of increased repetition. Overall, these methods fall short in achieving holistic alignment across a broad range of aspects. In this work, we frame decoding from a language model as an optimization problem with the goal of strictly matching the expected performance with human texts measured by multiple metrics of desired aspects simultaneously. The resulting decoding distribution enjoys an analytical solution that scales the input language model distribution via a sequence-level energy function defined by these metrics. And most importantly, we prove that this induced distribution is guaranteed to improve the perplexity on human texts, which suggests a better approximation to the underlying distribution of human texts. To facilitate tractable sampling from this globally normalized distribution, we adopt the Sampling-Importance-Resampling technique. Experiments on various domains and model scales demonstrate the superiority of our method in metrics alignment with human texts and human evaluation over strong baselines.

Published as a conference paper at ICLR 2024
LANGUAGE MODEL DECODING AS DIRECT METRICS OPTIMIZATION
Haozhe Ji Pei Ke
Hongning Wang Minlie Huang'
The CoAI Group, DCST, BNRist, Tsinghua University, Beijing 100084, China j jihaozhe@gmail.cm com
aihuang@tsinghua. edu. cn
ABSTRACT
Despite the remarkable advances in language modeling, current mainstream de- coding methods still struggle to generate texts that align with human texts across different aspects. particular, sumpling-basee methods produce less-repetitive texts which often disjunctive in discourse, while search-based methods main- tain topic coherence at the cost of increased repetition. Overall, these methods fall short in achieving holistic alignment across broad range of aspects. In this work, we frame decoding from a language model as an optimization problem with the goal of strictly matching the expected performance with human texts mea- sured by multiple metrics of desired aspects simultaneously. The resulting decod- ing distribution enjoys an analytical solution that scales the input language model distribution via sequence-level energy function defined by these metrics. And most importantly, we prove that this induced distribution is guaranteed to improve the perplexity human texts, which suggests better approximation the un- derlying distribution of human texts. To facilitate tractable sampling from this globally normalized distribution, we adopt the Sampling-Importancee Importance-Resane technique. Experiments on various domains and model scales demonstrate the superiority of our method in metrics alignment with human texts and human eval- uation over strong baselines.
INTRODUCTION
Although pre-trained on large corpora of hu- man texts with scaled up sizes, existing auto- regressive language models (LMs) (Radford
Pa, xc pa(a)) (a(e))
Epp (w)
pa(x))
15
et al. 2019. Brown et al. 2020; Zhang et al.
probability
2022) are still struggling to produce human- like texts measured in various aspects, such as repetition, coherence, and consistency (Pillutla
0.5
10
pa (a)
et al.] 2021 Dou et al. 2022 Existing de- coding methods are mainly driven to address two main mis-specifications of an LM's distri- bution: (i) The long tail of the distribution is un- reliable (Holtzman et al. 2020), such that sam- pling from these low-probabilip regions often produces low-quality contents that are incoher- ent. (ii) The mode of the distribution is degen- erated (Welleck et al. (020),, where samples with high probabilities exhibit low diversity with repetitive patterns. As result, sampling-based decoding methods (Fan et al. 20188 Holtzman et al. 2200 Meister et al. (002)) use various trun- cation strategies to avoid sampling from the unreliable long tail of the distribution, while recent search-basee methods (Li et al. 2022] Su et al. 2022) incorporate additional contrastive objectives to avoid the collapse of degenerated repetitions. Since these two mis-specifications reside at op- posing extremes of the probability spectrum, current decoding methods inevitably concentrate on
0.0
Figure 1: The decoding distribution Pe, induced by DAEMON scales the input LM distribution po with sequence-level energy function Ep, which leads to a more accurate recovery the underly- ing data distribution Pa-
"Correpoonding Author.

---

Published as a conference paper at ICLR 2024
just one of them which addresses only a limited subset of aspects. Although heuristic designs and sophisticated
tuning allow trade-offs, these approaches usually cannot effectively align with human texts with respect to broad range of critical aspects simultaneously.
Attempts have been made to fix the mis- siiispcciiiiatiinn issue of LM distribution by directly augment- ing the standard Maximum Likelihood Estimation (MLE) with auxiliary training objectives (Welleck) eet al.
2020
Su et al. 2022 Xu et al.] 2022). However, exposure bias (Chiang & Chen [021]] Arora 2022 prevents the effectiveness of such attempts. Specifically, since during training the auto-
al.
regressive LM is conditioned on the ground-truth context, it is not guaranteed that the imposed properties in these training objectives would be preserved during decoding time, where the context is progressively generated by the LM itself. On the other hand, approaches based on Reinforce- ment Learning (RI. (Ranzato et al., 2016) Yu et al., (017) address the exposure bias issue, but often encounter challenges to maintain proximity to the distribution of human texts (characterized by a low perplexity) (Caccia et al. (020). Overall, these methods do not guarantee general enhance- ment over the standard training paradigm, owing to the potential conflicts between their designated objectives and MLE (Lin et al. 2021b). More related work discussion is provided in Appendix B
In this work, we focus on the decoding route and present novel framework, Decoding As DirEct Metrics OptimizatioN (DAEMON) that explicitly targets at aligning desired aspects with human texts. DAEMON frames decoding from a language model an optimization problem with the goal of locating the optimal decoding distribution where sampled texts can strictly match with human texts in multiple evaluation metrics simultaneously. Formally, given the input LM distribution pe learned on the human text distribution Pa. DAEMON searches for the decoding distribution q that minimizes the reverse Kullback-Leibler divergence (KL), (xL Pe), subject to the constraints of matching the expected evaluation metric scores under and pa. We choose the reverse KL to in- duce the decoding distribution q, as it forces 9 to recover the major probability masses within the support of Po (Huszar) 2015) Malinin & Gales 2019). which contains mostly high-quality samples. Moreover, besides directly enforeing alignment on chosen metrics, we also rigorously prove that the optimization problem guarantees an improvement of the solution over the input I.M in perplexity, which indicates a more general gain in aligning with human texts.
In addition to the theoretical guarantee, the decoding distribution induced by DAEMON also en- joys an analytical solution denoted as pa. It scales the locally normalized LM distribution pa with a sequence-level energy function Ep, which depicts the underlying distribution pa from vari- ous perspectives by satisfying the corresponding constraints. In Figure [] we visualize pe,, in an illustrative example where the energy captures the disjoint regions of modes in Pa, which empowers the input LM distribution po to facilitate a better approximation of pa- To enable tractable sam- pling from ue which is globally normalized over the space of all possible sequences, the
adopt Resampling (SIR) technique (DB) 1988; (mmih & Gelfand| 1192)) that first samples candidates from pv and then resamples based on the importance weight defined by the energy function. We empirically demonstrate the effectiveness of DAEMON in open-ended text gen- eration by considering wide range of critical aspects including repetition, coherence, diversity, and information content across different model scales and data domains. Experimental results show that DAEMON outperforms strong decoding baselines in both automatic evaluation of metrics alignment with human texts and human evaluation.
METHOD: DECODING AS DIRECT METRICS OPTIMIZATION
We consider conditional language generation from a pre-trained language model specified by the
distribution pa, where the model is provided with a relatively short prefix &C T lo and required to generate a continuation that results in a full text Øंх (5x ) 1=1 of total length T. In the following, the subscript of æ<T is omitted for convenience. Instead of directly sampling from
{xr] of length
pe. we look for decoding distribution induced from pe to produce human-like like texts measured by set of chosen metrics. For example, in the canonical top-k sampling (Fan ct al. 2018), the decoding distribution is obtained by truncating the conditional distribution [x [1:1-1) to keep the top-k candidates at every decoding step, so as to improve the reliability of generated content.
Ideally, perfect decoding distribution Jopt assigns arbitrary text sample E with the probability equals to pa(x), where Pa is the underlying distribution of human texts. In practice, this is infeasible since we only have samples from pa, rather than pa itself. However, given text evaluation metric
2

---

Published as a conference paper at ICLR 2024
are interested in (such repetition and coherence), formally f X - R that maps in the text space X' to a real value, an alternative criterion for measuring the closeness from 9opt to Pa can be achieved by matching the expectation of f under 4opt and pd. i.c., [ [f( -xpp [( This expectation matchiing criterion, commonly employed in prior studies (Holzzman et al.) 2020
Meister et al. [022 Su et al., (022)) as an empirical evaluation of the resemblance of generated texts against human texts. This forms the basis of our proposed optimization-base decoding framework that directly aligns the generated texts with human texts against the set of chosen text evaluation metrics.
2.1 FORMULATION OF THE OPTIMIZATION PROBLEM
At the core of our proposed decoding framework, we look for the optimal solution gopt of the following constrained optimization problem which searches for the decoding distribution q closest to the given LM distribution pa and strictly matching the expectations on the generated texts with that of human texts measured by a set of chosen evaluation metries:
4opt = arg min DKL (gllpe)
(1)
yEP
s.t. E 2g Lf = Ex (x)())), E { ... KK},
where f = [[ is a set of evaluation metries we concern, and P is the set of all probability densities in the imput space X.
The formulation of our proposed optimization problem hinges on our key insight of constructing a decoding distribution from language model to acquire samples that closely resemble human texts. The constraints, as defined to match the performance of evaluated metrics on generations with those obtained on human texts, explicitly ensure this goal in expectation. The reverse KL divergence in the optimization objective, i.e., Dxt. Dxt (gliPe), restricts the decoding distribution to deviate minimally from the LM distribution pe by encouraging mode-seeking behavior, which satisfies the quality- demanding nature of decoding. Although the forward KL is extensively employed as an optimization objective in learning data-drived probabilistic models (Radford et al. 2019). its induced distribution is shown to mismatch with the quality assessment of human (Pang & He (021) by overestimating the long tail of the target distribution (Ji et al. 2023) due its mean-seeking behavior. More discussion is provided in Appendix C We believe the learning and decoding phases posit different goals: the former is to capture all modes in the data, while the latter is to decode high-quality ones. Hence, we require the decoding distribution to only explore within the support of the given LM distribution, which is naturally realized by minimizing the reverse KL. Existing truncation-base
sampling (Fan) ee al. 2018 Welleck et al.) 2020 Meister al., 2022) can be deemed as heuristic that shares the same spirit of ours in maintaining a finite reverse KL, since the support of the truncated distribution is always the strict subset of the support of the given L.M distribution.
The formulation of our optimization problem is also known as information projection in previous literature of information geometry (Csiszár & Matús, 2000) Nielsen| 2020), which can be deemed as finding the projection of pa on the manifold of distributions that constrains Pa. In the following proposition, we show that it actually leads to a nice analytical solution. The full proof of Proposition is provided in Appendix A.I
Proposition The distribution that solves the optimization problem 7 is in the form of:
p (x)) x po(x)) exp F=u(a)) Va E S(po.p)
(2)
where Fx((x) = n. fax and S(p) = (æ p(x)) > 0} is the support of distribution p. μ TR. determined by the constraints in 7
The unnormalized form of pa (z), also known as the Energy-Basse -ased Model (EBM) (Rosenfeld et al., 2001] Hinton 2002 LeCun et al. 2006), takes advantage from both the given LM distribution
pe and the energy function Ex)()) that serves as a sequence-level assessment about the satisfaction
of constraints measured by the evaluation metrics. The contribution of individual metrics to the
overall alignment performance is characterized by the derived coefficients μ {px 1 Decoding from Eq. (2) requires determining μ and tractable sampling from the normalized density, which will be discussed in $2.3 In the next subsection, we take a step further and demonstrate that the optimal solution of the problem (1) guarantees theoretical improvement in perplexity of human texts.
3

---

Published as a conference paper at ICLR 2024
2.2 THEORETICAL IMPROVEMENT IN PERPLEXITY
Although explicitly driving the generation to align with human texts under the chosen evaluation metrics is appealing, are still confronted with the question of whether the resulting decoding dis- tribution is generally a better approximation to the underlying distribution of human texts. For most existing heuristic decoding methods, distribution-level evaluation (e.g., perplexity) infeasible because of their ad-hoc treatments on the input LM distribution. For example, distribution trunca- tion (Fan al., 2018: Welleck et al.] 2020; Meister et al.) 2022) leads to a sparse support which is smaller than the underlying distribution of human texts, while heuristic searching algorithms (Li ctt al. 2022, Su et al. 2022) such as beam search do not have parametric decoding distribution. Martins et al. (2020) proposed a variant of the standard perplexity, e-perplexity by smoothing sparse distribution, which still can not faithfully reflect the true perplexity of the truncated distribu- tion.
For the decoding distribution derived from the proposed optimization problem, we show that not only is the perplexity feasible to compute, but it also improves the perplexity of human texts against the original LM distribution. The full proof is provided Appendix AA.2
Proposition 2.
The optimal solution Gopt of the optimization problem satisfies:
1. S(yop)) 2 S(pa), where S(p) {or p(x)) 0}.
2. 11 (pá: (opt) H (pa: pa) DKL
where (p. E. p(x) log)(z).
Proof sketch. The proof starts with the convexity of the set C of distributions that satisfy the con- straints Eq. (T). We then consider Pa. =1-a) + apa Cor [0, 1]. The key insight is the following observation:
DxL (Pall a=
Hpa: po) H Hpa. 9opt.) DKL. (9ooa9pplllp)).
(3)
da
0D (Pa po)/Oa can also be written as the limit of (Pallp)) DKL (aopt llPo)]//a which is non-negative when - 0+ due to the optimality of qope. Therefore, for Eq. (3) to be non-negative. we must have (xr) + 0 for any x E S(pa) (otherwise it converges to - -oc), which proves the first
claim. Next, given S((opt) 2 S(pa), there exists some a' < 0 such that Pa is probability density
function, which by definition also belongs to C. Therefore, (Pa (llp))
(9opt llpp))/aa is
non-positive when a' + o leading to 0D (Pallpoo)))) claim.
= O. which proves the second
LI
The first outcome of Proposition establishes the feasibility of computing perplexity under pa, when evaluated using the underlying human text distribution Pa. And the second result re- veals the perplexity improvement over po: (Paa < 2Pa..0)) due to the non-negativity of DxL (gope [pp).. Note that the perplexity of g is defined as 2)
Intuitively, more powerful constraints in the optimization problem that better measure the alignment with human texts cause a larger deviation from the input LM distribution, which in turn leads to a better approximation of underlying human text distribution, and thus lower perplexity.
2.3 DECODING FROM THE OPTIMAL SOLUTION
In this section, we describe the method to decode from the sampling distribution derived from the optimization problem (I). First, we describe method to estimate the coefficients f by satisfying the constraints with a conditional proposal distribution. Then we introduce a tractable sampling method to obtain samples from the decoding distribution defined by the EBM.
2.3.1 COEFFICIENTS ESTIMATION
The only degrees of freedom in the analytical solution of the optimal decoding distribution ,e,, are the coefficients estimated by first calculating F Ee-pe... (x)) and then approximating the target expectation F= = Ex [x)) to satisfy the constraints with iterative gradient updates. Note that this procedure done on a small development set once for all before the inference stage.
K {px} ) k= 1 in the energy function E, (2). whose optimal values Hopt can be
4

---

Published as a conference paper at ICLR 2024
Algorithm 1 Mopt estimation with WIS
First, F can be estimated by Weighted Impor- tance Sampling (WIS) (Geweke 1989 Hester- berg 1995) which first obtains N i.id. tra-
Input: po, F. learning rate
Output: Mops
jectories [ ~ pa. and then computes
1: Initialize ja randomly
the weighted sum of f(x') with importance
2: Sample trajectories
( Pa
weight proportional to exp(-En((²)) normal-
3: repeat
ized over all trajectories. As the asymptotic bias and variance of F estimated by WIS are both proportional to N-1 (Hesterberg) 1995). the target expectation can be approximated with required estimation error by drawing enough samples from the proposal. Detailed derivation of WIS is provided in Appendix A.4.I
4:
F Disi exp( Eyp (
expl (er(-E,(''')1
5: μ p- V
k F
6: until convergence
7: Mopt f /
Next, given F as parametric function of the variable μ., propose approximate the target expectation F by minimizing the Root Mean Squared Relative Error (Shcherbakov et al. 2013).
V / where the estimation eπгоr of each f& is normalized to the same scale. Then the optimal coefficient Mopt is obtained by iteratively updating μ until convergence, i.e.. reaching a desired error level. The algorithm coefficients estimation is shown in Algorithm [] We also analyze the convergence of şu in Appendix Gand find insensitive initialization. Runtime analysis of Algorithm[[]
iis is provided in Appendix F which demonstrates the advantage over the typical hyper- parameter search procedure for most other decoding methods (Meister et al. 2022
Li al. 2022).
2.3.2 CONDITIONAL SAMPLING FROM EBM
Sampling from the decoding distribution defined by EBM in Eq. (2) is non-trivial, given that it is globally normalized over the whole sequence space. We first present the conditional probability of sampling a continuation Tyt, from pa given prefix gg::::
Po ( (0 (t))))) po(a > (e exp | (x x ( 12(e (()
(4)
(x where ((xgit)= = 1, -pp(- (( |exp(=Ep St Tzyt,)) ))] is the marginalization over future to-
kens sampled from the conditional proposal given the prefix. The detailed derivation is provided in Appendix [ As direct sampling from this auto-regressive factorization is computationally prohibitive (Deng al., 2020), we instead turn particle-basee approximation of po.,a using the (SIR) technique (DB 1988: Smith & Gelfand 1992).
Specifically, we first leverage the given LM Po as proposal to generate a set of M plausible continuation candidates LM
Algorithm 2 Conditional Sampling with SIR
Input: po. Ep. prefix Eg M, Output: continuation Uyto I: for i 4 1-11 to M do
{ [0',,, Ji=1 given the prefix Øgo in par- allel. Then the final generation result is resampled from the distribution defined by the importance weight which is pro- portional exp(-F,, ( $:,,,)) nor- malized over all candidates { j We present the SIR approximation procedure of the conditional probability A ( (:(x))))) in Appendix [A.22] In the limit of M
> In parallel
2: Sample 20 p% (-
3: Compute w: <- exp( E (I $5to))
4: end for
5: Sample ~ Categorical
Σ wi
Set Mi <- 25
oo, the empirical distribution pod (-100g1)) induced by SIR recovers the exact conditional distribu-
tion Pe, (,)) ) for arbitrary T>te Skare et al. (2003) proved that the wisnt-wise relative error of (see Theorem 2.1 in the original paper). In practice where M is finite, we propose to sample from the the temperature modulated proposal pT with lower temperature to increase the chance of obtaining high-quality candidates within a real- istic computational budget. The algorithm of conditional sampling is shown in Algorithm 2 In fact, various existing sampling methods ture sampling as preserves the feasibility to compute perplexity (see Appendix (A.3). We provide complexity and runtime analysis of Algorithm 2 and baseline decoding methods in Appendix F
the empirical distribution induced by SIR is O(M
be used for candidate sampling, choose use tempera-
5

---

Published as a conference paper at ICLR 2024
3 EXPERIMENT
3.1 DATASETS
We evaluate our method on the Wikipedia and News domain for open-ended text generation. For the Wikipedia domain, the data comes from documents in the Wikitext-103 corpus (Merity al.] 2017). For the News domain, the data comes from news articles in Wikinews We follow the data pre-processing procedure suggested by Li et al.] (2022), and randomly select 512 samples as the development set for hyper-parameter tuning for all decoding methods. The data statistics of each domain and detailed data pre- pre-processing steps are provided in Appendix
3.2 EVALUATION METRIC SETTINGS
In this section, we introduce the set of evaluation metrics we consider in aligning with human texts, which correspond to f in Eq. ( These metrics cover a wide range of aspects including repetition, coherence, diversity, and information content.
Repetition. We evaluate repetition at both sequence level and token level. The sequence- metric measures the portion duplicate 7--grams the generated texts (Welleck et al.] 2020):
SEQ-REP-N = 100 x (1- lunique n. nr.grmms(a))) (+) where é is the generated text (SR-N in short). The token-
[otal x-grams(
level metric measures the average frequency of each generated token reoccuring in the previous /
tokens (Fu al. 2021 Ji & Huang 2021): TOK-REP-L 100 A
E [ -
(TR-L in short). We adopt SR-N with m 3,43 and TR-L with = (8, 16,32), respectively.
Coherence. We evaluate coherence following [Su et al. (002 by calculating the cosine simi- larity between the sentence embedding of the prefix Est. and the generated continuation i> COll 100 cos(emb(#=sto) emb( where emb(-) is parametrized by the pre-trained sen-
tence embedding model SimCSE (Gao et al. 2021 based on RoBERTa (Liu et al. 2019).
Diversity. We evaluate diversity following Li al. (2022) by aggregating the n-gram repetition rate of = (2. 3, 4): DIV 100 mT=2(1 SEQ-REP-N). DIV reflects the overall lexical diversity of the text at different levels of granularity.
Information Content We evaluate the average amount of information contained per word given the preceding contexts, by calculating the exponential of the entropy rate on the generated text à using language model: E cxp(- | Σl log ( (Shannon 1951 Braverman al. 2020). A low e suggests that the information the text redundant, while high easT indicates high surprisal in the text according to the language model. In experiment, general-domain language model GPT-2 XL to calculate the log probability the generated texts.
Note that these metrics are also used in the automatic evaluation part of experiment ($3.5) to measure how well the generated texts align with human texts in different aspects. Thus, the criterion for these evaluation metrics is the closeness between the metric scores of generated texts and those of human references.
In addition, we also report MAUVE score (Pillutla et al. 2021 (MAU in short) which measures the distributional similarity between the set of generated texts and that of references by calculating the area under the divergence frontier of the two empirical distributions. As this metric cannot provide an evaluation score that corresponds to certain aspect for each text sample, we only adopt it to assess the final performance of different decoding methods. We use GPT-2 XL extract features from texts which was restricted to maximum length of 256.
3.3 BASELINES AND IMPLEMENTATION DETAILS
We thoroughly compare DAEMON with various sampling-based and search-based methods in par- ticular. We consider three canonical sampling-based methods: Top-k sampling (Fan et al. 2018), Nucleus sampling (Holtzman al. 2020) and Typical decoding (Meister et al. 2022). For search-
http:///www.iiiiieee.ooo
htttp://een.cr http::///e.viiieiia.... wwtp:///////oo/ooii..ooooo

---

Published as a conference paper at ICLR 2024
Wikipedia
News
Method
SR-4
TR-32
COR
DIV
SR-4
TR-32
COH
DIV
Reference
0.48
21.3
62.3
92.5
23.2
0.29
18.7
66.6
94.1
13.8
Greedy
60.9
65.5
60.2
8.03
2.29
59.7
53.2
58.2
63.8
13.2
2.19
65.2
Top-k
2.11
23.4
60.9
87.8
10.1
77.8
0.95
20.3
64.7
91.7
8.17
96.3
GPT-2X2
Nucleus
1.19
20.0
57.3
92.4
17.3
78.3
0.80
18.7
60.8
93.5
11.0
95.3
Typical
0.81
17.4
54.9
94.5
30.1
78.7
0.42
16.9
57.2
95.3
18.2
95.0
1.31
28.2
68.7
85.9
7.55
77.8
0.63
23.2
71.2
90.5
6.55
95.1
1.78
23.0
56.9
90.6
5.25
83.3
0.77
19.2
63.6
94.1
4.18
95.7
DAFMON
0.42
62.5
92.2
94.5
13.7
Greedy
54.8
60.4
62.0
0.12
2.78
64.8
45,2
$1.0
63.6
0,22
2.72
70.7
Top-k
2.44
24.1
61.3
86.6
13.9
77.5
1.53
19.9
65.7
90.5
10.7
95.7
OPT-6,7B
Nucleus
2.33
21.9
59.1
88.6
18.9
80.1
1.37
19.2
63.3
91.5
12.2
95.3
Typical
1.06
19.6
57.0
92.9
31.9
77.7
0.95
17.7
59.4
93.7
19.4
95.2
CD
2.90
26.5
68.6
82.3
71.7
87.5
9.20
L.3
21.7
57.7
91.8
8.72
83.3
1.18
18.3
62.9
93.2
6.69
94.0
DARMON
0.38
21.6
62.3
92.6
22.7
90.7
0.25
18.8
64.8
94.9
13.6
97.2
Table 1: Main results of automatic evaluation on the Wikipedia and News domain using GPT-2 XL and OPT-6.7B. For all metrics, the best scores are the closest to the human scores except for MAU, which is better when higher. The best score in boldface and the second best is underlined.
based methods, besides vanilla Greedy decoding, we also consider two recent methods that maxi- mize the contrastive objectives: Contrastive Decoding (CD) (Li et al. 2022) and Contrastive Search (CS) (Su et al., (022)).
To demonstrate the effectiveness of our method across different language model families and scales, we consider GPT-2 XL (1.5B) (Radford et al. 2019) and OPT-6.7B (Zhang et al. (022) as the base models for all decoding methods. For baselines, we follow the hyper-parameter settings the original papers which are shown to work well in general. For DAEMON in the main results, we use the nine metries (described in $3.2) the constraints. During sampling, we set the size of candidate set from the proposal model M 25 as it balances efficiency and performance. We set T 0.97 for the Wikipedia domain and т = 0.99 for the News domain. We leave more implementation details of the baselines and DAEMON in Appendix J.2
3.4 HUMAN EVALUATION
We further conduct human evaluation assess the quality of generated texts. We consider three widely used criteria for open-ended generation: Fluency, Coherence, and Informativeness (van der Lee al., 2019). Specifically. Fluency is characterized by the grammatical correctness and natural- ness of the text without repetition; Coherence is characterized by topic maintenance with the input prefix and well-structured discourse; Informativeness characterized by the adequacy of elaborat- ing engaging details in a coherent plot. We randomly sample 100 prefixes and conduct pair- wise comparisons between DAEMON and baselines. Three annotators on Amazon Mechanical Turk are hired to choose the better continuation (i.e., win, lose, tie) from the ones generated by our method and the baselines in terms of the three criteria above. More detailed settings of human evaluation provided in Appendix D
3.5 MAIN RESULTS
Automatic Evaluation. We first present the main results obtained via automatic evaluation in Table
I where we benchmark our method against strong basclines across different domains and model scales under the evaluation metries described in $3.2 Due to the space limit, we only present rep- resentative metrics and leave the full results in Appendix J.3 DAEMON outperforms all baselines in aligning with human texts on four aspects including repetition, coherence, diversity, and information content, and it also achieves the highest MAUVE score across different domains (Wikipedia/ / News) and model scales (1.5B / 6.7B). The consistent performance improvement demonstrates the effec- tiveness and generalizability of DAEMON. Notably, DAEMON achieves the lowest sequence- repetition (indicated by SR-4) across all settings and maintains human-level coherence and high MAUVE score at the meantime, compared with baselines that have similar repetition level, e.g.,

---

Published as a conference paper at ICLR 2024
Model
ori
Wikipedia imp
ori
News
imp
Ours vs.
Win
Fluency
Lose
Win
Coberence
Lowe
Informativenes
Win
Lose
GPT-2 XL
23.1
22.0
13.9
13.1
CD CS
0.54
0.35
0.48
0.36
0.48
0.27
OPT-6.7B
16.4
16.2
10.8
10.2
0.34
0.47'
0.41
0.33
Nucleux
0.54
0.33
0.66
0.15
0.45"
0.30
Typical
0.53"
0.30
0.62"
0.19
0.44
0.23
Table 2: Perplexity evaluation results. "ori" is the original perplexity of the LM distribu- tion. "imp" is the improved perplexity of the optimal decoding distribution.
Table 3: Human evaluation results that compare our method with baselines Wikipedia domain. indicates a significance with p-value < 0.005.
the Typical decoding method. The results indicate that DAEMON is the only method that effectively aligns multiple aspects with human texts without hard trade-off among those aspects.
For sampling-based methods, Nucleus sampling generally preserves more diverse candidates than Top-k sampling and achieves better diversity at human-level at the cost of low coherence. Typical decoding over-emphene werremmhasizes the long tail of the distribution, which produces texts with highly diverse lexicality (highest diversity) but severe topic shift (lowest coherence). On the other hand, search- based methods generally have substantially lower information score than sampling- based methods, which indicate the redundancy of information in the generated texts. Specifically, CD achieves the highest coherence score at the expense of having the worst diversity except for greedy decoding. By analyzing its generated texts, we found that CD tends to repeat tokens from the previous context (indicated by high TR-32), which could be bias captured by SimCSE to compute representations with high similarity. Additionally, we found that the MAUVE score of CS is generally higher than CD when the evaluation length is set 256, which contradicts the previous result (Li et al.) (022) where the length is truncated to 128. Our result is consistent with our observation that CD generates semantically more repetitive texts its longer generations. To demonstrate the universability of our method, we additionally conduct an experiment on a text summarization dataset in Appendix !|
Perplexity Evaluation. We caleulate the perplexity of DAEMON's decoding distribution, which can be directly computed using a proposal model with temperature modulation. The derivation of perplexity calculation is presented in Appendix A.3 From the results in Table Z] we observe that the perplexity improvement over the original LM distribution is consistent across model sizes and data domains. Notably, the perplexity improvement is more obvious for the model with a smaller size (GPT-2 XL., which manifests the advantage of our approach in enhancing the capacity of the autoregressive LM modeling language with a constrained computational budget.
Human Evaluation. We conduct pairwise human evaluation and selectively compare our method against strong baselines including Contrastive Decoding (CD), Contrastive Search (CS), Nucleus sampling, and Typical decoding. As shown in Table 3 DAEMON is more preferred than all four baselines fluency, coherence, and informativeness respectively on the domain of Wikipedia ac- cording to the human judgment. Specifically, DAEMON generates significantly more coherent texts compared to all the basclines. We provide a list of qualitative cases produced by different decoding methods in Appendix J.4 to help readers comprehend the difference in their generation quality.
3.6 ABLATION STUDY
Ablating Metries in Constraints. We ablate the metrics in the constraints of DAEMON to inves- tigate their individual contributions to the overall alignment performance. In Table 4 the results of ablating different metrics while keeping other settings verbatim. The ablation ex- periments are obtained using GPT-2 XL on the Wikipedia domain. We first observe that for most metrics, removing them lead to drastic deterioration in the corresponding metric scores, e.g., SR-4 (0.42 - 3.57), COH (62.5 - 57.6), eNT (22.8- + 19.9). We also observe clear inter-dependence between certain metrics pairs, removing one leads notable performance drop in another, c.g., SR-N and DIV, ENT and TR-L, which reflects the intrinsic correlation among the evaluated aspects.
we present
Number of Candidates for Resampling. We then study the impact of the number of candidates for resampling (M described in 2.3.2). In Figure 3 we present the results on five metrics and the relative decoding latency with batch size of I. We set the temperature of the proposal distribution to 1.0 to isolate the impact of M. From the results, we observe that the aligmment on all metrics gen- erally improves with larger M, which indicates better SIR approximation to the optimal decoding
8

---

Published as a conference paper at ICLR 2024
Metrics
SR-4
TR-32
CON
100
1.05
Reference
62.3
94
0.9
0:97
DAEMON
0.42
22.5
62.5
92.2
22.8
w/o SR-N
3,57
22.6
63.0
86.8
22.1
88
0.8
w/o TR-L.
0.19
20.2
62.0
93.9
25.1
wlo
0.37
22.3
57.6
92.6
23.8
Temperature
wlo DTV
0.30
22.1
62.4
93.0
23.3
Nucleus
wlo EXT
76
0.7
DAESIOO
Table 4: Ablation results of different met- rics where rows in gray are results with corre- sponding metrics removed, "SR-N" and "TR- L" denote the set of metrics with = 2, 3, 4 and = 8, 16, 32 respectively.
70
55
56
57
59
60
61
62
63
64
65
Figure 2: COH versus DIV when tuning the temperature of the proposal model of DAE- MON and hyper-parameter of other baselines.
0.6
94
Graniy
SR-1
loteney
0.3
10 50 70 100
25 111
75
100
155 GO 75 100
30 30
54
100
10
25
DU
T3
50) 100
25
75
100
M
34
24
Figure 3: Ablation results of varying the number of candidates for resampling (M). Results on the five metrics are compared with the reference and the latency is relative to Greedy decoding.
distribution. Specifically, the convergence rate of different metric varies, e.g., DIV and e converge slower than TR-32 and COH with the increase M. Finally, increasing M inevitably incurs higher decoding latency, and thus we choose M 25 with slightly lower temperature in the main result to maintain both efficiency and performance. We test the robustness of our method by investigating the performance of different metrics when we optimize a single metric in Appendix H
Temperature When Sampling from the Proposal Model. As suggested by (Caccia et al. 2020 Zhang al. 2020), quality and diversity two important aspects which can be traded off by sweep- ing hyper-parameters (e.g., temperature) to alter the sharpness of the distribution. For DAEMON, we tune the temperature of the proposal model (r described in 22.3.2) with other settings unchanged and plot the curve in the dimensions of coherence and diversity in Figure 2 We also plot the re- sult of tuning the syper-parameters of different baseline methods. We first observe that DAEMON dominates the compared baselines in terms of coherence for all interested diversity level. Second, DAEMON is able to achieve human-level performance on these two aspects by slightly tuning the temperature lower, which demonstrates the effectiveness and practicality of our approach.
CONCLUSION AND FUTURE WORK
In this study, we introduce Decoding as Direct Metrics Optimization (DAEMON), a decoding frame- work that explicitly aligns the generations with human texts under various aspects, e.g., coherence, repetition, and etc. The induced sampling distribution harnesses candidates generated by an auto- regressive LM, which are re-weighted according to a sequence-level energy function. We demon- strate both theoretical and empirical benefits of DAEMON, which outperforms strong decoding base- lines in both human evaluation and automatic evaluation in terms of metrics alignment human texts, perplexity improvement over the original LM, and superior quality-diversity trade-off.
As for the future work of DAEMON, we consider exploring directions to generalize the framework, e.g., extending the equality constraints to more general constraint types, such as inequalities and structural equations. It is also necessary to consider more aspects along with evaluation metrics beyond text quality, e.g., human value. This can therefore complement with other training-time alignment methods, such as RLHF And finally more efficient method to sample from the distribution defined by the EBM is also important to ensure the practicality of DAEMON. Overall, firmly believe this work paves the way for advanced methods that guide the language model towards desired behavior by incorporating constraints that capture intended regularities.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the National Science Foundation for Distinguished Young Scholars (with No. 62125604), the NSFC projects (with No. 62306160 and No. 61936010), the China Na- tional Postdoctoral Program for Innovative Talents (No. BX20230194), and the China Postdoctoral Science Foundation (No. 2023M731952).
REFERENCES
Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Chi Kit Cheung. Why exposure bias matters: An imitation learning perspective of error accumulation in language generation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 700-710. As- sociation for Computational Linguistics, 2022. doi:
wttp://///2222.2niniiii ac1.58. URL
https://doi.org/ 18653/v1/2022 fiwniiinnnnacii.
Sumanta Bhattacharyya, Amirmohammad Rooshenas, Subhajit Naskar, Simeng Sun, Mohit lyyer, and Andrew McCallum. Energy-based reranking: Improving neural machine translation using energy-based models. In Chengging Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pa- pers), pp. 1
Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/202) .acl-long..49. (tLL:ttts://accattioto org 2021 1121..cc2 acl acl-long. .399
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
Mark Braverman, Xinyi Chen, Sham M. Kakade, Karthik Narasimhan, Cyril Zhang, and Yi Zhang. Calibration, entropy rates, and memory in language models. In Proceedings of the 37th Inter- national Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, vol- ume 119 of Proceedings of Machine Learning Research, pp. 1089-1099. PMLR, 2020. URL http://proceedinn
mlr www.preesssiiii http:////1////aaaraa22..... html
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.). Advances in Neural Information Processing Systems 33: An- nual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL. htttps:/pproceedingg wewpeeeddiiinnnnnnnnsneeeeeeeeee 1457
2000/ 20020/hhsh//
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Char- lin. Language gans falling short. In 8th International Conference on Learning Representa- tions, ICLR 2020. Addis Ababa, Ethiopia, April 26-30. 2020. OpenReview.net, 2020. URL https://openreviier wete//oorumiiddlgaaaaaa
Alan Chan, Hugo Silva, Sungsu Lim, Tadashi Kozuno, A Rupam Mahmood, and Martha White. Greedification operators for policy optimization: Investigating forward and reverse kl diver- gences. The Journal of Machine Learning Research, 23(1):11474-1155) 2022.
Ting-Rui Chiang and Yun-Nung Chen. Relating neural text degeneration to exposure bias. In Jasmijn Bastings, Yonatan Belinkov, Emmanuel Dupoux, Mario Giulianelli, Dieuwke Hupkes, Yuval Pinter, and Hassan Sajjad (eds.), Proceedings of the Fourth BlackboxNLP Workshop on An- alyzing and Interpreting Neural Networks for NLP, BlackboxNLPGG EMNLP 2021, Punta Cana, Dominican Republic. November 11. 2021, pp. tics, 2021. doi: 10.18653/v1/2021.. www.kboxn..bbccbbbbboo... 1.16. URL https:::p//. stttps://ooi //oi org/10. 18653/665/ 2021. blw.bbobboooxmpp....6
Association for Computational Linguis- URL
10

---

Published as a conference paper at ICLR 2024
Imre Csiszár and Frantisek Matús. Information projections revisited. 2000 IEEE International Symposium on Information Theory (Cat. No.00CH37060), pp. 490-, 2000.
RUBIN DB. Using the sir algorithm to simulate posterior distributions. In Bayesian statistics 3. Proceedings of the third Valencia international meeting, 1-5 June 1987. pp. 395 402. Clarendon Press, 1988.
Yuntian Deng, Anton Bakhtin, Myle Oit, Arthur Szlam, and Marc' Aurelio Ranzato. Residual energy-based models for text generation. In 8th International Conference on Learning Repre- sentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httpss://ppeneevier hett/oorumi.d-11155515.
Yao Dou, Maxwell Forbes, Rik indistinguishable from human text? scarecrow: A framework for scrutinizing machine text. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 7250 7274, Association for Computational Linguis- tics, 2022, URL htttps::/accaatthoottttooooo
Noah A. Smith, and Yejin Choi. Is GPT-3 text
org/ 2022 acl acl acl.on...l...n..........
Angela Fan, Mike Lewis, and Yann N. Dauphin. Hierarchical neural story generation. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume I: Long Papers, pp. 889-898, Association for Computational Linguistics, 2018. doi: 10.18653/v1/
P18-1082.
Patrick Fernandes, António Farinhas, Ricardo Rei, José GC de Souza, Perez Ogayo, Graham Neu- big, and André FT Martins. Quality-aware decoding for neural machine translation. arXiv preprint arXiv:2205.00998, 2022.
Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. High quality rather than high model probability: Minimum Bayes risk decoding with neural metrics. Transactions of the Association for Computational Linguistics. 10:811 10:811-825, 825, 2022. doi: 10.1162/tacl 00491. URL https:
//aclanthology. , wrg/2022.........o 2022. tacl-1.41
Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi. A theoretical analysis of the repetition problem in text generation. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Thire Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event February 2-9, 2021, pp. 12848-12856 12856. AAAI Press, 2021. URL https: //ojs. aaai.
org/index, php/ AAAI/ AAAT//arricee/vve//111200
Kuzman Ganchev, João Graça, Jennifer Gillenwater, and Ben Taskar. Posterior regularization for structured latent variable models. J. Mach. Learn. Res., 11:2001-2049, 2010. doi: 10.5555/ 1756006.1859918 https:/ts://dl acm. www.ooggg/o/ooooooooo 555..0555551171665551110 1859918
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 1890- Association for Computational Linguistics, 2021. doi: 10. 10.18653/v1/0011. emnlp-main.555 552.. http:t//////oii http://66...rrr//0..6/////////00 emnlp-main. 552
Stuart Geman and Donald Geman Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEFF Transactions on Pattern Analysis and Machine Intelligence, PAMI-6:
721-741, 1984. URL httph: //pii. www.nticscoolar.r....e.occcccac. wwwp/ooggoaaeeeeeee ::: 5837272
John Geweke. Bayesian inference in econometric models using monte carlo integration. Economet- rica: Journal of the Econometric Society, pp. 1317-1339, 1989.
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Sheood- AAAI Conference on Artificial Intelligence, (A4AI-18),
11

---

Published as a conference paper at ICLR 2024
the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18). New Orleans. Louisiana, USA, February 2-7, 2018, pp. 5141-5148. AAAI Press, 2018. URL https: /wwp://wwww..m aaai. org/ocg./
index. AAAT/AAAAAAt/ppppp//el/////////
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications natural image statistics. J. Mach. Learn. Res., 13:307-361, 2012. doi: 10.5555/2503308.2 188396. URL htthhttp://ddl acm. org/ dti//0.1110.0///// 000.0005555525530000 218883
Tim Hesterberg. Weighted average importance sampling and defensive mixture distributions. Tech- nometrics, 37(2):185-194, 1995.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002.
Ari Holtzman, Jan Buys. Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. UtLattiss//opperrriov.o net/
Forum/id=ryg@@yrFvt
Ferene Huszar. How (not) to train your generative model: Scheduled sampling, likelihood, adver- sary? CoRR, abs/1511.05101, 2015. URL
05101
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind W. Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. CoRR, abs/190) 1907.00456 2019. URL httt:://arxt prg/abs/190/) 00456.
Haozhe Ji and Minlie Huang. Discodvt: Generating long text with discourse-aware discrete vari- ational transformer. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih (eds.). Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 4208 -2244. Association for Computational Linguistics, 2021. doi: 00.18653/v1/022. emnlp-main.344 347.
1863301112653/2/222222 emnlp-main 347
Haozhe Ji, Pei Ke, Zhipeng Hu, Rongsheng Zhang, and Minlie Huang. Tailoring language genera- tion models under total variation distance. In The Eleventh International Conference on Learn- ing Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL http:////ppeereeeveeeeeeetddddde
Nitish Shirish Keskar, Bryan McCann, Lav Varshney, Caiming Xiong, and Richard Socher. CTRL: A conditional transformer language model for controllable generation. CoRR,
abs/1909.05855 055588 2019. http:///://// wwwrexiv..or/aaee. wrw//bs/////oooooooooog.oot..s.r http//999..055000
Muhammad Khalifa, Hady Elsahar, and Mare Dymetman. A distributional approach to controlled text generation. In 9th International Conference Learning Representations, ICLR 2021, Vir- tual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL /tllhtttp///ooeeeeviee net/ forum?id www.:///ddwww599900A0900000
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang. Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. CoRR, abs/2210. 15097, 2022. doi: 00.48550/arxiv22200.5559 URL phttps:///o// org/10. (8550/arXiv. 2210 15097]
Chu-Cheng Lin, Aaron Jaech, Xin Li, Matt Gormley, and Jason Eisner. Limitations of autoregressive models and their alternatives. In NAACI, 2021a.
12

---

Published as a conference paper at ICLR 2024
Xiang Lin, Simeng Han, and Shafiq R. Joty. Straight to the gradient: Learning to use novel tokens for neural text generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference
Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 6653..653. PMLR, 2021b. URI. html
http://procceedings mlr. wal..peee (ttess/v139/iimmm
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692 2019. http::////axxi wttgg//b///./.11///0
Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for condi- tional models: Consistency and statistical efficiency. In Ellen Riloff, David Chiang, Julia Hock- enmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference ods Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 3698-3707, Association for Computational Linguisties, 2018. doi: 10. 18653/v1/018-14000 URL https://doi /ttpps://doi.org/1.... otg/10. 18653/v1//11
Empirical Meth-
Andrey Malinin and Mark Gales. Reverse kl-divergence training of prior networks: Improved un- certainty and adversarial robustness. Advances in Neural Information Processing Systems, 32, 2019.
Pedro Henrique Martins, Zita Marinho, and André F. T. Martins. Sparse text generation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 4252-4273. Association for Computational Linguistics, 2020. doi: 10. 1..18653/v1/2222 emnlp-main.344. https:////doo orgg 10. 10653/11/20222 emm.p00mmmmmmmmmaammmm..mm. 348
Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. Typical decoding for natural lan- guage generation. CoRR, abs/2202.00666,, 2022. URL/httts://aaxi wrg. wwwggabs/2202.00000 00666
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: //openreview httpiiiwwwwwwteeeeeeeeeee net//forunidddBy)77aoaa
Nicholas C. Metropolis, Arianna W. Rosenbluth, Marshall N. Rosenbluth, and A. H. Teller. Equation of state calculations by fast computing machines. Journal of Chemical Physics, 21:1087-1092, 1953. https:tts:///pp http:p:://pp.ssmmmniiccccoooooo semantccscoolar .ww.a...ggooooeeeee www.oogg/roorr.... weng/CCrppusI:::: 1046577
Frank Nielsen. An elementary introduction to information geometry. Entropy, 22(10):1100, 2020.
doi: 10.3390/e22101100. UtL/attss:///oi.oo
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Informa- tion Processing Systems, 2022. URL https::///peneeeee
Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations. In 9th Inter- national Conference on Learning Representatiinss ICLR 2021, Virtual Evenr, Austria, May 3-7, 2021. OpenReview.net, 2021.
nett/forum?id=RozZ-upp
Tetiana Parshakova, Jean-Marc Andreoli, and Mare Dymetman. Global autoregressive models for data-efficient sequence learning. In Mohit Bansal and Aline Villavicencio (eds.), Proceedings of the 23rd Conference on Computational Natural Language Learning, CoNLL 2019, Hong Kong, China, November 3-4, 2019, pp. 900-909, Association for Computational Linguistics, 2019a. doi: 111815/v1/K19911/ 1084. URL httpt: //doii org/10 wttp:o...//...///..... 113863/////1911110
Tetiana Parshakova, Jean-Mare Andreoli, and Mare Dymetman. Distributional reinforcement learning for energy-based sequential models. CoRR, Abs/1192..08517, 2019b.
URL http:
farxiv.orge abs/ 1912. 08517
13

---

Published as a conference paper at ICLR 2024
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. An information divergence measure between neural text and human text. arXiv preprint ar arXhv:2102.0144, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. In OpenAl Technical Report, 2019.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level train- ing with recurrent neural networks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. UtLhttp://aree http:/aaxvvv.....aaaa.. wrgg/bs///t 06732
ROSENFELD. maximum entropy approach to adaptive statistical language modelling. Com- puter speech & language, 1063):187-228, 1996.
Ronald Rosenfeld. Stanley Chen, and Xiaojin Zhu. Whole-sentence exponential language models: vehicle for linguistic-statistical integration. Comput. Speech Lang., 15:55-73, 2001. URL
https::/api.. semapaiiccanitchhhla....
Claude E. Shannon. Prediction and entropy of printed english. Bell System Technical Journal, 30:
50-64, 1951. URL https::/api. http://app...saaaaaccccccccoooeo httpaaaci...ciioaooooooooorooaaa org/CorpusIDD.
:911029 9101213
MV Shcherbakov, A Brebels, NL Shcherbakova, AP Tyukov, 'A Janovsky, and VAe Kamaev. A survey of forecast error measures. World Applied Sciences Journal, 171-17 2013.
Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Minimum risk training for neural machine translation. Proceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguisties, 2016. doi: 10. 10.1863//1/p1-111 1159.
URL https://ddg org/10 18653 18653/v11p16-1111
Zhan Shi, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. Toward diverse text generation with inverse reinforcement learning. In Jérôme Lang (ed.), Proceedings of the Twenty-Seventh In- Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stock- holm, Sweden, pp. 4361 -3677 ijcai.org, 2018. doi: 10.24963/jcci.21188666 URL https:
ternational Joint Conference
//ãoi.org/10 http//00.24933jjcaaa.oo 22116// 2018/666
Øivind Skare, Erik Bølviken, and Lars Holden. Improved sampling-impprtancee resampling and reduced bias importance sampling. Scandinavian Journal of Statistics, 30(4):719-737, 2003.
Adrian FM Smith and Alan E Gelfand. Bayesian statistics without tears: resammpling
perspective. The American Statistician, 46(2):84 88, 1992.
Yixuan Su and Jialu Xu. An empirical study on contrastive search and contrastive decoding for open-ended text generation. CoRR, abs/2211. 2022. doi: 00.48550/rxxxv2211.10777 URL htll:httss://o// org/10, 48550/ Xiv. 2211.10799
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. In NewrIPS, 2022.
URL http:://aapet nips cttppperrcppperr_cce ftlles/paper/2222// Abstract- hhtml
Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. Best practices for the human evaluation of automatically generated text. In Kees van Deemter, Chenghua Lin, and Hiroya Takamura (eds.), Proceedings of the 12th International Conference on Natural Language Generation, INLG 2019. Tokyo, Japan, October 29 November 1. 2019, pp. 355-368, Association for Computational Linguistics, 2019, doi: 10.18853//1/W11-888 URL https:///cllantholggy
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
URL http::////ppaaereeeee.////////ts
14

---

Published as a conference paper at ICLR 2024
Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation. ArXiv, 0tts/2200602366, 2022.
Lantao Yu. Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Satinder Singh and Shaul Markovitch (eds.), Proceedings of the Thirty- First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pp. 2852-2858. AAAI Press, 2017. URL http://aaai org ocs/ index. / GAAIT//paper//ree//44553
Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. Trading off diversity and quality natural language generation. CoRR, 0bs//2004.10450 2020. URLhttpss//arxix
2ts/abb//2000 10450
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Out, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettle moyer. OPT: open pre- pre-trained transformer language models. CoRR. abs/2205.0106 2022. doi: 10.48555 010668. URL https://doi orgoooo//// 10. 18850/arxiyy 2205.01068
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. CoRR, abs/1909.08593,, 2019. URL URL/tttp://////e http://aaxi:: www.a/v.rrggb/b//eeee/eeo 1909//9999990993
15

---

Published as a conference paper at ICLR 2024
A DERIVATIONS AND PROOFS
A. PROOF OF PROPOSITION
Proposition The distribution that solves the optimization problem ① in the form of:
po.p (x)) po(a)) exp Fe((ar] Va E S(po.x)
(2)
where (x) 1 f()) and S(p) {æ p(x) > 0} is the support of distribution p IR is determined by the constraints in 7
Proof. The optimization problem with full constraints is follows:
arg min (gl(p))
(5)
s.l. Ex (#) = lx [xx),,,
E (I,...,,
(6)
((x) 0, Vo E X'
Σ #EX
((x)) 1,
(8)
where Eq. 7 and ( are implicit constraints that guarantee to be probability distribution over X. The Lagrangian of the above optimization problem is:
((x) pO(
Llg.u, u, x) Σ
g(x) log
+ Σ xEX
(())f(x)
g(x))(2)+T
1- 1
{(x)))
xEX
#E
EEX
where f.p RK
X F E Eowpa (f (x)).
The optimal set of solutions (gopt: Mopt: Aope, Topt) satisfy the Frrrush-Kuhn-Tuckee conditions:
9op)(2:) (x)
log
+ + Mop f(x) >spr((x) - Topt = 0,
Va E X
(10)
(x(xx)
Σ Copt ((x)
(xm((x)) 1, (ope(20) 20.
Vx X
(11)
#EA
Aapa (x=)
2 O,
(xp = =0,
Vo X
(12)
We first obtain the form of Gopt from Eq. 10P:
(x) = (x(x) exp foptf(z)) (x) + 2xpx(a))
(13)
where 7 = EmEx Pa (m)) expl- f + (a=)) is the normalizing constant due to Eq.
11
For every x' such that (x)) > 0, from Eq. 12 we have Aupa (x') = O. which thereby proves Eq. (2).
A.2 PROOF OF PROPOSITION 2
Proposition The optimal solution gopt of the optimization problem
satisfies:
1. S(gopt) 2 S(ps), where S(p) {x: p(x) 0}.
2. H (ps, (opt) H (pa. pa) DKL (gopppellpp) (pp), where (p. E. p(x) log
Proof. Denote the set of probability densities that satisfy the constraints in the optimization problem D as C:
C = {g P Ex-e [f]( (f) Vk [K]]. Then we consider P.. = (1 - a)q + apa E for a E [0, 1]. where qom is the optimal distribution that solves the optimization problem. Due the convexity of C. Pa also belongs to C.
(14)
16

---

Published as a conference paper at ICLR 2024
Next consider the following derivative 0D (Pa (po)lllo)aaaaaa at 0:
a
Pa (x(
DKL (pallpe))
Σ Par (a (z)
log
(15)
da
a==
po(x))
a==
(1)
Pa (a(
(xx()))
log
(16)
da
po
aa
a==0
Σ log Po (x)
+ 1) (pa(2e) - (ope(
(17)
px(x))
rm0
Σ (pa(æ)) - 40p((x)) log Po (xc)
(18)
(z)
la=0
Gopt (x:
E(na(a)) - (2))()))) log pa (x))
(19)
Σ -pa(xr) log (a())) + pa(x:) log (ap(( - (xpe( (a) log lop((z) (x)
(20)
po (x)
H (pa. H (pa, Yops) - DxI (popellpo)
(21)
While 0Dx (pa (olllloo///oo can also be written in the limit form:
0
DKL (Pa (pp) DKL (9opilpp))
DKL (Pa (pallp))
lim a-+
(22)
a=0
a
Due the optimality of Yopc: DKL (ooplll))pe) < DEL (PoilPs), Va [0, 1]. which proves that 0DkL (Po pa) 2 O. Therefore, for Eq. (19) to be non-negative, we must have (x) # 0 for any S(pa) (otherwise converges to -GG), which proves the first claim, S(gop) 2 S(pa).
Next, we show that there exists some a' < such that par is a probability density. For any a E ((op)):
Per (x: = (1- a (app( + a' a'pa(x))
= Qop + a' (p(()) - Aopi (3)].
Pa())) 2opt (x)) O, we always have (ax) ) = ()p((2)) 2
If pa(x) Yape (x) < O, we always have pa (xx) Yop (2x) - a' 1yope - pa(a)]] 0.
pa(x)) - Qope (x) > 0, we set = -ssup pa(a²)
1|-1 as Pa (z) (op(( is always greater than
(a))(ar))
1. Since a' 2 (a) ".) have Pa (x)) 2 O.
(()(
Therefore, we prove the non-negativity of pa'. of expectation. Therefore we show that:
Then it is trivial to show that par C by the linearity
(Pa llpo) - DKI (Topilll (loo)lpo))
DxL (pallel)pe)
lim 40
< 0.
(23)
da
a'
Combining Eq. 22 and 23 we arrive at 0D (80) = 0, which proves the second claim, H(ps, 9opt H (pa. po) Dea (gupulpp))
A.3 PERPLEXITY OF THE OPTIMAL DECODING DISTRIBUTION
The perplexity of the decoding distribution pa,, can be derived by taking the exponent of the cross- entropy between Pa and ps, In practice, we evaluate the test set {x" N which forms empir- ical distribution by drawing samples from P.- We also modulate the proposal model by temperature (denoted p%) to make the result consistent with the inference stage.
17

---

Published as a conference paper at ICLR 2024
We derive the cross-entropy (pa, p),,)) at the token-level by plugging in Eq. (
Hpss po,u) =
log pe, a (x")
ΣN x
log (2 )Ful)))))/22 (z)
Ei=1 x
- Ey((x()) - log Σ Je Bula)))
Σ
axd
1
H(pa, Pō) +
Σ Fa + log
Σ (p5(a")) (uull))e-Bllt)))
Ei=
.V
|art|
where (pa. pb) is the cross-entropy the proposal model with a temperature modulation:
N m
exple(x))TM) hw/r] E wEv exple(w) ht/T]
Hpd: Pτ)
where e(w) is the output embedding of and h, is the hidden state at time step t.
Finally, the perplexity of the decoding distribution 4 in DAEMON is ("(a.eee))) in nats.
A.4 DETAILS OF COFFFICIENTS ESTIMATION WITH WIS
A.4.1 WIS DERIVATION
We derive the estimation of F using Weighted Importance Sampling (WIS) with N ii.d. trajectories ( jam from the proposal po:
F E Egapa P [x()]
Σ (0)
En Polax) exp( (x)
Ee po(x) exp( exp(-EEp1 (x))
A=l exp( - F ④1 f
L=l exp( - (
The last step is obtained by approximating (=(x)) with the empirical distribution defined by { N i... Po (a)
Tim1s
5(a,
A.5 DETAILS OF SAMPLING USING SIR
A.5.1 THE CONDITIONAL. FORM OF THE OPTIMAL DECODING DISTRIBUTION
We first start with the auto-regressive factorization of Eq. (2) at step which marginalizes out the future from step t:
Basi mo (oce)) (exp( - (x a'
Po pe (2, x2e) = po (2o (x2))
Enyg, upol (me)) (exp( exp((=Ex(xep (x 222,)
This form can be derived by breaking down the probability at the last step pa y (IT xc xT)) and then generalize the derivation to previous steps.
18

---

Published as a conference paper at ICLR 2024
Then the conditional form of pa,, given prefix Esto is the product of the per-step probabilities:
T
Po (1 (2) = II
Bns, -pol (asg)) lexp( Eu (x 2',1,
po Fgtto)
II
t=fo+1 Bung mpo( (= lexp( Ex (x <t= 222
exp( y / (t <tp It +ztaa)
= po (g
(24)
Bu''.. ~po(t ( (exp( - E.j (2 x 20
The last step is obtained by canceling out the intermediate terms in the product from step to 1 to T.
A.5.2
DERIVING THE SIR APPROXIMATION
Ph The empirical distribution pa (-1=3tt)) induced by SIR approximation can be derived from Eq. (24) by substituting the conditional proposal po(+(=gt,) with the empirical distribution f9 ta ( = N7 Ei= 8( (3tol, where ( I are continuation candidates sam- pled from the conditional proposal po
exp( F, Sto S ) Σ م xx(al+
M1
p0 (x xZ66)) =
ΣS= exp( F2y (tr x2t0))
B RELATED WORK
B.I DECODING METHODS
Decoding methods typically categorized into two main categories: sampling- methods and search-based methods. Sampling-basee methods, which introduce randomness into the selection of next token, are commonly employed in open-ended generation settings yield diverse texts. Ex- isting sampling-based methods select the next token by sampling from truncated set of candidates based on heuristies that control the statisties of the next-token probability distribution. For instance, Top-k sampling selects the set of highest A: probabilities (Fan et al.) 2018), Nucleus sampling focuses on candidates within the p-percentile of the highest probabilities (Holtzman et al. (020), and Typi- cal sampling targets the --eercentile, which has entropy close to that of the language model (Meister (ct al., (002). However, remains unclear how these controlled statistical quantities correlate with the aspects of generation quality one might concern. In particular, sampling based methods often reported to struggle with maintaining topic relevance and long-term coherence. Conversely, vanilla search-based methods, which search for the sequence that maximizes the probability der the language model, tend to produce repetitive and recursively structured texts in open-ended scenarios. Consequently, frequency penalty (Keskar al.) 2019 Dou al.) 2022) is employed to discourage generating tokens that frequently present in the context. Recently, several works have been proposed to maximize contrastive objectives. Contrastive Decoding (Li al. 2022) seeks the token that maximizes the probability difference between an expert LM and an amateur LM. Con- trastive Search (Su et al. 2022) searches for the token that maximizes the probability given by the LM while minimizing its representation similarity to the previous tokens. Nevertheless, the output of these methods still frequently exhibits redundancy the long term due to the intrinsic bias in the language model, despite the aim of their objectives reduce semantic repetition. Our method, cate- gorized as a sampling-bar based method, aims to induce the optimal decoding distribution that explicilly aligns with the underlying distribution of human text, assessed by evaluation metrics related to set of chosen aspects.
B.2 LANGUAGE MODEL TRAINING METHODS FOR QUALITY IMPROVEMENT
Prior works also attempted to improve the generation quality by devising new training objectives to further train the language model. One approach involves the design of auxiliary objectives aimed
19

---

Published as a conference paper at ICLR 2024
at discouraging the model from learning implausible samples features by the standard Maximum Likelihood Estimation (MLE) objective. WWeleck et al. (2020) proposed unlikelihood training ob- jective that directly penalizes token-level and phrase-level repetition. Xu et al. (2022) proposed the DITTO objective that penalizes sentence-level repetition loop. SSu et al. (2022) proposed a con- trastive objective that separates the representations of distinct tokens in the context to facilitate de- coding with repetition penalty. Despite the direct mitigation of undesired behavior of the model, these objectives are usually applied to the local probability of next token prediction by the language model, which is conditioned on the ground-trum contexts. This leads to discrepancy between train- ing and inference, i.e., the exposure bias, raising concerns regarding the preservation of the learned properties at the inference stage. Another research direction involves the application of Reinforce- ment Learning (RL) to optimize the generation model towards sequence-level metries (Ranzato) et al.) 2016;; Shen et al.] 2016 Yu et al. 2017] [uo et al., 2018 [Shi et al. 2018). Although tar- geting at optimizing the model at a sequence level. most RL approaches often underperform MLE especially with pre-trained base model. Pang & He (202T) demonstrated the effectiveness of off- policy RL that learns from human demonstration with quality- reward over the approach of direct fine-tuning. More recent research explored learning the reward function from human judg- ment (Jaques et al. 2019 Ziegler et al.] 2019 Ouyang et al.) 2002)) and has shown to promote the pre-trained language model to generate texts whose quality more aligned with human prefer- ences. Our method circumvents exposure bias and the challenges associated with RL optimization by deriving the analytic formulation of the proposed optimization problem, which directly aligns the expected metric scores of generated texts with those of human texts.
B.3 SHERGY-BASED MODEL FOR TEXT GENERATION
The Energy-Basee Based Model (EBM) (Hinton 2002 LeCun et al. 2006) is a generative model that learns the underlying distribution by relocating energy based on sampled data points. In text mod- eling, the EBM is attractive due to its global sequence scoring, as opposed to the locally normalized score factorization in Auto-Regressive (AR) models (Rosenfeld et al. 2001 ROSENFELD 1996). Theoretical work has demonstrated that the EBM family offers greater expressiveness than the AR model family by encompassing broader range of computable text distributions (Lin et al. 2021ai However, two critical aspects of the EBM, namely, learning and sampling, pose challenges for its practical application in text generation. While AR models can be trained by directly maximizing the likelihood of reference data, learning a parametric EBM typically involves Noise-Contrastive Estimation (NCE) (Gutmann & Hyvärinen 2012) Ma & Collins 2018). With the recent advances in pre-training, previous studies proposed to construct the EBM based on a pre-trained language model with an exponential energy term that is parametrized by neural network (Deng et al. 2020) or log- log-linear model (Parshakova et al. 2019a). Specifically, the latter form emerges as the solution based on the generalized maximum entropy principle (or minimum discrimination information prin- ciple), incorporating linear constraints on model expectations (see Eq. (I). which is also explored in Posterior Regularization technique (Ganchev et al.) 2010). This formulation has also found appli- cations in areas such as controlled text generation with distributional control (Khalifa et al. (02T) and calibrating the entropy rate of language models (Braverman et al. 2020). among others. On the other hand, achieving scalable and tractable sampling from the EBM has long been challenge, primarily because of its globally normalized form. Traditional MCMC approaches such as Gibbs Sampling (Geman & Geman 1984) or Metropolis Hastings (Metropolis el. 1953) often face scal- ability issues concerning data dimensionality and model complexity. Recent endeavors (Parshakoval et al. 2019b; Khalifa et al. 202l) proposed to learn another AR policy guided by the EBM. Never- theless, this approach compromises the modeling capacity of the EBM, as it involves learning an AR policy that minimizes the forward Kullback- Leibler (KL) divergence towards the solution of the op- timization problem. Finally, energy-based reranking methods (Bhattacharyya et al. [202]] Fernandes et al. 2022, Freitag et al.) (202)) are explored in machine translation where the generations are re- ranked according to pre-deñned reference-basee metrics. However, these methods cannot guarantee to improve the distribution of the base generation model to resemble the underlying text distribution from humans. They only focused on maximizing certain quality metrics while neglecting various aspects of human texts (e.g.. quality, diversity, repetition and etc.) which are necessary for achieving human-lik generation. Our approach generalizes the constraint functions to encompass evalua- tion metries across various aspects in the text decoding formulation with the goal of aligument with human texts, and facilitates tractable sampling from the EBM by leveraging strong AR proposal
20

---

Published as a conference paper at ICLR 2024
through Sampling- hrchingg-mpotttnnteemoo Resaennlei---eeinnggggg (SIR), which possesses well-defined convergence rate.
C DISCUSSION ON THE FORWARD AND REVERSE KL DIVERGENCE
We provide an in-depth discussion on the forward and reverse KL divergence. We reuse the notation in Section 2. and denote the distribution induced by a language model as Pa and the decoding distribution as q. Although both the forward and the reverse KI. divergence can used measure the distance between and Pe. they are not symetric (Bishop & Nasrabadi] (2006), i.e., DXL (Pelly 1l4) + DxL (gllpe), and have distinct differences shaping the decoding distribution.
Minimizing the reverse KL divergence, Dkr. (glpo) is known to encourage zero-forcing behav- ior (Malinin Gales 2019), as forces g(x)) 0 when po p0(x) = This restricts the decoding distribution q to be mode-seeking and only explores the modes of the target distribution pa which contains samples with high likelihood under the ground-truth model's distribution po, and thus ig-
nores the outliers.
Whereas, minimizing the forward KL. divergence, DKL (Poll4) 11q) encourages zero-avoiding behavior, Le., it avoids (xx) when po 0. This leads be mean-seeking which spreads its support to cover all the non-zero density regions of the model distribution pa. As the distribution of language models known have unreliable long tails (Holtzman al., 2020), the decoding distribution that
minimizes the forward KL can further overestimate those tails to produce low-quality samples (Chan
al. 2022).
Overall, comparing to the forward KL, minimizing the reverse KL leads to more focused distri- bution and ignores the long tail in the given language model distribution, which more suitable for our decoding scenario that demands generation quality. We also add an intuitive visualization to illustrate the different behaviors of reverse and forward KL Figure
0.8
0.8
target forward KL
target reverse KL
pbx)
0.4
0.4
0.0
0.0
2
0
6
(a) Forward KT..
(b) Reverse KI.,
Figure 4: The mean-seeking behavior of forward KL and the mode-seeking behavior of reverse KL.
D HUMAN EVALUATION SETUP
D.1 ANNOTATION PROCESS
We conduct pair-wise human evaluation with Amazon Mechanical Turk workers, providing them with comprehensive instructions and detailed explanations of annotation features. To ensure that participants can fully comprehend and make accurate judgments, we conduct pilot qualification test using 30 pairs. For the formal annotation process, we randomly select 100 prefixes, each with length of 32, from the set of Wikipedia. We employ GPT-2 XL (1.5B) with four baseline decod- ing methods (i.e., contrastive decoding, contrastive search, nucleus sampling, and typical decoding) and our proposed DAFMON algorithm to generate text continuations, each with a maximum length of 256.
Each Human Intelligence Task (HIT) consists of the prefix, the two continuations, and evaluation questions regarding three evaluation criteria. The HITs are randomly shuffled and evaluated by three annotators, adding up to 1,200 annotation samples (100 prefixes x baselines x criteria). Annotators required to assess and choose the better continuation or opt for draw according to three evaluation criteria: coherence, fluency, and informativeness. Annotators are paid at a rate of S0.5 per HIT.
21

---

Published as a conference paper at ICLR 2024
Participation in the annotation task is exclusively open to qualified workers that meet the following qualifications:
HIT Approval Rate must be greater than 95%.
Number of HITs Approved must exceed 500.
For quality control, we conduct manual cross-validation to check the annotation results based on the following rejection principles. The unqualified HITs are rejected and requested re-annotation.
Review time is less than 500s.
Obvious Misjudgment. Key points include: (i) Assigned higher fluency to the text with more grammar mistakes or repetition. (ii) Assigned higher coherence to the text with more jumbled topics. (iii) Assigned higher informativeness to the text with more redundant ex- pressions.
D.2 ANNOTATION GUIDELINES
Instruction:
We will give short prefix and two continuations generated by two systems. You required to compare the fluency, coherence, and informativeness of two continuations based on the detailed guidelines. The three annotation features explained follows:
Coherence:
Coherence assesses the semantic consistency of the generated text with the prefix. Key
aspects encompass:
Discourse Coherence:
The generated text maintains semantic connections and appropriate
transitions.
Logical Coherence:
The generated text maintains logically consistent and avoids self- contradiction (e.g., chronological, cause-and-effect, narrative).
Global Coherence: The generated text aligns with the provided prefix and remains the same topic, event, entities.
Fluency: Fluency refers to the extent to which the generated text is smooth, highly readable, and easily comprehensible. In contrast to coherence, fluency primarily underscores intra-sentence read- ability. Several pivotal facets encompass:
Grammar: Grammatical correct and adheres to proper English grammar.
Structure: Clarity in sentence structure.
Semantic: Absence of repetitions or omissions.
Vocabulary: Avoidance of inappropriate phrase combinations symbols.
Informativeness: Informativeness refers to the richness and diversity generated text. Key aspects include:
The text should offer valuable information with engaging details, avoiding redundancy and triviality.
Comprehensive Information:
Lexical Diversity: Use varied vocabulary and sentence structures.
In-depth Details: Provide pertinent, in-depth details.
Novelty: Introduce elements while maintaining coherence.
E RUNTIME ANALYSIS OF ALGORITHM
We first analyze the computational cost of Algorithm which mainly consists two parts.
The first part takes
of unconditional sampling from the base language model (line 2 of Algorithm[I] T). We typically set the number of samples to be the same as the development set, so that this cost is equal to time sampling on the development set.
22

---

Published as a conference paper at ICLR 2024
The second part is estimating the parameters {4r)e1 K by iterative gradient descent (line 3-6 of Algo- rithm []. As the number of parameters is small (equals to the number of constraints which is usually in the scale of dozens in practice), and the metric scores {ff(z)) can be pre-computed, the total computational overhead very small, In our experiment, it takes less than one minute to perform thousands of gradient descent steps, which can be ignored comparing to the cost in sampling from the language model.
Comparing to typical grid search of hyper-parameters which requires H R runs of sampling in the development set, where H stands for the number of hyper-parameters and R is the number of trials for each hyper-parameter Even the labor intensive manual tuning methods require at least one run of sampling on the development set, which do not have any advantage over our Algorithm inn terms of computational cost.
F COMPLEXITY AND RUNTIME ANALYSIS OF DECODING METHODS
We provide analysis about the computational complexity and the runtime in practice of different de- coding methods and our method (Algorithm 2) In the following, we consider the case of generating a completion with full length T given an imput prompt with length to (much smaller than T). To evaluate the real runtime performance, we follow the setting in Section 3.6 which uses GPT2-XL to generate completions with maximum length of 256 on the test set of Wikitext. The experiment was done on Tesla V100.
Greedy Decoding, At each decoding step, the Transformer model attends to the previous tokens with the complexity of OT)) and then performs positional transformation, e.g., linear mapping. layer normalization and etc., which is not dependent on T.. Hence, the full complexity of generating the entire completion is O As greedy decoding only picks the token with the maximum probability, it does not incur additional complexity at each decoding step. In our experiments, the runtime for decoding a completion using greedy decoding 9.33 seconds on average.
Top-k Sampling. Top-k sampling has nearly the same complexity as greedy decoding. The only difference is that each step samples from the subset that has top-k probabilities rather than taking the maximum. In our experiments, the runtime for decoding a completion using top- decoding is 9.34 seconds on average, which is almost the same as greedy decoding.
Nucleus Sampling. At each decoding step, Nucleus sampling sorts the probability distribution, which incurs an additional complexity of O(V V) where is the vocabulary size comparing to greedy decoding top-k sampling. In our experiments, the runtime for decoding completion using nucleus decoding is 10.54 seconds on average, which is 1.13 times of the latency of the greedy decoding.
Contrastive Decoding. Contrastive decoding uses two language models with different sizes for decoding, where a single language model has the complexity of O During decoding, they use beam search which increases the total forward complexity to O(BT2) for a beam size of B. Additionally, at each decoding step, beam search has sort the probabilities in the beams which has a complexity of ( log( BV).. In our experiments, we followed the beam size of 5 suggested irLi et al. (2022), and the runtime for decoding a completion using contrastive decoding is 12.10 seconds on average, which is 1.29 times of the latency of the greedy decoding.
Contrastive Search. At each decoding step, contrastive search has to perform two forward passes of the language model. After the first forward pass, the algorithm selects the top-k candidates and performs the second forward pass to get the hidden representations of these candidates. This increases the per-step complexity to Ok(T), which results in the total complexity of O O.kT2). Note that the two forward passes cannot parallelized and the serialized execution slows down the run time. In our experiments, we set k to 5 according to Su et al. (2022), and the runtime for decoding a completion using contrastive search is 1 11.82 seconds on average, which is 1.27 times of the latency of the greedy decoding.
DAEMON (Ours). Our decoding method DAEMON has two stages, i.e., candidate sampling and re- sampling to approximate the optimal sampling distribution. In the first sampling stage, sample M candidate sequences in parallel, whose total complexity is OMT22) In the second re-sampling stage, compute energy for each candidate sequence and re-sample from the distribution defined
23

---

Published as a conference paper at ICLR 2024
by the energy. The second stage has complexity of O(CM) where C denotes the complexity of calculating the energy, which much smaller than the cost of sampling. Note that unlike contrastive decoding and contrastive search, in the first sampling stage of our method, sampling M sequences can be fully parallelized, which is highly optimized by modern GPU hardware. Hence the actual inference lateney grows much slower when increasing M. In our experiments, set M to 25, and the runtime for decoding a completion using DAEMON is 12.58 seconds on average, which is 1.35 times of the latency of the greedy decoding.
G CONVERGENCE AND SENSITIVITY ANALYSIS OF ALGORITHM
We analyze the convergence of μ in Algorithm| | Jsing different initializations of Concretely, we consider the following three initializations:
zero: Initializing all dimensions of μs to O.
randn: Initializing dimensions of μ with random numbers sampled from a standard normal distribution (0, 1).
rand: Initializing dimensions of with random numbers sampled from a uniform distri- bution (O,
We run Algorithm T and consider the following nine metrics in the main experiment: SEQ-REP- 2, SEO-REP-3, AFQ-REP-4, TOK-REP-8, TOK-REP-16, TOK-REP-32, COH, DIV, en on Wikitext. Each /4 (i E (1....,9) corresponds to the above metric respectively.
We plot the optimization trajectory of each 14 under Algorithm with different initializations in Figure We took runs with different random seeds for each initialization.
2008 2004
1880
2900
20008
2000
1880
38
2008 2003
18:0 2800
10K8 X00
2508
1000 200
Figure 5: Optimization curve of each μ4 with three different initializations: zero, randn, rand.
From the result, we observe that the optimization of μ is quite stable and all the trajectories converge to almost the same optimal solutions under different initializations.
H ROBUSTNESS ANALYSIS
We study the robustness of the alignment results on different metrics when one chooses to optimize a single metric. We conduct our experiments using GPT-2 XL on Wikitext and perform a grid search with different combinations of M E {10, 25, 50, 100) and E {0.96, 0.97, 0.98, 1.0] to optimize towards aligning a single metric one of the following metries: SEQ-REP-4, TOK-REP-32, сOH, DIV, eNT, From the result in Table 5 we observe that the performance different metrics has very small variance even when we optimize for single metric, which indicates the robustness of DAEMON.
I ADDITIONAL RESULTS ON TEXT SUMMARIZATION
To demonstrate the universability of our method, we conducted an experiment on the stan- dard summarization benchmark CNNi/ailyMail using off-the-shelf summarization model pegasus-cnn ceannddallmaillll As shown in Table compared with the baseline sampling methods, our method achieves better performance in aligning with most aspects including repetition (SR-3, TR-8), coherence (COH). diversity (DIV) and information (CT (better when the score is closer to that the reference), and ROUGE scores (better when the score higher).
httpp///wuuggnggdeeecoogggggggge hatpoo//ee/pppuuuusemmaaaaaaaaaa cnn_dailymal
24

---

Published as a conference paper at ICLR 2024
Model
SR-4
TR-32
ENT
Reference
0.48
21.3
62.3
92.5
23.2
opt. SR-4
0.42
22.5
62.5
92.2
22.8
opt. TR-32
0.38
21.2
62.4
94.1
24.3
opt. COH
21.2
94.1
opt. DIV
0.55
22.9
63.3
92.7
22.2
opt. e
0.40
21.5
61.6
93.9
23.1
Table 5: Quality of generation measured under all metrics when one only optimizes for single metric at decoding time. For instance, "opt. SR-4" means only optimizing SR-4.
Model
SR-3
COH
DIV
eNN
R-1
R-2
R-L
Reference
0.10
2.93
80.1
99.1
31.2
Top-k
0.15
3.10
79.6
99.2
35.0
38.7
15.0
35.6
Nucleus
0.15
3.00
72.3
99.0
104.3
34.4
12.2
31.6
Typical
2.71
76.3
99.2
81.8
34.3
12.1
31.5
CS
0.15
3.14
80.9
98.8
28.4
40.9
16.9
37.8
Daemon
0.10
2.98
80.1
99.2
29.7
41.9
18.4
38.9
Table Results the CNN/DailyMaii..
J EXPERIMENT DETAILS
J.1 STATISTICS OF DATASETS
Domain
# Dev set
# Test set
Prefix length
Full length
Wikipedia
512
802 1488
24.8 25.6
183.1 309.5
News
512
Table 7: Statistics of the data used in the experiments.
The data statistics of each domain are summarized in Table 7 For each example, the first 32 words used the prefix. Each word is tokenized into subwords before fed into the embedding layer of the language models. Both GPT-2 and OPT use Byte-Pair Encoding subword segmentation. The average length of subwords in the prefix is around 32. The maximum length of subwords in the complete generation is restricted to 256. During evaluation, the subwords length of human references also truncated 256 for reasonable comparison.
J.2 IMPLEMENTATION DETAILS
We first describe the implementation details of the baselines. We first consider three canonical sampling-based methods: Top-k sampling (Fan al. 2018) samples from the of candidates with
top-k probabilities. Nucleus sampling (Holtzman et al. 2020) samples from the set of p-percentile that has minimum number of candidates. Typical decoding (Meister al., 2022) samples from the set of r-percentile whose entropy is close to the entropy of the LM. For search based methods, besides vanilla Greedy decoding, we also consider two recent methods that maximize contrastive objectives: Contrastive Decoding (CD) (Li et al., (022) searches for the token that maximizes the probability difference between an expert LM and an amateur LM scaled with temperature 7 within candidate set controlled by Contrastive Search (CS) (Su et al. 2022) searches for the token that maximizes probability given by the LM while minimizing its representation similarity to the previous tokens with intensity of within the top-k candidate set. For all baselines, we the recommended hyper-parameter settings in the original papers. Specifically, we use 50 for Top-k sampling, p 0.95 for Nucleus sampling, and 0.95 for Typical decoding. For
25

---

Published as a conference paper at ICLR 2024
Model
SR-2
SR-3
SR-4
TR-8
TR-16
TR-32
eNNT
Reference
5.82
1.40
0.48
5.77
12.8
21.3
62.3
92.5
23.2
Greedy
66.2
62.9
60.9
13.5
39.4
65.5
60.3
8.03
2.29
59.7
Top-k
8.48
3.28
2.11
6.74
14.5
23.4
60.9
87.8
10.1
77.8
Nucleus
5,51
1.84
1.19
5.98
12.4
20.0
57.3
92.4
Typical
3,98
1.24
0.81
5.07
10.7
17.4
54,9
94.5
30.1
78.7
CD
10.5
3.00
1.31
8.00
17.3
28.2
68.7
86.0
7.55
77.8
CS
6.71
2.51
1.78
6.40
14.1
23.0
56.9
90.6
5.25
83.3
DAEMON
6.28
1.31
0.42
6.59
13.6
22.5
62.5
92.2
22.8
88.1
Greedy
61.4
57.3
54.8
12.6
35.5
60.4
62.0
12.6
2.78
64.8
Top-k
9.14
3.77
2.44
7.33
15.1
24.1
61.3
Nucleus
7.69
3.36
2.33
6.48
13.5
21.9
59.1
88.6
18.9
80.1
Typical
5.16
1.70
1.06
5.70
12.2
19.6
57.0
92.9
31.9
77.7
CD
11.8
4.90
2.92
6.81
15.3
26.5
68.6
82.3
11.7
78.6
CS
5.89
1.94
1.14
6.32
13.7
21.7
57.7
91.8
8.72
83.3
DAEMON
5.89
1.33
0.38
6.19
13.3
21.6
92.6
22.7
Table 8: Full main results of automatic evaluation on the Wikipedia domain using GPT-2 XL and OPT-6.7B. For all metrics, the best scores (in boldface) are the closest to the human scores except for MAU, which is better when higher.
contrastive decoding, we directly use the generated texts provided by the official implementation with hyperparameter setting: 0.1, 0.5 for GPT-2 and search, we follow the official implementation on the datasets (Su & Xu 2022) that uses a 0.6, k
1.0 for OPT. For contrastive
For our method DAEMON in the main results, we use the following metrics in the constraints: SR-2, SR-3, SR-4, TR-8, TR-16, TR-32, COнI, DIV, EST described $32 For coefficients estimation, we estimate the target expectation on the dev set and fit the coefficients using Adam with learning rate 5e-3 until a minimum error le-3 reached. The inference DAEMON with the base model of either GPT-2 XL. or OPT-6.7B can be done on Tesla V100 with batch size of 1.
J.3 FULL MAIN RESULTS
We present the full results of all metrics on the Wikipedia and News domain in Table and Table
respectively.
J.4
QUALITATIVE CASES
We present qualitative cases generated by four baselines and DAEMON in Table T0
14
15
26

---

Published as a conference paper at ICLR 2024
Model
SR-2
SR-3
SR-4
TR-8
TR-16
TR-32
COH
DIV
ENT
MAU
Reference
4.76
0.93
0.29
4.72
10.8
18.7
66.6
94.1
13.8
Greedy
58.7
55.1
53.2
8.06
28.0
58.2
63.8
13.2
2.19
65.2
Top-k
6.16
1.80
0.95
5.26
11.8
20.3
64.7
91.7
8.17
96.3
Nucleus
4.91
1.39
0.80
4.93
10.9
18.7
60.8
93.5
11.0
95.3
Typical
3.62
0.83
0.42
4.50
9.98
16.9
57.2
95.3
18.2
95.0
CD
7.45
1.78
0.63
5.97
13.5
23.2
71.2
90.5
6.55
CS
4.45
1.23
0.77
4.68
11.0
19.2
63.6
94.1
4.18
95.7
DAEMON
4.64
0.71
0.18
4.71
10.8
18.7
66.3
94.5
13.7
97.4
Greedy
51.3
47.2
45.2
7.75
24.5
51.0
63.6
21.9
2.72
70.7
Top-k
6.73
2.42
1.53
4.93
11.4
19.9
65.7
90.5
10.7
95.7
Nucleus
6.03
2.13
1.37
4.95
11.1
19.2
63.3
91.5
12.2
95.3
Typical
4.59
1.47
4.52
10.1
17.7
59.4
19.4
CD
8.34
3.23
1.94
4.91
11.5
21.0
71.7
87.5
9.20
95.2
CS
4.74
1.74
1.18
4.52
10.6
18.3
62.9
93.2
6.69
94.0
DAEMON
4.30
0.67
0.25
4.52
10.7
18.8
64.8
94.9
13.6
97.2
Table Full main results of automatic evaluation on the News domain using GPT-2 XI. and OPT- 6.7B. For all metrics, the best scores (in boldface) are the closest to the human scores except for MAU, which is better when higher.
27

---

Published as a conference paper at ICLR 2024
Prefix
Morris Dickstein of The New York Times was more critical of the book, calling it " a miscon- ceived and badly executed novel, sideshow distraction from
CD a much bigger story" " and saying the book's author had failed " in her most basic duty to the public."
The Times review, by David Streitfeld, is here and the book review by Morris Dickstein, by Times, here. The review by Morris is more critical, calling "a misconceived" and badly executed novel," a sideshow distraction from a much bigger" story and calling the author a failure "in" her "basic" duties. The review by David is more positive. He says that it "has its charms, and the author is clearly talented story teller," and he praises the author for "her skill in weaving is more critical. He calls the novel sideshow distraction and a "misconceived and" hadly "imperfectly" executed novel, a "fails in its most basic" duties
engaging tale." The book is available here. The hook by Morris
CS
the main event."
In an interview with the Los Angeles Times, Morris Dickstein said, "There's 's no reason to write a book about this that's going to get people's attention except to make it into a movie, and I'm movie person all. The idea of making a movie out of this bad idea in my opinion."
Morris Dickstein's 's comments were met with derision by a number of authors, including David Mamet, who wrote in his review of the book, "It's hard to imagine writer more in the vanguard of the literary avant-garde than Morris S. Dickstein, whose work is the stuff of literary legend: The Man Who Knew Too Much, A Passage to India, All Quiet on the Western Front, To Kill a Mockingbird to name a few his masterpicces." Mamet went on to say." "Dick
Typical
its own lame screenplay" while reviewing edition number 632, 1952. Here's an excerpt from an actual phone interview that took place between Normen's 's editor, Margaret Woods Hamlin, and a reporter from Life magazine. You can try remaining objective when it comes to incom- petent film scripts. then.
Madame Normen 1952 with one of the terrible terrible script-writing venues where she allowed herself to stand and present like an academy award nomince (the anything can be anything, anybody can fly, anybody can speak ad lib, "These words and ideas can put tray of mutton bones into mouse's muffin; to mc they mean nothing.")
Princess Ping-Pong: What do you mean you hate that book? Should I deny it?
Normen: That book is re-naming ballet for beginners. Do you know, one the earliest posters coming from front pages in the finest role they can find
Nucleus
22900 and One Hundred Thousand Grapes. In 2010, published book called The Disap- pearance of the Jews in Europe: A Statistical History, and just came out today. In this episode, which makes for very odd sort of audio, A.O. Scott takes over the show and discusses the Holocaust in nearly eight minutes, describing the era, its mood, the statistics, its victims, its symbols,
orchestrating forces. and even some of its realities. All of chilling. really, coming from a man who was born one month after the holocaust ended. Best listen: Enjoy the full minute and twenty-five seconds if you can. This week, we" ve got exclusive preview of a whole new book from George Steiner, the Holocaust: www. www.housedistor..o org hoousmacherrookss. You can also hear A.O. Scott and
DAEMON
questions of it's own dubious existence, a carbon copy a similarly flawed book that deals with the woman the author has always hated- her mother in a more kind and humble way."
A correct assessment of the novel as book beats an unduly critical one quoted correctly dismissive one. The book is good it's an intelligent and emotional account of the social influence of the publishing industry on children's 's books, part Dickstein's ongoing concern with capitalist and legal reform.
Dickstein doesn't 't spoon- feed the reader plot points and he allows the reader to create her own meaning in constructing the novel's narrative. The author doesn't 't wander into the busy life of teenager, which we remember often enough.
The opinions of our readers are important to us - just click on one of these and pass along to friends.
This opinion article was written by independent writer. The opinions and views expressed herein
Table 10: Randomly selected generations from the test set of the Wikipedia domain.
28

---

Published as a conference paper at ICLR 2024
Prefix
Chad is a landlocked country in Africa whose northern region lies within the eastern reaches of the Sahara Desert and whose southern region lies within the castern portion the
CD
Arabian Desert. Chad's capital city is Ndjamena and it horders the Republic of Mali on its west, Sudan to its north, the Central Sudanese Republic to the north-east, the Central Sahel on its eastern side and the Republic of Nigeria in the east. Chad is bordered to its south-west and east by Cameroon, Sudan and Libya, while its west by the Republic of Chad and to its north by Algeria. Chad shares a land boundary with Sudan to its west and with Libya and Mali on the north and west respectively. Chad's population approximately 11 million people are mainly Christian, while a small Muslim population exists. Chad's economy largely agro- based with agriculture and cattle raising the major economic activities, while mining is also a significant economic sector. Chad's main export products include cocoa and oil palm, while its major import is crude petroleum and refined oil products. Chad's main natural resource are its mineral deposits, including
CS
Sahel.
Chad has population of approximately 18 million, which more than half are ethnic Tu- areg and Berber populations. In recent years, there has been a rise in Islamist extremism in the country, which has contributed to the country's security situation. The government has taken steps to counter this threat, including the deployment of military forces the border with Mauritania and the use air strikes against al- al-Qaeda-lintee militants in the north of the country.
Terrorism Threat Worldwide 2014 World Terrorism Report (WT) The 2014 Worldwide Threat Assessment (WT) from the Combating Terrorism Center at West Point, New York, assesses that the threat from transnational terrorist groups remains high. The report, based on data from January through December 2014, identifies four groups that pose the greatest threat to the United States: ISIL (Islamic State of Iraq and the Levant), Boko Haram, Jemaah Islamiyah (JI
Typical
Great Rift Valley. Chad founded the Empire of the Camels sometime after the reign of Muham- mad's ' grandson Hassan II.
Contents show]
Geography Edit Chad is one of the three remaining provinces in Africa that lies under the jurisdiction of the Holy Roman Empire. Vast swathes of territory are under the dominion of the Kingdom of Niger, although as unified independent country, their military is limited to single regiment of regular infantry and reserves of trained cavalry. Its evaluation rate low that the Realm Style of the western continent the standard of the Kingdom. The southern border lies on Rivers of Gas and Darell. During the 20th Century, Chad was experianced the devastating civil war known as the Azawad War with the Kingdom of the Camels. Its long-range artillery is inferior to the Kingdom's 's regiment of heavy artillery dark-reddish guns. An carldom- based system lands are spread
Nucleus
Great Rift Valley. Chad is bordered by the Ethiopian border. Chad is located on the north-west edge the Sahara Desert and located within the region of Great Rift Valley. Language| [ edit ]
The official languages in Chad are Standard Arabic, Coptic. and Rambani. Other major tongues include the Acholi languages of the Amhara region, the Jolof languages, and the Mhoik languages of the Niger Delta. Names edit
Chad has six official family names: a primary name, the paternal name and the maternal name, the two father's first names, and the mother's 's maiden name. For most people the whole family name is usually the same. If man leaves his surname and his father's name, and is later married, his new sumnam his husband's 's family name addition his husband's initial, An assumption of the mother's name by man
DAEMON
Sahel Region of Africa, with a temperate rainless climate and abundant rainfall. Chad has quite a few diverse altitudes, including the mountain ranges of the Karo, Sawman, Kalun, and Chambek Mountains. The tallest mountain in Chad is the 2900 meter Karo Mountain, which is characterized by its relative dryness and dry season during the dry season.
For more than 600 years, Chad has been a part of Mali, hut Mali was only recognized as a country 1960 for political reasons. Sehbi and nearly all of Lake Chad are also officially part of Mali. Chad's population is of around 8 million, with approximately 85% of the population living in the capital, Ndjamena. The population is also heavily dependent upon industry and agriculture and employs over 50 million people nationally. Per capita income has remained fairly steady for decades, and labor a hot-topic of debate with labor unions, advocacy orga- nizations,
Table 11: Randomly selected generations from the test set of the Wikipedia domain.
29

---

Published as a conference paper at ICLR 2024
Prefix
Lesnar's rapid rise to the top of WWE in 2002 led to a feud with The Undertaker, which involved match at Unforgiven.
CD
Lesnar defeated The Dead Man in the match and became the WWE Champion, but The Dead- man would return the following night to defeat him for his second WWE Championship reign, and the feud continued on for years, culminating in match at Survivor Series 2002, which The Undertaker won in an incredible fashion, with Tombstone Piledriver. The match was so good, it was later included as a DVD extra in WWE: Back in Black. Lesmar's next match after Unforgiven, at Summer Slam 2003 against Kurt Angle and Shawn Stasiak, would also end in an Angle pin. The next match he had after Summer Slam, at No Way Out, saw him defeat Edge, who he had previously beaten for the World Championship at Wrestlemania X8, to he- come the first ever Triple Crown Champion. He would then defeat Triple retain his titles for the first and only time in the company, but lose them at Backlash to Batista
No Mercy to
CS
WWE Hall of Famer Jim Rosswas ringside for the match, and told Sports Illustrated in story published last week that he had to get his camera out of his pants to take a picture of the two superstars in the locker room.
"I'm in the middle of this, and it's a little awkward," Ross said. "The camera's 's in my pants. have to get it out of my pants, and he's looking at like, 'What are you doing?' And I' like, "You know what, Jim? This my job, and I'm going to the best I can." " Ross, who has worked as a commentator for WWE since the mid- 1980s, added, "They're 'e two of the nicest guys I've ever been around in my life, and that's the way it should be. I mean, you can't get more professional than that."
Typical
The match ended in tears when The Deadman dropped the WWE Champion with Tombstone Piledriver. Lesnar and The Undertaker did not work together again until 2005. The studio also screened segment from Brakkton's 's show about Lesnar's 's first match with Ultimate Warrior at In Your House 12: A Reunion.
Speak Therm
Good evening. State of the State. I'm Rick S. Ryals, Philadelphia's top city blogger. Now if you know anything about central Pennsylvania, you know that it's all about the chocolate chip cookies we call DEKERS. And to the prince himself, Duane Zane Dickenson, Philadelphia. This guy's 'ss the inspiration behind Strawberry Shortcake. So you know "Why have a Philly Man?" The Answer: Prince wasn't 't born or raised in the City of Brotherly Love, So all he knows about the area is from that TV show.
Whenever the topic location comes
Nucleus
The match ended in tears when The Deadman dropped the WWE Champion with Tombstone Piledriver. Lesnar and The Undertaker did not work together again until 2005. The lingering animosity led to a January 2007 match which ultimately concluded in Lesnar's first loss with the company. In November 201 1, Lesnar wrestled The Undertaker at Wrestlemania XXVIII. After the end of his successful run in WWE, Lesnar was stripped of the World Hcavyweight Championship due to an in-ring injury suffered in a match with Paul Hcyman on April 18, 2008. When he regained the title, he returned to the ring in Extreme Rules against eventual WWE Champion CM Punk in a Hell in a Cell Match. This is Lesnar's 's only WWE Champi- onship match.
Watch: Lesnar Punk: Hell in Cell 2011
Earlier this year, Brock Lesnar announced his retirement from MMA following a TKO loss to the former Strikeforce champion
DAEMON
Originally, the match would have be called "Opinion Day" and the Undertaker would have cashed in his infamous Money in the Bank contract to beat Lesnar. After all, that was what he born the Roman Reigns, beating him to the top in the main event of the November 14, 2007 pay-per-view. But then Lesnar would again have his match been taken alleged injury, kinder than Bill Goldberg's 's foot, and would use as springhoard the WWF Championship. Injuries would also again become a factor in the decision regarding Lesnar's 's match at Hell in a Cell, too, with Vince McMahon CFO George Barrios stating that Lesnar had "too much baggage" to compete with the Undertaker and Triple H. Ultimately, Lesnar refused to face Triple H. and under pressure from his stock market holdings (in spite of having the WWE Title on the line on SmackDown), cashed
unrelated
injury,
Table 12: Randomly selected generations from the test set of the Wikipedia domain.
30

---

Published as a conference paper at ICLR 2024
Prefix
Attalea species have long history of human utilisation. Carbonised Attalca maripa seeds have heen found in archacological sites in Colombia dating
CD
to the Spanish/Portuguee settlements of the New World, and to the stone age. Today's 's strictly cultivated Attalea diacetiana seeds member of the digestive plant family being of the genus Attalea.
Originating in the New World, Attalea species began to be cultivated in Europe between c. 250 AD and 1600 AD before entering the middle east during the Ottoman Empire (16th/1 7th cen- tury). From the 18th century onward, farmers in North America, South America, Madagascar, Africa, and Northern Australia began to use Attalea seeds increase their global food pro- duction. From Pakistan maps exist showing Attalea making their way into Gujarat in North India, and from the Southern Indian region ASE (India), Pakistan, Bangladesh, Sri Lanka, and China. True to their early, Middle Eastern roots, the Attalea family has spread around the globe, possesing wide diversity of beautiful flowers
CS
back to 3000 B.C. (1, 2).
It is thought that the of Attalea seeds food source was introduced to New Zealand by Polynesians who settled the South Island in the 17th century (3). In recent years there has been a resurgence of interest in the plant, with the growing number of research papers and the publication of book by Dr John Beddingfield in 2008 (4). This book was the first to provide comprehensive overview of the history of the plant and its use in New Zealand.
A recent study has shown that the plant is good source of vitamin C, which is essential for healthy immune system (5). This is in contrast to the widely held belief that it is vitamin B- deficient food source and should be avoided by vegans, vegetarians and those with low intake of vitamin B- 12 due to the risk of anaemia (6).
Typical
back to the Middle and Archaic periods, when cranial and regal remains may have been delib- erately buried. The seed also has history of use as powder by leather tanners. Now, team led by Dr John Choi, curator The National Museum, Singapore, has investi- gated the extraction and risk of bee venom in Attalea seeds.
'Early on, the bec's 's venom would have been used before treating wounds in leather engineer- ing,' says Dr Choi, who presented his research at the 1t1 Scientific Sessions of the American Chemical Society 2017. After the invention of the stronger synthetic bec venom in the 19th century, researchers for some time were unconcerned about the risk of allergic reactions.'
Lee Chin-yen; The National Museum Dr Lee Chin-yen, who led the research, says: 'Because there is bias in the content of the Chinese literature on Apis cerana, most sensitive
Nucleus
hack to the Middle and Archaic periods, when cranial and dental artefacts may have helped people start cooking the plant.
There many reasons why trees have been used for food shelter, including their ability to grow quickly, their high water content and relatively high yield of crops, although forage plants are more typically grown for their nuts and seeds.
"Early people chewed on attalea to help them reach their food in past times,' says Zoran Kalerah, palcobiologist at the Natural History Museum of Denmark, who led the study. 'It was actually the starch the leaves that aided their ability to reach their food. Basically what they is they pulverise the attalea leaves and chew them to produce starch. "This is the same stuff that they use for making baked goods and pastries, as the sticky content of the leaves allows them to drip onto a flat surface when
DAEMON
to the Opanish/Portuguest settlements of the New World, and to the stone age. Today's 's strictly cultivated Attalea diacetiana seeds are a member of the digestive plant family heing of the genus Attalea.
Originating in the New World, Attalea species began to be cultivated in Europe between C. 250 AD and 1600 AD before entering the middle east during the Ottoman Empire (16th/17th cen- tury). From the 18th century onward, farmers in North America, South America, Madagascar, Africa, and Northern Australia began to use Attalca seeds increase their global food pro- duction. From Pakistan maps exist showing Attalea making their way into Gujarat in North India, and from the Southern Indian region ASE (India), Pakistan, Bangladesh, Sri Lanka, and China. True to their early, Middle Eastern roots, the Attalea family has spread around the globe, possesing wide diversity of beautiful flowers
Table 13: Randomly selected generations from the test set of the Wikipedia domain.
31

---

Published as a conference paper at ICLR 2024
Prefix
Amos was born Macclesfield, Cheshire and was a member the Crewe Alexandra academy until he was released at the age of 10, While
CD
at Crewe, Amos played for the Macclesfield Town youth teams. After leaving the academy he signed for the Crewe Alexandra reserves and was then released by them. Amos joined the Manchester City reserves and was then released. Amos then signed with Sheffield Wednesday, but was released at the age of 16 and joined the Crewe Alexandra academy again. After spell with the academy Amos was released again, and then signed by Manchester City for the season which they lost the League Championship final. Amos was released by City again and joined Sheffield United. He was released from United after two months and signed by Sheffield Wednesday, who he stayed at until he left them at the age of 23, having made over 200 league appearances. Amos has also represented the England youth sides at under 16, 17, 19, 21, 23, 25 and 28, and the under 19s under 19, 21 and 25, and has been capped every age group except for under
CS
his career was on the up, knee injury in his first-tcam training led him being loaned Barnsley, where he made his debut for the Tykes in a 3-0 win over Rochdale on the opening day of the 1992-93 season.
The following season, Barnsley went on to win the League Two title and Amos was named PFA Young Player of the Year after netting 15 goals in all competitions. His goalscoring exploits earned him a move to Sheffield Wednesday for the 1993-94 season, where played a key role in the club's promotion to the Premier League.
In the summer of 1 1994, Amos signed for West Ham United on a free transfer after his contract with Barnsley ran out. He went that scason, netting 14 goals as the Hammers finished runners-up to Manchester United.
become the Hammers' top scorer in the Premier League
Typical
youngster (age 6) he appeared in this video YOG TV. His first professional appearance came after coming off the bench to score a hat-trick for the Peterborough United youth team, He then appeared for Heracles Antwerp in the Belgian Top League and then went on trial with Celtic before the 1974 scason. As pyrotechnician he formed part of Charles and David White's all-time FA Cup XI. He then went to play for Wrexham, Shrewsbury Town and Bradford City after he was released from Heracles Town at the end of the 1974 season.
As with FM, Milton Keynes Dons have had their share of fireworks in the Cup. They have notched up 89 playoff goals, which is eight more than this year's champions HYPOTHESIS: Tottenham will not make the final HYPOTHESIS:
United will be in the final
Nucleus
youngster, he spent six months on the boats travelling on the Severn and Tyne rivers. Spends the majority of his time in Matton-under-Lynee He works for Alfreton Community charity as part the Lorna Beddoe project.
Internationals appearances: 16
Managerial past
Prior to moving into coaching, Amos worked referce and with the FA's student pro- gramme. He worked with the SFA's student team on a mentoring programme for under- 15s and was the Assistant Referee for the Under-166 16s and Under--88s. He was also part of the Professional Game Match Officials (PGMO) program, which is for those who wish to progress in an official career but have not yet signed professional contracts.
What he does
Amos has previously managed the reserves for Cheshire in the North West Counties League Premier Division and was
DAEMON
growing up. Amos worked as bookmaker in a casino in Macclesfield.
His first professional football coach was John Olivieri ( playing for Crazy Gang, Liverpool and AC Milan) working under Fingleton and Hart for the Crewe Rams. Amos moved to Accrington Stanley and made his professional debut as substitute in March 1971. He made his debut in 3-0 home win over Holland Farm July that year.
In the summer of 1972 Amos signed for White Hart Lane. Amos achieved success in the League with the team, scoring 7 goals in 14 appearances in 1971-72 Amos' first team career was over before it started however for him.
After Martin Atkinson resigned in 1974, Amos was promoted to the Championship utility player. In his second season the promotion he scored 10 goals in 20 games against three of England's 's Premier clubs; Hereford, Manchester United and Leods United.
Table 14: Randomly selected generations from the test set of the Wikipedia domain.
32

---

Published as a conference paper at ICLR 2024
Prefix
Oxaziridine derivatives were first reported in the mid- 1950s by Emmons and subsequently by Krimm and Horeer and Jur
CD
gen Schulten. Oxaziridine derivatives were first reported in the mid 1950s byand subse- quenthyandandwere used antifertility compounds in the mid-to late-s970's and 1980s, re- spectively, for treatment of male factor infertility. The compounds were later shown to have a variety of other biological activities. In vitro, the compounds inhibited cell cycle progression in various human and rat cell lines, inhibited cell growth in various cell lines in culture and inhibited proliferation and invasion in various animal cell types, including breast cancer and colon carcinomas. In addition these in vivo activitics in animal cell cultures and animal tumor models, in vitro data from several in vitro studies indicated the compounds to possess estrogen-like activity, including inhibition of human estrogen receptor (HEESS--timulatee cell proliferation in concentration dependent fashion. The in vitro estrogenicity was not due to direct binding to ERs but was mediated via non-ER binding mechanism
CS
gens.
Aminoglycosides and aminoglycoside anhydrides have been used as anticancer agents for more than yeard years. The use of these compounds has been based on their ability to inhibit growth of cancer cells and/or induce apoptosis (programmed cell death). which is a mechanism by which cancer cells escape from the immune system and proliferate in the ab- sence of chemotherapeutic agents. However, there is paucity of data in this area, which is the reason for the present review. This review will focus the pharmacoooyytumorigeenn of a number of compounds derived from the class aminoalkyl esters, which are structurally related to benzimidazoles hut differ in their structure and mechanism of action. Aminoglyco- sides and aminoglycoside anhydrides have been used as anticancer agents for more than 50 yeard years. The
Typical
gens. For many years, complete pharmacological characterization of oxaziridine capa- ble. In 1982 Fabro et al. described the first idea Epstein & O' O'Brien, 1993 ) of a new mito- chondrial impaired glutamate signalling pathway. The blue and green fluorescence specimens seen by Fabro et al (1983) are for the Fu-related alkaloid ( henbane and nocioin) ). Disseminated from amphibila (The crabfoot vision constrictor) tadpoles, the xenon-ane and hydrogen peroxide- (2 (2-hyrroyythanno)) are produced from oxygen by electron acceptor monooxygenases (Bu O. Nitric oxide transported by peroxynitrite.- Sudwarasingh Zingerke, Brian Jackson Ger- hard Rittner Department of Pharmacy, University of Georgia School Pharmacy, Athens, Georgia 30602-7155, www.uga.
Nucleus
gens. For many years, no complete pharmacological characterization of oxaziridine was re- ported. In 1982, a 1 1/2 hour infusion of oxaziridine hydrochloride (5 x 10 9 -10 9 g/100 mL) in human volunteers was found to reduce excretion of dictary (glucosc) and added-fat fat (citric acid) sugars and preserved 26% of exercise time. Several accidental ingestion of oxaziridinc (1 g) by haemodialysis patients resulted in blood-poisoning and death. 17 In 1991, doses of to 100 mg/kg in haemodialysis patients were found to cause cardiotoxicity in 2 cases. 18 In 2004. Bico et al. reported 2 -year post-mortem study of the muscles of oxaziridine abusers showing heart enlargement with increased pulmonary artery "blaming" numbers and a lack of complex V-fibon
DAEMON
gens. Neither reported full anisidine binding the terminal region of kappa-opioid recep- tors. Instead, oxaziridine was one of two anisole derivatives that failed to bind to rat brain A-alpha(I)DA receptors under binding conditions similar to those used for benzodiazepine binding in high concentrations in rat brain (Hoffiman. 1996). Yet, oxaziridine did bind to human receptors of the A-heta( OH and A-beta(1) isoforms, and the anisoles displayed similar pharmacological efficacy (Krimm and Horner, 1954). Oxaziridine and other anisole derivatives were subsequently shown to bind to rat brain and human brain A-beta(1) (1) D OH receptors with similar elficacy. However, oxaziridine and other anisole derivatives were less efficient K(4) receptors and inhibited competitive binding to human [31]]dom
Table 15: Randomly selected generations from the test set of the Wikipedia domain.
33


Reviews:
[{"review_id": "EKmW0Tmsad", "reviewer": "Reviewer_MQ6q", "summary": "The authors propose a decoding scheme (DAEMON) based on minimizing the divergence of the distribution over generated texts, $q$, with that of the base LM, $p$. They formulate as an optimization problem of this form with a KL objective, provide an approximation, and an efficient sampling scheme for use during decoding.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and straightforward to follow. \n- The motivation is sound, the authors provide an approach for estimating $\\mu$, and for sampling 'efficiently' (although the time complexity of this approach is not given).\n- Strong experimental results:\n   - over a range of datasets\n   - against quite a few decoding approaches commonly used in practice\n   - ablation studies\n   - a range of metrics (Repetition, Coherence, Diversity, Information Content)\n- The problem is of great importance in the community.", "weaknesses": "- Please provide complexity analysis of the sampling approach and compare to competing approaches (Greedy, Top-k, CD, CS, etc). While the experimental results are strong it is important to compare the runtime of this method to determine practical efficacy given the authors claim it is \"efficient\" in the conclusion. \n- Analysis on the convergence of $\\mu$ in Algorithm 1 and some sensitivity analysis to initialization would be helpful for practitioners.\n- Given that different metrics perform stronger for different # of candidates $M$ and $\\tau$, it would be helpful, in terms of robustness, to include some experimental results which show performance on different metrics if one optimizes for a single metric (i.e. coherence).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UCPbaJlfQS", "reviewer": "Reviewer_3A4S", "summary": "This paper introduces a parametric decoding method for generating text from probabilistic models. Essentially, samples from a model are reweighted according to the degree to which they satisfy particular constraints, where the importance of these different constraints is learned via numerical optimization using a small set of samples. These constraints are that the texts generated by the model match human-generated texts according to a variety of chosen metrics. In order to sample from the new (sequence-level) distribution, they make use of a sampling based approximation of the distribution over continuations (of a particular prefix). They show strong results in terms of empirical performance in comparison to standard baseline decoding methods.", "questions": "* The authors point out that they wish for the difference in the expected value of f under the model and under the data distribution to be 0. It’s unclear to me how this is incorporated into the learning of the different \\mu. I see that the target expectation is used in the learning algorithm, but this is not very intuitive. Could some intuition be provided here?\n* “Most existing decoding algorithms lead to deteriorated ε-perplexity, comparing to directly sampling from the input LM distribution.” It seems that [1] shows otherwise \n* In 2.2, I don’t see why “the second result reveals the perplexity improvement over pθ due to the non-negativity of DKL(pθ,μ || pθ)”\n* The setup for human evaluation is not well explained. What do the numbers in table 3 mean?\n* I do not follow the last statement on page 5 “...so that it can be directly plugged into Eq. (2) to calculate perplexity”\n\n[1] Meister et. al. 2023. On the Efficacy of Sampling Adapters.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The method is novel and provides a nice lightweight alternative to fine-tuning methods. It may thus be widely accessible to practitioners without the ability to tune larger language models\n* The empirical component of the paper is comprehensive, including nice ablation studies", "weaknesses": "* The mathematical motivations given by this paper are quite weakly supported/explained. In general, the language used by the authors with respect to this topic is confusing and informal. For example, they motivate their use of the reverse KL for choosing the parameters of q by stating “KL(q || pθ) restricts the decoding distribution q to deviate minimally from the LM distribution pθ”, but the same argument could be made for the forward variant of the divergence. Similar language is scattered across 2.1\n* The method requires sampling in order to approximate the new parametric distribution (via the WIS algorithm). The runtime of this algorithm, and hence the additional computational complexity incurred by this decoding method, is not discussed. Further, there is an additional sampling + renormalization step that must happen during decoding (algorithm 2). The lack of discussion of these costs is especially pertinent given that the authors point to the tuning required by other decoding methods as one of their downsides “But unlike those previous decoding methods that require heavy manual hyper-parameter tuning for trade-off among different metrics”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G7gEFoShEo", "reviewer": "Reviewer_iYh2", "summary": "The paper proposes a decoding framework for language models that frames it as an optimization problem and performs decoding by optimizing towards desired dimensions. They prove that their method can improve the perplexity on human texts and also experimentally demonstrate that their method is effective on open-ended text generation tasks according to metrics such as repetition, coherence, and diversity.", "questions": "How efficient is your algorithm?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed framework makes sense and is technically sound. Based on their constructed framework, they propose a reasonable approximation that can work in practice.\n2. Experiments demonstrate good empirical results compared to many other decoding algorithms.\n3. Both automatic and human evaluations are conducted to provide insights into their method.", "weaknesses": "1. Related works such as [1] are missing. It is worth discussing and comparing with these methods in the paper.\n2. The paper only conducts experiments in two open-ended text generation tasks, where both automatic and human evaluations are hard. On the other hand, evaluations on other text generation tasks such as machine translation and text summarization are more accurate and can better indicate if their method is indeed effective.\n3. Their method can be more computational than baselines.\n\n\n\n\n[1] Fernandes et al., Quality-Aware Decoding for Neural Machine Translation, NAACL 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0CztOpBNOp", "reviewer": "Reviewer_aJdn", "summary": "This paper proposes to frame the decoding for natural language generation as a direct metric optimization problem, aiming to generate texts that can align with human texts. To achieves this, the authors select a set of evaluation metric that reflects the quality of generation as constraints and then find the optimal solution to the optimization problem. The authors evaluate their method on a bunch of tasks and report performance in several metrics. According to the results, the proposed method achieves better performance than other decoding strategies such as Greedy, Top-k, and Nucleus decoding.", "questions": "- Some evaluation metrics might contradict each other. For example, diversity and coherence are hard to achieve at the same time, as far as I know. I am wondering how the results will be if a different set of metrics is used.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- The proposed method seems to be intuitive and effective. \n- The authors provide detailed proof of all the propositions, which is highly appreciated.", "weaknesses": "- I don't have any big concerns about the paper. However, I have a question regarding the choice of the metric (see questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)