Beyond MLE: Convex Learning for Text Generation
Authors: Chenze Shao
Zhengrui Ma
Min Zhang
Yang Feng
Venue: NeurIPS 2023 poster (2023)
Keywords: Maximum likelihood estimation
Convex function
Text generation
TL;DR: Convex learning enables text generation models to focus on generating highly probable outputs without having to estimate the entire data distribution.

Abstract: Maximum likelihood estimation (MLE) is a statistical method used to estimate the parameters of a probability distribution that best explain the observed data. In the context of text generation, MLE is often used to train generative language models, which can then be used to generate new text. However, we argue that MLE is not always necessary and optimal, especially for closed-ended text generation tasks like machine translation. In these tasks, the goal of model is to generate the most appropriate response, which does not necessarily require it to estimate the entire data distribution with MLE. To this end, we propose a novel class of training objectives based on convex functions, which enables text generation models to focus on highly probable outputs without having to estimate the entire data distribution. We investigate the theoretical properties of the optimal predicted distribution when applying convex functions to the loss, demonstrating that convex functions can sharpen the optimal distribution, thereby enabling the model to better capture outputs with high probabilities. Experiments on various text generation tasks and models show the effectiveness of our approach. It enables autoregressive models to bridge the gap between greedy and beam search, and facilitates the learning of non-autoregressive models with a maximum improvement of 9+ BLEU points. Moreover, our approach also exhibits significant impact on large language models (LLMs), substantially enhancing their generative capability on various tasks. Source code is available at \url{https://github.com/ictnlp/Convex-Learning}.

Beyond MLE: Convex Learning for Text Generation
Chenze Shao" Zhengrui Ma²1.2 Min Zhang & Yang Feng" 1.2
Key Laboratory of Intelligent Information Processing
Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences
School of Future Science and Engineering, Soochow University chenzzeshaottenceett com, mazhengrui 21b cetiictaae ac. zhangminmt Shotma com, Fengyang lict. ac. cn
Abstract
Maximum likelihood estimation (MLE) is statistical method used to estimate the parameters of a probability distribution that best explain the observed data. In the context of text generation, MLE is often used to train generative language models, which can then be used to generate new text. However, we argue that MLLE is not always necessary and optimal, especially for closed-ended text generation tasks like machine translation. the most appropriate response, which does not necessarily require it to estimate the entire data distribution with MILE. To this end, we propose a novel class of training objectives based on convex functions, which enables text generation models to focus on highly probable outputs without having to estimate the entire data distribution. We investigate the theoretical properties of the optimal predicted distribution when applying convex functions to the loss, demonstrating that convex functions sharpen the optimal distribution, thereby enabling the model to better capture outputs with high probabilities. Experiments on various text generation tasks and models show the effectiveness of our approach. It enables autoregressive models to bridge the gap between greedy and beam search, and facilitates the learning of non-autoregressive models with maximum improvement of 9+ BLEU points. Moreover, our approach also exhibits significant impact on large language models (LLMs)) substantially enhancing their generative capability on various tasks.
these tasks, the goal of model is to generate
Source code is available at /estts::/gitaae wtt. Cwaccctnlp//ooneeee Caa.appp///ooeeeLeaannnng
Introduction
Text generation an important field within natural language processing that aims to generate human-like texts for specific tasks. It can be broadly divided into two categories: open-ended and closed-ended text generation. Open-ended tasks encourage the model produce novel and diverse outputs without specific expected outcome or structure. Representative tasks in this category include language modeling [4] 77, chatbot (64), storytelling [13), etc. contrast, closed-ended tasks are more constrained and adhere to specific rules or formats. Representative tasks in this category include
machine translation {8] 22 text summarization [45). etc.
In recent years, learning neural probabilistic models with maximum likelihood estimation has become the dominant approach for both open-ended and closed-ended text generation (5 2 Maximum likelihood estimation (MLE) is a statistical method used to estimate the parameters of probability distribution that maximize the likelihood of the observed data [33]. Since directly maximizing the
"Equal contribution. Order determined by coin flip. Cooresponding author: Yang Feng
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

likelihood can be numerically unstable, it is common to minimize the negative log- likelihood loss function, which is also referred to as cross-entropy loss. It is equivalent to minimizing Kullback- Leibler (KI.) divergence [25 I between the true distribution and the predicted distribution, which ensures that the optimal predicted distribution is the true data distribution.
While MLE has gained widespread adoption, it does not always align with the objective of text generation, especially for closed-ended text generation tasks such as translation and summarization. In these tasks, the goal of the model is to generate the most appropriate response, rather than producing diverse outputs. For example, in the task of machine translation, though there may exist multiple translations for the same input sentence, we usually want the most accurate and commonly used translation result. Generally speaking, the desired output can be mathematically defined as the output with the maximum probability in the true data distribution, which does not necessarily require the model to estimate the entire data distribution with MLE.
In terms of generating the most probable output, MLE is also suboptimal for current neural text generation models. For autoregressive models, even if the model can perfectly fit the data distribution, it still requires decoding algorithms like greedy or beam search to generate the output, which do not guarantee the exact result with the maximum probability. To our knowledge, only Stahlberg and Byrne (55) proposed an exact decoding algorithm for autoregressive models, but it is too slow for practical applications. The limitation in exact decoding can be overcome by non-aut aon-auorregresive models [1]] which independently predict the output at each position. However, fitting the data distribution by MLE is theoretically beyond the ability of non -on-eutoreerrssvve models (T]. In light of these issues, alternative training objectives should be considered to better address the specific requirements of text generation without incurring the shortcomings associated with MILE.
Based on the analysis above, MLE is suboptimal that it trains the model to estimate the data distribution, which complicates the training and decoding of text generation models. It would be advantageous if the model could converge to a sharper optimal distribution under an alternative loss function, as this would enable autoregressive models easily find high probability outputs and also allow non-autoregreesive models to converge to a better distribution. Exploring loss functions with this characteristic could lead to improved performance and efficiency of neural text generation models, particularly for closed-ended tasks.
In this paper, we propose novel class of training objectives based convex functions, which help text generation models capture highly likely outputs without estimating the entire data distribution. Intuitively, the concave shape of prgorbbabbiltyy discourages the model from assigning large predic- tion probability to a single sample, as the marginal benefit diminishes with increasing probability. If the learning criterion is convex less concave, then intuitively the model would converge to sharper distribution, which is the motivation of this work. We further investigate the theoretical properties of the optimal predicted distribution when applying convex functions to the loss. Our findings demonstrate that convex functions can sharpen the optimal distribution, allowing the model to better capture outputs with high probabilities.
Experiments on various closed-ended text generation tasks and models show the effectiveness of our approach. Specifically, it enables autoregressive models to bridge the gap between greedy and beam search, and facilitates the learning of non-autoregressive models with a maximum improvement of 9+ BLEU points. Moreover, our approach also exhibits significant impact on large language models, substantially enhancing their generative capability on various tasks.
2 Preliminaries
2.1 Maximum Likelihood Estimation
Maximum likelihood estimation (MLE) is a statistical method used to estimate the parameters of probability distribution that best explain the observed data. This is achieved by maximizing a likelihood function so that the observed data is most probable. Since directly maximizing the likelihood can be numerically unstable, it common to minimize the negative likeliield loss function, also referred to as cross- cross-entropy loss. Given the data distribution Pdato and a parametric model with parameters 0, MLE training minimizes:
(M) -Ex (a)) [log po (.)).
(I)
2

---

MLE can be viewed as an attempt to minimize KL divergence between the true underlying distribution of the data Piito and the estimated distribution pe provided by the model [T]. The following equation reveals the relationship between MLE loss and KI. divergence:
Pdata (x) (M) Hantas
k (Pdatall |I po) Σ Pdaata (ar) log
po(
where Hata is the Shannon entropy of the data distribution, which remains constant with respect to the model parameter 0. Therefore, the MLE loss and KL divergence share the same minimizer that the estimated distribution pe equals to the true distribution Pdata- By minimizing the MLE loss, the predicted distribution encouraged to be as close as possible to the true data distribution. In the context of text generation, this ensures that the model learns to generate text that closely resembles the text in the training data.
The above discussion can be extended to conditional scenarios. In such cases, the log- logeliicl loss can be expressed as:
LM.LF(0) = - Be--pseta (c) (errpeaaaa (ir e) [log po (x([c)]]
where e represents the input context. This extension allows the MLE framework to accommodate wide range text generation tasks such as machine translation, summarization, dialogue system, etc.
2.2 Text Generation Models
Based on how the sequence probability is factorized, neural text generation models can be broadly categorized into two types: autoregressive (AR) models and non-autoregressive (NAR) models. Au- loregressive models generate text sequentially, predicting one token at time based on the previously generated tokens. In AR models, the probability of generating sequence x (21, Z2 TT) is factorized as:
T
(=(c)
II pe (Z)) t=1
(4)
where represents the input context. With the autoregressive decomposition, AR models can perfectly fit the data distribution if satisfies p0 [ c) = Pdtta (Ig c) for every I; c, f. In inference, AR models can perform deterministic decoding like greedy/beam search to generate a high probability output, or sample from the model distribution to generate diverse outputs.
In contrast to autoregressive models, non-autoregressive models 17, 14 generate text in parallel, predicting all tokens simultaneously without conditioning on previously generated tokens. This approach can significantly speed up the generation process, it removes the sequential dependency between tokens. NAR models, the generation probability is factorized as:
Po(xlc) = II (x(l/)))
(5)
Unlike AR models, NAR models can efficiently find the most likely output by using argmax decoding at each step. However, MLE beyond the ability of NAR models since they are theoretically unable to fit the data distribution. Huang et al. [21] showed that KL divergence from po to Pdata is bounded by a non-negative constant:
DKL (Pdatal) pa) 2 C - (x/c) + Σ Hdooaa (F
(6)
The MLE loss is minimized when NAR models achieve the equality by ignoring sequential depen- dency and predicting po = Pauta ( Therefore, NAR models trained with MLE often suffer from reduced performance, as they lack the ability to model dependencies hetween tokens.
Approach
In this section, we will explore alternative loss functions for the learning of text generation models, which overcomes the limitations of MLE. We begin by introducing general learning framework that allows arbitrary loss functions. Next, we discuss the benefits of applying convex functions to the loss within this framework. Finally, we use convex functions to construct composite loss functions, which can be used in practical text generation scenarios.
3

---

3.1 General Learning Framework
For simplicity of notation, we omit condition in the probabilities, with the data distribution represented as Paata (ar) and the model predicting the distribution po(x). The derived theoretical results hold in both unconditional and conditional settings.
First, we introduce the general learning framework for text generation, characterized by the following loss function:
() - - (2)) (f(o
where f an arbitrary function of the prediction probability po(z). We impose some basic re- quirements on f: (1) The domain of function f should contain the interval (0. 1]; (2) f must be differentiable on the interval (0, 1] since we need to compute its gradient; and (3) [ should be an (0, 1] to encourage the model to generate the current sample. Under this framework, we can explain maximum likelihood estimation as special case of f log, which is a differentiable and increasing function within the interval (0, 1j. We also establish some reasonable assumptions:
increasing function
Assumption (Countability of Sample Space). The sample space X is countable, which allows us to enumerate all samples in systematic way. Note that [X] can be either finite or infinite.
Assumption 2 (Distinctness of Sample Probabilities). In the data distribution Pdoto- the probabilities of all samples are distinct, which allows us to arrange samples in a strietly descending order of sample probabilities
Assumption D naturally holds in text generation tasks due to the inherent discreteness of textual data. With countable sample space and probabilities lying in dense subspace of real number, it is reasonable to assume the distinctness of sample probabilities. While Assumption Lis not strictly necessary, removing it would introduce many corner cases that would complicate the subsequent analysis. In the following, we will assume that Assumptions 1-2 always hold, and we arrange the samples such that Pdata (21) Pdata (22) > ... Pdata (x:) > .... Since the sample space X' is
countable, the loss function in Equation 7 can be reformulated as follows:
(0) = Σ
Pdata ((p((::))
In this framework, our primary focus is to analyze the probability distribution pe that the model is inclined to predict when the loss function is Ly. We use Py to denote the optimal distribution that minimizes the loss Ly. which represents the expected outcome of the model. If & has multiple optimal distributions, we use py to denote an arbitrary optimal distribution. This choice does not harm the generality of our analysis, as the subsequent discussion is applicable all optimal distributions. Currently, it is only established that the optimal distribution for the MLE loss Liog is the data distribution Plog = Pdota- For other loss functions, the following theorem reveals a general property of the optimal distribution. With samples organized in descending order of their probabilities in the data distribution, i.e., Pdata (I1) ) > Pdata (x²2) > ... > Pdata (x;) > ..., the optimal distribution of an arbitrary function f maintains this order as ( ) 2 py pf(22) 2.... ... 2 Pf(1;) 2 ..... The proofs
for the theorems presented in this paper can be found in Appendix A
Theorem 1. Given an arbitrary differentiable and increasing function f. the optimal distribution Pj satisfies (x)) 2 (x2) 2 ... (x:) 2
In the following, we will further explore the properties of optimal distributions associated with specific loss functions.
3.2 Loss with Convex Function
In certain text generation scenarios that require precise and deterministic outputs, it is beneficial for the model to converge to an optimal distribution that is sharper than the data distribution. In
'When X' is countably infinite, an arbitrary sequence of sample probabilities forms a convergent series since their sum is 1. This guarantees the existence of maximum point in the series, ensuring that the sample probabilities can be arranged in strictly descending order.

---

this section, we demonstrate that this objective can be achieved by employing convex functions as learning criterion.
The MLE loss function is based on log-probability, which is a concave function whose gradient decreases the probability increases. The concave shape of the learning criterion prevents the model from assigning a large prediction probability to single sample, since the marginal benefit diminishes as the probability increases. If function f is convex, then intuitively the model would converge to sharper distribution. The following theorem validates this intuition, which shows that the optimal distribution ps is a one-hot distribution when f is convex.
Theorem 2. f is an increasing convex function on [0, 1]. then the optimal distribution pf is one-hol distribution that py (x1) ) = and py (x:) = 0, > 1.
The one-hot characteristic of the optimal distribution advantageous for text generation models seeking precise and deterministic outputs. For autoregressive models, they do not need the computa- tionally expensive beam search decoding any more if the model distribution is nearly one-hot. For non-autoregressivve models, they suffer from reduced performance under MLE due to their inability to fit the data distribution. However, fitting a one-hot optimal distribution well within their capabilities, allowing these models to generate high-quality outputs.
However, the direct application of loss with convex functions in training text generation models comes with an inherent limitation, impeding its practical utility. Specifically, the gradient of the parameter 0 tends be very small when the prediction probability approaches thereby rendering the training process inefficient. The gradient of 0 can be formulated as follows:
(C₂(0) 00
@pp(x)
Pa)ce (m)
00
T
(o po(
0e
where we have omitted the autoregressive history condition of (x)))) for simplicity. The equation above indicates that the gradient is proportional to the sentence probability po()). In text generation models, the sentence probability po(x) is the product of token probabilities p0(2)), which causes (o( to be typically close to 0, especially when the model is newly initialized.
To counter this effect, the gradient (x)) would need to approach infinity as (o(r) approaches For instance, the proorbbbbyyltyy function has the gradient 1 po(z)* which offsets the impact of (a)) such that ('(p (x)) (x)) = 1. However, for an increasing convex function (pe(z)) whose gradient increases with p=(z), its gradient must be bounded when pe approaches 0. leading to an extremely small gradient update for the parameter 0 during training. This inherent limitation of loss with convex functions poses a significant hurdle to their practical applications.
3.3 Loss with Convex-composition Function
3.3.1 Theoretical Analysis
In the preceding discussion, we illustrate that while convex functions can induce desirable one- hot optimal distribution, their inherent limitations during training pose significant impediments to practical applications. Consequently, we consider relaxation of the convexity requirement, with the objective of rendering the function f less concave. This approach aims to obtain an optimal distribution that is sharper than Pdata. thereby providing a practical solution that augments model performance without sacrificing training feasibility.
The standard loss function in maximum likelihood estimation is the negative hog-robabblity, where log- pegbabbbability is a concave function that yields a smooth optimal distribution. To render the learning criterion less concave, propose a convex-composingg f with the original concave function g. This composition yields the following loss function:
approach that combines convex function
(0) = Σ
Paata (x:) fg(po(x:)),
(10)
5

---

where f is an increasing convex function and is an increasing concave function. The objective of this composition to moderate the concavity of the overall loss function, thereby allowing for sharper optimal distribution. The subsequent theorem and corollaries outline the theoretical properties associated with the optimal distribution under this function composition framework.
Theorem 3. Let be an increasing convex function and g be an increasing concave function. Then, there exists positive integer m such that the following inequalities hold:
1. (xi) 2 Pr. for all i m,
2. Prg( S (xi) for all i 2 m.
Corollary 1. The Shannon entropy of Pig is less than or equal to the Shannon entropy of Pa-
Corollary 2. For any n E 21.,...) the sum of the probabilities of the n most probable samples increases: 2: (1s.)(( 2 Pz.)..
Theorem 3 indicates that the convex-compose comeo-commpoiiionnn loss function tends to allocate higher probabilities to the more probable samples, while simultaneously diminishing the probabilities assigned to less probable ones, resulting in a sharper optimal distribution. Corollary [ quantitatively establishes this observation, demonstrating that the incorporation of convex function into the loss function effectively sharpens the optimal distribution, as evidenced by a reduction in the Shannon entropy of P5g compared with pg. Furthermore, Corollary / reveals an increase in the cumulative probability of the n most probable samples. Consequently, text generation models are better equipped to capture the highly probable outputs without explicitly modeling the data distribution.
In the above analysis, we only assume the original loss 9 to be an increasing concave function. By imposing specific conditions on g, we can derive more desirable properties from the optimal distribution psg. as demonstrated in the subsequent theorem:
Theorem 4. Let f be an increasing convex function and be an increasing concave function. If satisfies grd (z) (z)
(z)² for all x E (0, (0. then the difference between pfy and Py exhibits
a monotonic order: Pfg(=1) Py (21) 2 Pfg(22) Po (x2) 2 ... P - Pg (2 2 0,
where is the positive integer described in Theorem
This theorem provides a more granular description of the relative difference between Pfg and Pg- When Pfg decreases the probabilities assigned less probable samples, it tends to reallocate this probability mass to the most probable samples. This enables text generation models to more accurately capture the most probable outputs. Note that the condition (z) g'(x) 2 g"(=)² > is not overly restrictive. For instance, the loss function g log in MLE readily fulfills this condition:
2 1
1
1
log"" (z) log' (x) (x))
(
-²
x4 >0
(11)
x3 x
3.3.2 Practical Applications
The preceding theoretical analysis highlights the effectiveness of function composition. Here turn to its practical applications and give some examples of convex-composition loss functions. The loss function in maximum likelihood estimation is typically the log rrg-pobaability, and length normalization is often applied in practical usage, resulting in the loss (()((x) log(pe(<)) where T denotes the sentence length. Common choices for increasing convex functions on (-oo, 0) include the exponential function fx) = fx k 2 0 and the power function f(x) = (-=x)* >*x 2000 k < 1.
Through function composition, we can derive the following losses:
(x))) - (o))
f(z) ka
fg(po(x))
(12)
f(z) = --(--)) (-
The gradient of the convex compex-composiiion function (((((o(x))) -g' (po (z)). Compared to the gradient of the original loss g' (p)( it has an additional term I (((((()) that can be interpreted as a weight for the loss. Given that ʃ is a convex function and 9 an increasing function, the weight f (z)) is larger for more probable samples, thereby directing the model's focus towards

---

B150 (eCamura)
EN-DE
DE-EN
260
BLEU
26.8
26.6
25:9
bearn size
beam size
Figure Translation quality (BLEU) of autoregressive model as beam size varies on WMT14 EN++DE test set.
E.Di C+
+ Outputt NL (oCorves)
BEE (ALE)
Outpet MLL (4L3)
Vanilla-NAT
23
TIN O0P00
BLEU
k-th Power
k-th Power
k-th Power
Figure 2: Translation quality (BLEU) and prediction confidence (Output NLL) of different NAT models as the exponent k varies on WMT14 EN-DE test set.
generating outputs with high probabilities. Specifically, the loss weights (((((p(x))) associated with Equation 12 are:
f((((o(=)))= k. (x(x) k. log( (1)))) (k-1
f(x) txz -(=))
(13)
(og(puls)))
where the exponential function weights the sample by the prediction probability, and the power function weights the sample by the log log-probability.
In practical applications, label smoothing [59] 631 is a widely used regularization technique for text generation models. The smoothing loss and log-probabilityy loss are typically combined using fixed hyperparameter <ls- To preserve the ratio of smoothing loss to ogg-robbbillity loss, we also apply the weight I' (=(()( to the smoothing loss before interpolating it with the convex-composition loss.
4 Experiments
To validate the practical advantages of loss functions with sharper optimal distributions, we conduct experiments on basic autoregressive (AR) models, non-autoregressive (NAR) models, and large language models (LLMs). We evaluate their performance on two representative closed-ended text generation tasks, including neural machine translation and text summarization. Following the theoretical analysis in previous sections, we combine the exponential function with standard
haa-prbbability. i.e. (0) = - Egwpaata (a) (P) $), our training objective in the following experiments. We have also attempted to combine the power function with log- preooabbbyiyty as training objective. We found that the power form encountered some difficulties during training, leading to worse performance compared to the exponential form. Due to the space limit, we leave the results under this setting in Appendix
Our theoretical analysis suggests that the model trained by convex- composionmoiiiion loss tends to predict a sharper distribution, in which the probability mass is more heavily allocated to the most probable samples. Such property leads the model becoming more confident about its prediction and facilitates the de-facto maximum posteriori (MAP) decoding framework in closed-ended text generation tasks. In the following, we will discuss and validate the effects of convexity the context of AR models, NAR models, and LLMs respectively. More details of settings can be found in Appendix B
7

---

Table 1: BLEU scores of autoregressive models on WMT14 EN+DDE test set with different decoding strategies.
EN-DE
DE-EN beam5
Model
greedy
beam5
greedy
A
Transformer
662
26.48
27.57
1.09
29.78
31.21
1.43
Transformer Convex
26.92
27.78
0.86
30.32
31.33
1.01
Table 2: ROUGE scores on CNN/DailyMail and XSum test sets. RG-1, RG-2, RG-L stand for ROUGE-1, ROUGE-2 and ROUGE-L scores.
CNN/DM RG-2
XSUM RG-2
Model
RG-1
RG-L
RG-1
RG-I
Transformer 62
39.03
15.98
35.88
31.04
10.68
24.77 25.09
Transformer Convex
39.56
16.84
36.26
31.55
11.13
4.1 Effects of Convexity on Autoregressive Models
In the context of autoregressive models, model distribution trained with compo-commoiiiionn loss tends to exhibit fewer modes and sharper distribution, thereby facilitating the task of approximate search algorithms identifying the most likely output. We validate this conjecture by investigating the performance of greedy and beam search when trained with standard MLE and convex- comee--composition loss in translation and summarization tasks. For translation task, we vary the beam size from 3,,2,3,5, 5,8,. 83. where beam size I can be considered as greedy search.
Figure D visualizes the results in terms of BLEU 38 with precise numerical values given in Table We observe a consistent improvement in translation quality when using convex-composition losses compared MLE, and a similar trend observed summarization tasks as detailed in Table 2 These results provide experimental support that the composition with convex function promotes those approximate searching algorithms to perform argmax decoding. Meanwhile, Table exhibits diminishing gap between greedy search and beam search when equipped with convex comex-ccompoiition loss. This outcome can be attributed to the efficacy of the convex function in reducing the complexity of the model distribution, as described in Theorem 3] Such property amplifies the potential of lightweight approximate decoding algorithms within the autoregressive structure, a desirable trait in the context of modern, computation-intensst autoregressive neural networks.
4.2 Effects of Convexity on Non-autoregressive Models
anne-utorrerrssvvee models face the challenge multi- multi-modality, where fitting data distribution with multiple target modes exceeds the capabilities of NAR models. Therefore, the mode collapse property of convex-composition loss would beneficial NAR models. Likelihood training will force the model to ignore sequential dependency, resulting in disfluency in its output (e.g., token repetition and omission). In contrast, convex-compe compo-com loss would encourage model to allocate most of its probability mass to the best among all proper candidates. Such property is able to help NAR model avoid generating a mixture modes, thereby alleviating disfluency issues.
To demonstrate its effectiveness, we investigate the performance of convex- conve-xcomositionn loss on three representative NAR models, including Vanilla-ATT (I7). CMLM [4 and CTC 46). Considering most of the NAR researches are restricted in the field of translation, we only conduct experiments on translation dataset. In addition to translation quality, we also assess the prediction confidence and generation fluency of NAR outputs. The prediction confidence is measured with negative log- likelihood of its generation and the fluency is measured by an external pre-traine pre-trained language model i We use the PPL value reported by the language model to quantify the fluency of generation. The exponent hyperparameter & manipulated to adjust the convexity our composite loss function.

---

Table 3: BLEU and COMET scores on WMT14 EN+ test set.
EN-DE
DE-EN
Model
Speedup
BLEU
COMET
BLEU
COMET
Transformer 62
1.0x
27.57
82.76
31.21
82.98
Vanilla- NAT 17
15.6x
10.41
40.69
16.01
56.03
Vanilla-NAT NT Convex
15.6x
16.74
57.25
22.63
68.83
CMLM [4
15.0x
11.22
43.62
15.26
56,63
CMLM Convex
15.0x
20.45
65.99
19.11
63.54
CTC [46
14.7x
16.98
54.77
20,53
66.26
CTC Convex
14.7x
23.34
67.38
26.68
74.75
Table Prediction confidence (Output NLL) and generation fluency (External PPL) of CMLM on WMT14 EN-DE test set.
k-th Power
1
2
5
8
Confidence (Output NLL) 4
20.57
13.72
10.09
6,85
4.88
Fluency (External PPL)
939.34
481.08
315.54
213.84
218.68
The results shown in Figure and quality across all NAT models with a maximum improvement of 9+ BLEU points on CMLM. Meanwhile, Figure Figure@iimplies implies that prediction confidence significantly gains and the gain increases as k gets larger. Such phenomenon reveals a descending trend of model entropy as applying convex function loss, which is consistent with Corollary [ More importantly, We between model entropy and generation fluency in Table 4 providing clear evidence that the mode collapse property of convex function indeed relieves NAR model from multi-modality problem.
and Table We observe consistent improvement in translation
strong correlation
4.3 Effects of Convexity on Large Language Models
Large language models have demonstrated remarkable capabilities in various applications, including both open-ended and closed-ended text generation tasks. For open-ended tasks, stochastic decoding methods such as temperature sampling are commonly employed to produce responses. In contrast, deterministic decoding methods like beam search are favored for closed-ended tasks like machine translation 222 69 30 Given that the convex nanve--composition loss enhances the model's ability to identify highly probable sentences, incorporating this loss function into the L.LMs' training process would be beneficial to closed-ended generation tasks.
To demonstrate its effectiveness, we assess the performance of LLMs in machine translation (Table 5) and summarization (Table (). rreveas that the LLaMA-7B 7B model, incorporating convex- composition loss, surpasses the baseline model across all language pairs, achieving average improvement of 1.84 BLEU, Likewise, the LLaMA-133 model with convex- comprextcoooiiinonn loss outperforms the baseline model in three out four language pairs. Table demonstrates the effectiveness of our method in text summarization. Due to memory limitations, we are only able to decode the text summarization dataset using the L.LAMA-BB model.
Related Work
Alternative Loss Functions Maximum likelihood estimation has become the dominant approach for learning text generation models, but it also comes with certain limitations. Various alternative loss functions have been proposed to improve the training process from different perspectives. Regarding the exposure bias problem 42 that autoregressive models are exposed to different distributions during training and inference, Bengio al. 4, Mihaylova and Martins 32 Zhang et [72] proposed to reduce this gap by sampling from the model's 's own predictions during training. Another issue with text generation models is text degeneration: output text may bland, incoherent, or gets stuck in
repetitive loops 200). To avoid text degeneration, Dieng et al. proposed learning criterion termed

---

Table 5: BLEU scores of Alpaca fine-tuned large language models on WMT22 test sets.
Model
EN-DE
DE-EN
EN-ZH
ZH-EN
AVG
LLaMA-7B
25,42
17.93
13.86
13.17
17.59
LLaMA-7B + Convex
27.57
19.88
15.00
15.28
19.43
LJaMA-13B
29.35 28.75
21.74
15.58
16.27
20.74
LLaMA-13B + Convex
22.20
16.25
20.08
21.82
Table 6: ROUGE scores of Alpaca fine-tuned large language models on CNN/DailyMail.
Model
RG-1
RG-2
RG-L
AVG
LLaMA-7B
28.66
12.49
26.37
22.51
LLaMA-7B Convex
32.76
14.67
30.00
25.81
reflective likelihood to penalize incoherent outputs, and Welleck et al. [66 proposed unlikelihood training that forces unlikely generations to be assigned lower probability by the model. Additionally, to address the discrepancy between likelihood training and evaluation metrics, loss functions that
more directly optimize evaluation metrics are proposed. Ranzato et al. 42] utilized the reinforcement learning technique to train recurrent neural networks with sequence level objectives. Shen et (53)
proposed to optimize evaluation metrics with minimum risk training. Norouzi et al. [35 Edunov al. 12 incorporated evaluation metrics into the maximum likelihood training objective. There are also efforts on learning more focused distribution for text generation models [37 71,
[56). However,
these approaches primarily reformulate the loss function at the word level, which is insufficient for guiding the model towards identifying high-probabilic sentences at the sentence level. In contrast, our method explicitly trains the model to concentrate on generating highly probable sentences.
Reinforcement Learning Our work aligns closely with reinforcement learning (RL) based training techniques for text generation [67] (88. While RL techniques typically maximize the expected reward by concentrating the probability mass on the sequence with the highest reward, our approach strives to put all the probability mass on the most likely sequence. RL allows for text generation models to optimize discrete evaluation metrics, which have wide usage in text generation tasks, including machine translation [42] 3) text summarization (39], image captioning [44], dialogue generation (27), etc. Furthermore, RL can be integrated with Generative Adversarial Networks [68] and can leverage human feedback for training [57 36).
Loss Functions for NAR Models The limitation of maximum likelihood estimation is amplified in non-autoregressive (NAR) models since they inherently lack the capability fit the data distribution 21 To address this issue, researchers have developed loss functions specifically designed for NAR
models, guiding them towards generating coherent text. Shao al. [50 (2). Ding et al. [00 proposed to train NAR models with sequence-level objective functions. Ghazvininejad et al. [15]. Du et al. [II] relaxed the alignment restriction in the cross-entropy loss. Shao al. (5T), Shao and Feng [49], Ma et al. (1 proposed n-gram based differentiable training objectives optimize n- n-gram prediction accuracy. However, these methods lack theoretical guarantees for the shape optimal distribution.
Conclusion
This paper investigates the theoretical properties and practical applications of novel class of training objectives based on convex functions. Our findings show that convex functions can sharpen the optimal distribution, enabling text generation models to focus on highly probable outputs without having to estimate the entire data distribution. Experiments on various text generation tasks and models verify our theoretical analysis and demonstrate the practical effectiveness of our approach.
7 Acknowledgement
We thank the anonymous reviewers for their insightful comments.
10

---

References
[1] H. Akaike. Information Theory and an Extension of the Maximum Likelihood Principle, pages 610-624. Springer New York, New York, NY. 1992. ISBN 978- 978-1-4612-0019-33 doi: 10.1007/ 978-1-4612-0919-5
URLhttps://doo org/10 /ttp:/0010100/776----666222--000
[2] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In Y. Bengio and Y. LeCun, editors, 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. uttpttp:///aa /ttpiv///rrivvv//aa org./iii..r///oooooe abs/ 140e./149...0 0473
[3] D. Bahdanau, Brakel, K. Xu, A. Goyal, R. Lowe, J. Pineau, A. Courville, and Y. Bengio. An actor-critic algorithm for sequence prediction. In International Conference on Learning Representations, 2017. URL /ttcpppe//peemmeeee......eteeeep
[4] Bengio, O. Vinyals, N. Jaitly, and Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In C. Cortes, N. Lawrence. D. Lee, M. Sugiyama, and R. Gamett, editors, Advances Neural Information Processing Systems, volume 28. Curran Associates, neurips cc/ wwwpsccccpppppppeeeeeeeeeee pappp_pppp__fe leteerfffleepppppppeeeeee palees//pperr221///tteee httpa///sa..aaaoo//////////////a pdf
Inc., 2015. URL
[5] Y. Bengio, Ducharme, P. Vincent, and C. Janvin. A neural probabilistic language model. Mach. Learn. Res., 3:1137-1155, 2003. URL http::/jjirrr...../// org/ http:/aars//eeeeeeeeeeeeeeaeeeae /3/ v3/bengio03a
html
[6] M. Bhandari, P. N. Gour, A. Ashfaq, Liu, and G. Neubig. Re-evaluating evaluation in text summarization. In Proceedings of the 2020 Conference Empirical Methods
in Natural Language Processing (EMNLP), pages 9347-9359, Online, Nov. 2020. Asso- ciation for Computational Linguisties. doi: 10116555/112222222mmmmmmmmmm
URL
wwwphhologoooooooooooooooooooooe 2020. mmnnp-main. mam-maiin..111.................1 751
[7] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[8] K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Bengio. Learning phrase representations using RNN
for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734, Doha, Qatar, Oct. 2014. Association for Computational Linguistics. doi: 10.3 10.3115/v1/D14-11799 URL https: www. awwwww.ww...... org/ anthology/ D14-1179
[9] A. B. Dieng, K. Cho, D. M. Blei, and LeCun. Learning with reflective likelihoods, 2019. URL
[10] L. Ding, L. Wang, X. Liu, D. F. Wong. D. Tao, and Z. Tu. Progressive multi coattiggranlarityy training for non-autoregressive translation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2797-2803, Online, Aug. 2021. Association for Computational org/
Linguistics. doi: 10. 10011866/1122221.fmml findings-accl acl.247. URL 122..findings-aee 247
[11] C. Du, Z. Tu, and J. Jiang. Order-agnostic cross entropy for non-autoregressive machine the 38th International Conference translation. In M. Meila and T. Zhang, editors, Proceedings the
on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 1849-2855 PMLR, 2021. URL http:htttp///ceeeeeee mlr. wwwpss/vvi9///eeeeeeee. du21c. html
[12] Edunov, M. Ott, M. Auli, D. Grangier, and M. Ranzato. Classical structured prediction losses for sequence to sequence learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 355-364, New Orleans, Louisiana, June 2018 8. Association for Computational Linguistics. doi: 10. 18553/11/118. 1033. URL https: aaclanhhologyy
org 10e/N18-1100 1033

---

[13] A. Fan, M. Lewis, and Y. Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 889-896 Melboure, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18 1082.
[14] M. Ghazvininejad, O. Levy, Y. Liu, and L. Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-UCCNLP), pages 6112-6121, 2019. URL
httpps//www. acwweb...wbb......
[15] M. Ghazvininejad, V. Karpukhin, I.. Zeutleooerr, and O. Levy. Aligned cross entropy for non-autoregressive machine translation. In Proceedings of the 37th International Conference on Machine Learning, ICML. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3515-3523, PMLR, 2020. URL URR/http:/ mlr. wwwpss/see////eeeeepe. http11////gaaaiiiieeee222222 html
[16] J. Gu and X. Kong. Fully non-autoregreessiv neural machine translation: Tricks of the trade. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 120- 133, Online, Aug. 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 findings-acle ac1.11. URL httttss:/aacaaathlllleeee .rgg 2021 fiidingss-cll.111
[17] Gu, J. Bradbury, C. Xiong. O. K. Li, and R. Socher. Non-autoregreessvve neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018, Van- couver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL
https://opeerevieer www.neee//oooddd/eeeee.
[18] K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL pttpppp//ooooceeeiiggg..ooo neurips. cc/paper.pppp..//e hatpppppr_flleseeee www.fflllspppppeeee / pddf
[19] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL http://arxiv.. org/ abs/1503, 1503..25311
[20] A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https idr.uum??ddrgGyyFvv.
//openreview.
[21] F. Huang, T. Tao, H. Zhou, L. Li, and M. Huang. On the learning of non-autoregressive transformers. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1335-9376, PMLR, 17-23 Jul 2022. URL mlr wwwpp/eevvvviee. paress/v162/ie waapeesvv22//hnnngggee huang22k. html
[22] W. Jiao, J. tse Huang, W. Wang, X. Wang, S. Shi, and Z. Tu. Parrot: Translating during chat using large language models. arXiv preprint arXiv:23044.0 02426, 2023.
[23] Y. Kim and A. M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods Natural Language Processing, pages 1317-1327, Austin, Texas, Nov. 2016. Association for Computational Linguistics. doi: 10. (0.186533v1/DD1--113 URL
1139
[24] D. P. Kingma and Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:
//axxiv. orgorrggggao aorg/bs/// 1412. 1411.69988 6980
[25] S. Kullback and R. A. Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-866 1951.
12

---

[26] M. Lewis. Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and pre-trainnng for natural language generation, translation, and comprehension. CoRR, (tx/1910.1346)) 2019. URL. http:// arxiv wwwxiv.org/bb//9000 13461
L. Zettlemoyer. BART: denoising
[27] Li, W. Monroe, A. Ritter, D. Jurafsky, M. Galley, and Gao. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pages 1 1192 1202, Austin, Texas, Nov. 2016. Association for Computational Linguistics. doi: 20118853//1/DI/ 1127. URL https: aaaas////laatholggggo org/D16-1127
[28] C.n. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 14-81, Barcelona, Spain, July 2004. Association for Computational Linguisties. URL https::/acllatthlllgg/.
org/ org/w04-02200 /////0041103
[29] D. Liu, Y. Yan, Y. Gong, W. Qi, H. Zhang, J. Jiao, W. Chen, J. Fu, L.. Shou, M. Gong, P. Wang, J. Chen, D. Jiang, J. Lv, R. Zhang, W. Wu, M. Zhou, and N. Duan. GLGE: A new general language generation evaluation benchmark. In C. Zong, F. Xia, W. Li, and R. Navigli, editors, Findings of the Association for Computational Linguistics: ACL/JJNNPP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 408 420. Association for Computational Linguistics, 2021. doi: 12216653/112221.fmgg findings--all.36. URL https://doi. org/ 10. www.111651/vv// wtt.:/22....nnninge.ee findings- acl.36
[30] Y. Liu, X. Zeng, F. Meng, and J. Zhou. Instruction position matters in sequence generation with large language models. arXiv preprint arXiv:2308. 12097, 2023.
[31] Z. Ma, C. Shao, S. Gui, M. Zhang, and Y. Feng. Fuzzy alignments in directed acyelic graph for non-autoregressive machine translation. In The Eleventh International Conference Learning Representations, 2023. URL
[32] T. Mihaylova and A. F. T. Martins. Scheduled sampling for transformers. In Proceedings of the S7th Anmual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 351-356, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/119-20000
URL httpttap/acalaattollloggg org/ 2049
[33] I. J. Myung. Tutorial maximum likelihood estimation. Journal of mathematical Psychology. 47(1):90-100, 2003.
[34] S. Narayan, S. B. Cohen, and M. Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797- 1807, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics. doi: 10.18653/v//D18. 12000
org/D18 1206.
[35] M. Norouzi, Bengio, Z. Chen, N. Jaitly, M. Schuster, Y. Wu, and D. Schuurmans. Reward augmented maximum likelihood for neural structured prediction. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and Garnett, editors, Advances in Neurul Information Processing Sys- tems, volume 29, Curran Associates, Inc., 2016. URL /Rt.ttpss///rroreednneee neurips. cc/ haper_files
/ haapop//...apeeee/////////2aaaa1
.df
[36]
L. Ouyang. J. Wu, X. Jiang, D. Almeida, C. Wainwright, Mishkin, C. Zhang, Agarwal, K. Slama, A. Gray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welin- der, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL htthttaaa//operrviieee net htt//formm??cc id= TGBKACXEON
[37] R. Y. Pang and H. He. Text generation by learning from demonstrations. In International Conference on Learning Representations, 2021. httpl./ttss///peeeeeeett net/forum? cd=RovX-uQ1Hi
13

---

[38] K. Papineni, S. Roukos, T. Ward, and W.JJ. Zhu. Bleu: a method for automatic evalua- tion of machine translation. In Proceedings of the 40th Annual Meeting of the Associa- lion for Computational Linguistics, pages 311 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguisties. doi: 10.3115/117399331773333 URL. htttp:://allanthoo..g/
org/Po2-1111 PP2--0400
[39] R. Paulus, C. Xiong, and R. Socher. A deep reinforced model for abstractive summarization, In International Conference on Learning Representations, 2018. URL https:///pperreiiee
[40] W. Qi, Y. Gong, J. Jiao, Y. Yan, W. Chen, D. Liu, K. Tang, H. Li, J. Chen, R. Zhang, M. Zhou, and N. Duan. BANG: bridging autoregressive and non-autoregressive generation with large scale pretraining. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8630-8639. PMLR, 2021. http: press/ v139/ v139/qi21a.....a.///// html
//proceedings.ml www.g.mm........ wtw..nigsmmm.......s...e..eeeeee
[41] A. Radford, K. Narasimhan, T. Salimans, Sutskever, et al. Improving language understanding by generative pre- pre-training, 2018.
[42] M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. Sequence level training with recurrent neural networks. In Y. Bengio and Y. LeCun, editors, 4th International Conference on Learning Repre- sentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URt.tttp::/ /ttt::///ariiii
org/ absa 1111111111111 06732
[43] R. Rei. J. G. C. de Souza, D. Alves, C. Zerva, A. C. Farinha, T. Glushkova, A. Lavie, L. Coheur, and A. F. T. Martins. COMET-22: inbaabl-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578-585, Abu Dhabi, United Arab Emirates (Hybrid), Dec. 2022. Association for Computational Linguistics.
URL /ttcpp://aclattaii wwwplogg.o...oggeeee.o org 2022. wmt 1. 52
[44] S. J. Rennie, E. Marcheret, Y. Mrouch, J. Ross, and V. Goel. Self-critical sequence training for image captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1195-1195, 2017. doi: 10.1109/CVRR.2117.11
[45] A. M. Rush, S. Chopra, and J. Weston. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing, pages
Lisbon, Portugal, Sept. 2015. Association for Computational htaattp://aallanhoooogy.oo
Linguistics. doi: 10.
[46] C. Saharia, W. Chan, S. Saxena, and M. Norouzi. Non-autoreeresssive machine transla- tion with latent alignments. In Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 1098-1108, Online, Nov. 2020. As- sociation for Computational Linguistics. doi: 10. 10.11663//112020.emmmppp https://aclantholegg
URL
org/ 2020022002222 cammppmmin.....
[47] A. See, P. J. Liu, and C. D. Manning. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 1/11711011111110999 URL
htttp:://aclanthol..e http://accaaatooooooooooo//o--op org/P17-1099
[48] R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/v1/p16-110 1162. URL https //doi.orge 10. 11653//1//166- 1162
[49] C. Shao and Y. Feng. Non-monotonic latent alignments for cic-based non-autooregressivee ma- chine translation. NewrlPS, 2022. URL, http://wwww.../pppppp...././///r 2022/ http://ssffoooeeeeeeeeeeeeeeeee0
.niss.iippp..................m.. cc/paper
html
14

---

[50] C. Shao, Y. Feng, J. Zhang, F. Meng, X. Chen, and J. Zhou. Retrieving sequential information for aonoeererereessivve neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3013-3024, Florence, Italy, July 2019. Association for Computational Linguistics. doi: (0.18653/v1/P19-11 1288. URLphttps 7/www.aclve org/ aathaolog/PPl /P9- 1288
[51] C. C.aoo,, J. Zhang, Y. Feng. F. Meng, and J. Zhou. Minimizing the bag-os les--of-npe of-ngrams difference for non-autoregressive neural machine translation. In The Thirly-Fourth AAAI Conference on Artificial Intelligence, AAA/ 2020, New York, NY, USA, February 7-12, 2020, pages 198- 205. AAAT Press, 2020. URL https: http://aaaaarrgg././..oo/ org/og/////ooooooo ht////dex/../..//////////////eee index.dxxxx.. php/ phpppppp///aapppppppppppppppp AAAI/AAAAA//AAA aat.Tt//riice/vveee 5351
[52] C. Shao, Y. Feng, J. Zhang, F. Meng, and J. Zhou. Sequence-Level Training for Non- Autoregressive Neural Machine Translation. Computational Linguistics, pages 1-35, 2021. ISSN 0891-2017. doi: 10.1 20.116//ooi_aa/00 a_00221. URL. ttttps://dos org/10gg//10 100 1162/coli_aa 00421
[53] S. Shen, Y. Cheng. Z. He, W. He, H. Wu, M. Sun, and Y. Liu. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1693-1692, Berlin, Germany, Aug. 2016. Association for Computational Linguistics. doi: 10. 10.18653/1//11--1559 URL
www.anthoooooooooooooooo..1... org/P16-1155
[54] R. Shu, Lee, H. Nakayama, and Cho. Latent-variable non-autoregressive neural ma- chine translation with deterministic inference using delta posterior. 34:8846-8853, Apr. 2020. doi: 10. 1609/aaa.. v34i05.6413. URL https://ojs. aaai. org/ hnt//indxx.../ hetg/nddxx.php/AAA
article/view/ 6613
[55] F. Stahlberg and B. Byrne. On NMT search errors and model errors: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages -366-3362, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-113
Cat got your tongue? In
URL htttp:://ccclttbbooooo.orrr
1331
[56] F. Stahlberg and S. Kumar. Jam or cream first? modeling ambiguity in neural machine translation with SCONES. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4950-4961, Seattle, United States, July 2022. Association for Computational Linguisties. doi: 10. 18.53vv//v1/ 2022.naacl 12222nacl-maan.365 URL httph:t/p////ttt httpt:::/tcaaaaaaaaaaa acl aclantholoyy.....omoooomoooo org. 2022. nwacl main. 365
[57] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. Christiano. Learning to summarize with human feedback. H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural In- formation Processing Systems, volume 33, pages 3008-3021. Curran Associates, Inc., 2020. URL htttps://procceedings. L1985556600000000000444448811110
fiaper_files/ppper///000 pdf
[58] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforce- ment learning with function approximation. In Proceedings of the 12th International Conference on Neural Information Processing Systems, NIPS 99, pages 1057-1063, Cambridge, MA, USA, 1999. MIT Press.
[59] C. Szegedy, V. Vanhoucke, S. loffe, J. Shlens, and Z. Wojna. Rethinking the inception archi- tecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818 2818-2826, 2016.
[60] R. Taori, I.
Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An cestruction-folloiinn Ilama model. htttps://githie com/tatsu 1ab/ stanforrdalpace 2023.
[61] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Ilama: Open and efficient foundation language models. arXiv preprint 2rXiv:2302.139711 2023.
15

---

[62] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. In Guyon, U. von Luxburg, S. Bengio. H. M. Wallach, Fer- gus, S. V. N. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998 6008, 2017. /ttphhttss:/pprreeeedii cc cc/ c htt.:////11..e////777/6606106663 Abstract html
neur ips
[63] A. Vaswani, N. Shazeer, N. Parmar. Uszkoreit, L. Jones. A. N. Gomez, u. Kaiser, and I. Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pages 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
[64] O. Vinyals and Q. Le. A neural conversational model. arXiv preprint arXiv: 05e.:1666600889 2015.
[65] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022.
[66] Welleck, Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston. Neural text generation with unlikelihood training. In International Conference on Learning Representations, 2020. URL https://oponrevist
[67] R. J. Williams. Simple statistical gradient-following algorithms for connectionist rein- forcement learning. Mach. Learn., 8(3 (1:3-4):229-222 May 1992. ISSN 0885-6125. doi:
10.1007/BF00999266. Uttptt :top:://doi... otg/10. httt:p://aol..rrrg00.00//////// 1007/0000/9/00990
[68] L. Yu, W. Zhang, J. Wang, and Y. Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, pages 2852 2858. AAAI Press, 2017.
[69] J. Zeng, F. Meng. Y. Yin, and J. Zhou. Tim: Teaching large language models to translate with comparison. arXiv preprint arXiv:2307.04400 2023.
[70] S. Zhang, Q. Fang, Z. Zhang. Z. Ma, Y. Zhou, L. Huang. M. Bu, S. Gui, Y. Chen, X. Chen, and Y. Feng. Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models. arXiv preprint arXiv:2306. 10966 2023.
[71] S. Zhang, S. Wu, O. Irsoy, S. Lu, M. Bansal, M. Dredze, and D. Rosenberg. MixCE: Training autoregressive language models by mixing forward and reverse cross-entropies. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguis- ties (Volume 1: Long Papers), pages 9027-9050, Toronto, Canada, July 2023. Associa- tion for Computational Linguistics. doi: 10.18653/1 1ttp33//1/222.accm act-ooo..0.l-- URL https
//aclanthology. org. acl-1ong.500 502
[72] W. Zhang, Y. Feng, F. Meng, You, and Q. Liu. Bridging the gap between training and inference for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4333-4343, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 1/P19- 1426. URL https:// aclanthology.ore hwwphhoooogyoooogPPPP. PP1- 1426
16

---

A Proofs
A.1 Proof of Theorem
Theorem 1. Given arbitrary differentiable and increasing function f. the optimal distribution Ps satisfies pf(x1) 2 (f())) 2 ...2 2.-2
Proof. We prove this theorem by contradiction. Suppose there exist an indice (f, such that i < j and (2;))) py (xz)) In this case, can construct distribution ply with lower loss than pf. which contradicts the optimality of py. Specifically, let ply be identical to py except for the changes p'y(2:) = pf( and (2²) = (y())). We denote the loss of model distribution p as Ly = p) and show that p'y has a lower loss:
(p ply) C, py)
Pdata Po
Ly py).
(14)
This inequality contradicts the assumption that Pf minimizes Ly, thereby proving the theorem.
-
A.2 Proof of Theorem 2
Theorem 2. If f is an increasing convex function on [0, 11, then the optimal distribution py is a one-hot distribution that (イ(1)) = 1 and py 0, > 1.
Proof. We prove this theorem by contradiction. Suppose py is not the one-hot distribution described above, then there must exist an index > such that py (a;) > O. In this case, we can construct distribution ply with lower loss than py. which contradicts the optimality of ps. Specifically, let p'y be identical to Py except for the changes ply (x11 = Py (21) + a and (x;) = (x())) a, where 0 S a S pf(())) Then we can calculate the gradient of loss Ly = p'y) with respect to a:
@L p''))
= Paaaaaa;;) f' Paota ('(p)(()))(
(15)
da
We can analyze the above equation via the following steps:
1. (x11) 2 (y..), from theorem
(p)((x)) 2 (x(r)(x)) O, from step and the convexity of f
3. Pdata ( > Pdata (x:) > 0
(16)
(pu())=)))) Da
< 0, from steps 2,3
a=
The above reasoning shows the loss can be further reduced, which contradicts the assumption that pf minimizes L, and proves the theorem.
A.3 Proof of Theorem
Theorem 3. Let be increasing convex function and be increasing concave function. Then, there exists a positive integer m such that the following inequalities hold:
1. Pfg(#x) 2 (1:)) for all i m,
2. Pfg( P.((()) for all i '
Proof. We prove this theorem by contradiction. Assuming the theorem does not hold, there must exist an indice (i,j) with < j. P)o((x)) < Py (xi). and Pfg(#;) > (Iş)). In this case, we can construct a distribution f/1 with lower loss than Pfg. which contradicts the optimality of Pfg-
First, we can establish the following inequality from the optimality of Pg:
(F.) - (Pp 2 Pdaza(I,) g'(p (z;)).
(17)
17

---

Assume the above inequality does not hold, then we can further reduce the loss, which contradicts the optimality of Pg. Let P (zx) = (2;) - a, and P'y (21) Pg PT( + a. The gradient of loss L/ with respect to = is Pduta (zi) g (Py (z:)) - Pduta (zz) g' (Py((a) (z,)) O, so we can further reduce the loss with positive a, proving inequality by contradiction.
Then, let F'ys be identical to Pfa except for the changes P'rg(z:) Pra(#;) + and p/1g (2z)) Pro(zz) - a, where S a S PPg(2z)) We can calculate the gradient of loss Llg(Pe p// with respect to a:
ic (po 1/19)
= Pataaa
('( (p)g(z)))
da
a==
(18)
We can analyze the above equation via the following steps:
1. g'(((s((xa)) 2 (1s)) from (fn < Py (1s) and the concavity of
2. g'(P5s (xz)) S (Pg((z))) from Pfy (23) > Py (my) and the concavity of
(19)
3. Paata (zi) g (P)()))) 2 (Zz) g (Pp )), from inequality 17
Combining steps 1-3, we obtain: Pdaata (24) (((pa((z)) 2 Pdata (1i) .gg (Pa (z:)) 2 Pdata (xz) .g' (P (x;)) 2 Pitara (zjaa (Pro(z))
(20)
Further, the following steps shows that the gradient is less than O:
1. (1g( 2 (x)) from theorem D
f'g(P(o( 2 f (x))))))) from step 1, the increasing property of g, and the convexity of f
(P
3.
from step and inequality 20
8a
(21)
The above reasoning shows the loss can be further reduced, which contradicts the assumption that P minimizes y and proves the theorem.
Corollary 1. The Shannon entropy of pfg is less than or equal to the Shannon entropy of Pg-
Proof. The Shannon entropy of distribution p. denoted by Hp. is defined as Hp
-E. log p(z). Consider a function h(Az) = -- + Az) log(z1+ +Zz) - log(22 Az). It's It's first-order derivative h' (A)) = log(xz2 Ar) log(21 + Az). Assuming observe that h'(rr) < O when Az > 0, so the entropy decreases when we reduce 22 and increase I accordingly. The transformation from Pg to Pfg can be viewed as a series of such adjustments, which implies that the Shannon entropy of fig is less than or equal to the Shannon entropy of Pg- 11
2 Z2, we
Corollary 2. For any n E (..,, the sum of the probabilities of the n most probable samples increases: E(=1P/o(1a) 2 Σh=1 (r,).
Proof. The theorem guarantees the existence of a positive integer m such that Pfg (zi) 2 Pg (g(()) for all i < m and (f; S (x;) for all 2 m. In the case where n < m, we have Pio 2 Py (x:) for all i with < i I n < m. This leads to the inequality E Pfg(a:) 2 Ex Pa (z:). In the case where n 2 m, we have Pfo S Pa();) for all with m S n < i. Therefore, we can write EL (P (x:) (xx)) = - Eien+ (P)g(a;)) Pg
2 O. Consequently, the inequality EE Pfg 2 EhmP Po (2:) also holds. Therefore, in both cases, the corollary is proved.
Li
A.4 Proof of Theorem 4
Theorem 4. Let J be an increasing convex function and g be an increasing concuve function. If satisfies o (x)--' (x:) 2 (x)²  > ffrrr all x E (0. 1), then the difference between pfg and Py exhibits
a monotonic order: Pfg(z1)- - (m)) 2 Prg(x2) (x2) P 2 2 P&g (F Py (2 2
where m is the positive integer described in Theorem
18

---

Proof. We prove this theorem by contradiction. Assuming the theorem does not hold, there must exist an indice indice (i.j) with j and S (z,(2;) -P Pz)(
- Pg (zy). In this case. we can construct a distribution P'ys with a lower loss than pfg. which contradicts the optimality of Pfa- Specifically, let y'rg be identical to Píg except for the changes P/gg(2) (z) + and
(2)(22)) P)g(zz) a, where 0 S P/g(zz)) Then we calculate the gradient of loss Lip(Po -''g)) with respect to a:
dLs 1/1g) da
Pdato
(22)
Our goal is to demonstrate that (pe 8c
< 0, which would contradict the assumption that
Pfg minimizes Lyg, thereby proving the theorem.
Given the optimality of Pg. we have (m;) g' (p,
Pdota (=z) g (P)P((z))) otherwise we know that Pfa (x.) 2 (z, 1
we can reduce (x;) to obtain a lower loss. From Theorem and because f is convex and g is increasing, we have (")(Pra(z;)) 2 f'9(p1a(z;)) Using these inequalities, we obtain a upperbound of equation 22
f = -''g))
(Paata(z;) (p)((x))))
da
la=0
(daaa(wz)) .. ('g(ps((a))) ('(Pro(a;)) Paita( I' ('g(Pro(ai)) lrligggg(pp (((p)((a))
Pdata( (Pg
Pdata (z,) (p)g(=:))
< Pdtaa (zz) f'g(Pia (x))) . ((,r)o(z))
Paata(x)) (' (x))
Pdata (2,) (P)
(x,) (())(az;))
(1)( (1,))(
('9(P1((a:)) - g(pps((x))
g((p((zz))
(((p((rs))
$ ('9(P)o(z3)) (ali))(ggeee (P(ra(E)))
d(prolaa)))
g((p((Iz))
g((,)(,,))
(23)
To demonstrate that aCfa(p9 r/4)
0, we only need to prove the following inequality:
8a
(((r)(Z))) d'(rr((za) <0. O. (()g))))) g(Pg(ar;)
(24)
Let Az = (z)( - Po(x.)) < P(a(zz)) Pz(z)). As result, Pfg (22) Az + Po( and thus ('(P)(Z3)) < o ('(n(1z)) + Ar). This allows us to further simplify the inequality:
g((prg(Zz)) (1,1)((1,)) g' g(mn((z;))
< g',,((,,) + Ar). g' ('(y((r;) + Ar)
(25)
g' g((g(1z;)
('(pg(a)))
g' g'(pg(z:)
To establish that the right-hand side of the above inequality is non-positive, can apply the logarithm transformation and show the following inequality instead:
log(g' (p, + Ar) - log(g' (po (x,))) I log(g' (Pg(x.) Ar) log(o' (z:()))).
(26)
Let's denote h(z) log(g' (x)),, 21 = Pg(xx), and Ø2 Pg(x)). The above inequality can be simplified to:
l(x2 Ax) Mx22)= S h(x + Ax) h(x1),
(27)
where Ar 2 0 and 22 S x1 according Theorem The above inequality holds when h(x)) is convex function. The second-order derivative of h(r)) log(g'(z)) is:
(z(z) 9" (x) ) (x) (x)
(28)
gx(x)2
Therefore, h(z) is a convex function under the condition (x) g((x) 2 (x)?.. This verifies that BC y (po
p < 0, completing the proof by contradiction.
8a
a=0
19

---

B Experimental Settings
B.1 Machine Translation
B.1.1 Datasets and Metrics
Datasets We conduct experiments on widely used translation benchmark: WMT14 (EN-DE, 4.5M), where the validation and test sets are newstes/201 and newstes/2014 respectively. We apply BPE 48 with 32K merge operations to learn joint vocabulary on the tokenized data. Considering the major topic of this research how to learn from a real-world data distribution, we don't apply any tricks that may have an influence on the distribution, e.g., knowledge distillation.
Metries The overall quality of translation assessed using metrics BLEU 38 and COMET 43 In
the case of non-autoregressive models, we additionally quantify the prediction confidence and transla- tion fluency of the generated output. Prediction confidence measured with negative log- ligelikllooodd (NLL) of model generation. A lower NLL value indicates more focused model distribution and higher prediction confidence. To evaluate translation fluency. we utilize an external pre-trained autoregressive language model. The generated translation is fed to the language model using teacher foreing and the resulting perplexity (PPI.) is calculated as measure of fluency lower external PPL. score indicates higher level of fluency.
B.1.2 Implementation Details
Architectures In order to validate the overall efficacy of convex- convex-composition loss, we perform experiments using various model architectures. We adopt Transformer-base [63] autoregressive baseline and Vanilla-NAT [77, CMLM [4] and CTC 46 as our non-autoregressive baselines. We apply uniform copy to construct decoder inputs in Vanilla-NAT and CTC. The decoder length in CTC is set to 2x the source length.
Training Although training with convex- cowpe-compooiiionn loss offers the desirable property of optimality, can encounter gradient vanishing issues during initialization as analyzed previously. To mitigate this, we employ a two-step training approach: MLE pre-training followed by fine-tuning with convex- composition loss. This approach allows us to avoid numerical gradient issues while still benefiting from the optimality achieved through convex composition. For training with convex-composition loss, we set the exponent hyperparameter to for the autoregressive model and tune it from (1,2,3,5,8) on the validation set for oon-eutoreeressivve models. Throughout both MLE and convex-composition training. all models are optimized using the Adam optimizer [24] with 3 = (0.9, 0.98) and = 10- Detailed information regarding other training hyperparameters can be found in Table 7
Table 7: Settings of training hyperparameters on WMT14 EN+DE dataset.
Transformer
Vanilla-NAT MLE
CMLM
CTC
MLE Convex
Convex
MLE
Convex
MLE
Convex
batch size
32k
64k
256k
64k
256k
64k
256k
learning rate
7c-4
2c-4
5c-4
3c-4
5e-4
3c-4
5c-4
3c-4
warmup steps
4k
1k
10k
500
10k
500
10k
500
training steps
200k
50k
300k
10k
300k
10k
300k
10k
dropout
0.1
0.1
0.3
0.3
0.3
0.3
0.3
0.1
weight decay
0.01
0.01
0.01
0.01
0.01
0.01
lahel smoothing
0.1
0.1
0.1
0
0.1
0
0.01
0
length loss factor
0.1
0.01
0.1
0.01
Decoding For the autoregressive model, we set the beam length to 5 by default and tune the length penalty on the validation set unless stated otherwise. For Vanilla-NAT and CTC, we utilize fully non-autoregressive argmax decoding. In the case of CMLM, we employ 5 length candidates and
'We checkpoint Unbabel/ wwtp2m comet-ooetteeee compute COMET score. It is available at https //gi figithub.dd com/ //be/el/bael
"ee use checkpoint transformer_ii ww. de to compute the extemal PPL score. It is available at
20

---

disable iteration for inference. The decoding speedup is measured with batch size of I on GeForce RTX 3090 GPUs.
B.2
Abstractive Summarization
B.2.1 Datasets and Metrics
We conduct experiments on two widely used summarization benchmarks: CNN//aalyMail [18] and Xsum [34]. MNal contains 220K articles from the Daily Mail newspaper and 93K articles from CNN. Each article contains a bullet point summary consisting of multiple sentences. We use the non-anonymized variant following 47 29 After the pre-processing, there are 311,971 (article, summary) pairs. XSum consists of 227K online articles from the British Broadcasting Corporation (BBC), containing professionally written single-sentence summaries. After the preprocessing, there are 226,677 (article, summary) data pairs. In order to maintain consistency with previous works 26 40 we employ GPT-2 tokenizer to tokenize raw MNai data, and Berttokenizer to tokenize raw Xsum data. The summarization quality is measured with ROUGE-1, ROUGE-2 and ROUGE-L
23 as discussed in [
B.2.2
Implementation Details
In our summarization experiments, most of the implementation details of the Transformer align with those used in translation. However, there are few modifications ensure consistency with previous work (26). We apply layer normalization to the embeddings. The attention dropout is set to 0.1, and the weight decay is set to 0.01. We utilize beam search with a size during decoding. The length set to 2.0, 140, and 55, respectively CNN/DailyMail dataset. We use a length penalty of 1.2 on Xsum dataset. For CNN/DailyMail dataset, we additionally employ tri- gramgrmm repetition prevention trick.
penalty, max_len_b, and min_len
B.3 Large Language Models
For the development of LLMs, we utilize LLaMA-7B and LLaMA-13B 61 as our foundation
models. We conduct instruction tuning using the Alpaca dataset by GPT4 665 60 which comprises
52K Instruction-followiii demonstrations. Instead of the standard cross-entropy loss employed during instruction tuning, we adopt the convex-composition loss of exponential form to fine-tune foundation models.
The generative capability of LLMs is also evaluated on the two representative closed-ended text generation tasks: machine translation and text summarization. For machine translation, we follow previous works [22
to evaluate the translation capability on four WMT22 translation English-to-Chin -hhineee, German-to-Engliste and English-to-GGerman) For text summarization, we follow Liu et al. 30 to conduct the evaluation on CNN/DailyMaii Dataset [8. We employ beam search with beam size of for machine translation and for summarization. The prompt for machine translation is "Translate the following sentences from [SRC] to [TGT)." The prompt summarization is "Write a brief and focused summary of the passage that follows.".
70
69 30
tasks CChinese-t--nngline
C Effects of k on AR Models
We study the effects of exponent hyper- rrper-parameter BLEU scores of autoregressive models as the exponent varies, showing that the optimal performance achieved when k 1. Other choices of k, such as k 0.5 0.75, also yield improvements, predominantly in the context of the greedy search setting.
autoregressive models. Table & presents the
Table BLEU scores of autoregressive models as the exponent varies on WMT14 EN-DE test set.
k-th Power
0.5
0.75
3
Greedy
26.89
26.89
26.92
26.78
26.13
Beam5
27.62
27.74
27.78
27.49
26.76
21

---

D Correlations on Other NAR Models
In Section we present compelling evidence in support of the mode collapse property of convex function effectively mitigating the multimodaliiyy issuc in the NAR model. This evidence is derived from the strong correlation observed between model entropy and generation fluency the CMLM model, as demonstrated in Table 4 In this section, provide additional evidence for other NAR models to further support our findings Table and 10
Table 9: Prediction confidence (Output NLL) and generation fluency (External PPL) of Vanilla-NAT on WMT14 EN-DE test set.
k-th Power
5
8
Confidence (Output NLL)
23.34
16. 17
6.27 463.78
11.32
6.25 344.56
Fluency (External PPL) 4
1000.06
730.91
353.40
Table 10: Prediction confidence (Output NLL) and generation fluency (External PPL) of CTC on WMT14 EN-DE test set.
k-th Power
8
Confidence (Output NLL) +
18.74 174.79
13.88 142.80
11.20
7.69 137.07
5.55 154.60
Fluency (External PPL)
134.28
E Results on Alternative Choice of Convex Function
In addition to the exponential function, we have also explored another choice of convex function in our framework of convex-composition loss. this section, we discuss the results of the choice
of power function, i.e., (0) -Ez (x) (=)[-(( (o)(n((a)) }*] 0 < & < 1. The results obtained
from applying the power function in convex-composition loss presented in Table I1
Table 11:
Results of BLEU scores by applying power function convex- convex-comoosition loss. We denote the MLE baseline by using k = 1.0. We employ greedy decoding for Transformer. In cases where training fails, we use "N/A" to denote such instances.
k-th Power
0.1
0.3
0.5
0.7
1.0
Transformer 26.64 Vanilla-NAT
26.68
26.60
26.52
26.48
N/A
N/A
10.74
10.51
10.41
We have observed that the benefits of applying the power function within the convex composition framework are significantly marginal compared the exponential function, especially the case of Vanilla-NAT. In addition, we have found the training process may encounter difficulties or failure when approaching O. We attribute such problem to the shape of (p(g(p( when power function is applied, i.c., k- (- [_og(po(z)))))
As shown in Figure 3 the value of (z)(()( will approach constant as approaches This phenomenon arises due to the reduction in the convexity of function f. resulting in decrease in gain. In case approaching O, the situation even worse where ['(g(po (x:)) will experience a sudden increase from an extremely small value near O. These factors result in unstable training process and contribute the power function being less suitable within the framework of convex- composcon loss.
22

---

14
K= 0.1, power power
12
X 3. exponendol
10
8
6
4
0
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0.0
Figure 3: Shapes of ((((p(()))) when different convex functions applied.
Results on Diverse Generation
We study the effects of convex functions VAE-based text generation models by replacing the log-probability-basse reconstruction loss in ELBO with the convex-composition loss. Formally, we train the model using the following loss:
[a - -( log p(]]())))) + KI(((>[=))lp(2);
(29)
where we opt for the convex function f to be e k 2 0. We perform experiments within the context of conditional generation, utilizing VAE-based non-autoregressive model [54] [6] for the task of machine translation. During inference, we randomly sample the latent variable times to generate diverse texts. We assess the quality with BLEU score computed against reference (reference-BLEi) BLEU) and measure the diversity with BLEU score computed against each other (pairwise-BLEU) The average value and standard derivation reported in Table 12
Table 12: Reference-BLEU and Pairwise-BLEU scores of VAB-based NAT models trained with different objectives on WMT14 EN-DE test set. The texts are generated by sampling the latent distribution 3 times.
ELBO
Convex + KL
Reference-BLEU 16.23+.14
23.35+.04
Pairwise-BLEU
29.52+.20
91.91+.03
During the training process, we have observed that KI. divergence tends to vanish more readily when the convex functions are applied. We attribute this phenomenon to the smaller norms of gradients associated with the convex-compositae the model update, leading to the KL divergence vanishing. We note VAE-based text generation models trained using the comvex-composition loss exhibit a higher generation quality while suffering from poor diversity, which consistent with the mode collapse property of convex function.
loss. As result, the gradient of the KI. divergence dominates
G Analysis of Convex Learning and Knowledge Distillation
With the ability to capture a concentrated distribution from datasets exhibiting multi-moda distribu- tion, the proposed convex learning approach shows similar dynamics to knowledge distillation [19), technique which encourages the student model to imitate output the two methods, we utilize autoregressive Transformer as the teacher and apply sequence-level knowledge distillation 2 to construct a dataset of lower complexity, and train the models using different losses.
teacher model. To compare
23

---

Table 13: BLEU scores of autoregressive and vanilla-NAT models trained with or without knowledge distillation (KD) on WMT14 EN-DE test set.
Transformer
Vanilla-NAT
MLE
Convex
MLE
Convex
KD
27.73
27.80
19.18
23.17
w/o KD
27.57
27.78
10.41
16.74
28.0
BLEU
27.2
MLE
26.4
MLE KD
beam size
Figure 4: BLEU scores of autoregressive models as beam size varies with or without knowledge
distillation (KD) WMT14 EN-DE test
The results in Table 13 and Figure demonstrate that convex learning and knowledge distillation have similar effects on text generation models. Both methods lead to significant improvements on non-autoregressive models and bridge the performance gap between greedy and beam search of autoregressive models. It is worth noting that training with the convex-composition loss avoids the intricate process of training an additional teacher model and decoding the whole training set to achieve the improvements. Moreover, convex-composition loss be combined with knowledge distillation to further enhance the performance.
24


Reviews:
[{"review_id": "mZpRQJJ2V6", "reviewer": "Reviewer_v7Us", "summary": "This paper discusses the potential limitations of likelihood/KL-guided training for natural language generators from a measure theoretical perspective. It analyses the discrepancy between maximizing the likelihood (thus recall-friendly) and producing high-quality samples (thus precision-oriented) in autoregressive and non-autoregressive models, and proposes an alternative to MLE as the concrete solution. Experiments on enhancing autoregressive/non-autoregressive language models on machine translation validate the effectiveness of the proposed objective.", "questions": "1. As stated in Weaknesses, is it possible to include additional experiments on diverse text generation with VAEs? You can try interpolation and I'm very curious if the proposed approach is eventually capable of mitigating the latent collapse of text VAEs.\n\n2. I would appreciate it if you can explicitly compare against models under setups with/without sample-based distillation from teacher models. As far as I'm concerned the dynamics of distillation (especially in machine translation) would be somehow similar to the proposed approach, I'd like to see if it is possible to achieve the performance of a model w/ distillation by simply replacing the objective with the proposed one. \n\nWould be more than happy to raise my scores if these two aspects are addressed in the revision period.", "limitations": "I don't observe any potential societal limitations of this work. This work is more of an algorithmic/theoretical contribution to the community.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The theoretical analysis is insightful, solid, and well-presented. It makes a lot of sense and the corresponding experiment results comply with the analysis, especially the ones on non-autoregressive models.\n\n2. The measure theoretical problem it studies has been a critical yet long-neglected one since the falling short of language GANs. This paper is a solid step towards the ultimate solution and I would really hope the publication of this paper can bring the community's focus back to this area.", "weaknesses": "1. It's less analyzed in this paper about the proposed objective on some potential (important and necessary) scenarios. For example, for text VAE/other types of latent-variable text generators, it's long been a dilemma to improve the generation quality while preventing the latent collapse problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We provide discussions and explanations about your concerns as follows.\n\n> Include additional experiments on diverse text generation with VAEs. You can try interpolation and I'm very curious if the proposed approach is eventually capable of mitigating the latent collapse of text VAEs\n\nFollowing your advice, we study the effects of convex functions on VAE-text models by replacing the log-probability-based reconstruction loss in the ELBO with the convex loss. Formally, we train the model using the following loss:\n\n$\\mathbb{E}_{z\\sim q(z|x)} -f(\\frac{1}{T}\\log p(x|z)) + \\mathrm{KL}(q(z|x)||p(z))$.\nHere, we opted for the convex function $f$ to be $\\exp(x)$, as outlined in our paper.\n\nWe conduct the experiements in a conditional generation scenario, using a VAE-based non-autoregressive model [1,2] applied to a machine translation task. During generation, we randomly sampled the latent variable using 3 different seeds to obtain the texts. We assess the generation quality with the BLEU score computed against reference (reference-BLEU) and measured the diversity with the BLEU score computed against each other (pairwise-BLEU). The average value and standard derivation are reported in the attached response PDF.\n\nDuring the training process, we observed that the KL divergence tends to vanish more readily when the convex functions are applied. We attribute this phenomenon to the smaller norms of gradients associated with the convex-composition loss. As a result, the gradient of the KL divergence dominates the model update, leading to the vanishing KL divergence. We noted VAE-text models trained using the convex-composition loss exhibit a higher generation quality while suffering from poor diversity, which is consistent with the mode collapse property of convex loss.\n\n[1] Shu, Raphael, et al. \"Latent-variable non-autoregressive neural machine translation with deterministic inference using a delta posterior.\" Proceedings of AAAI 2020.\n\n[2] Gu, Jiatao, and Xiang Kong. \"Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade.\" Findings of ACL-IJCNLP 2021.\n\n>Compare against models under setups with/without sample-based distillation from teacher models. As far as I'm concerned the dynamics of distillation (especially in machine translation) would be somehow similar to the proposed approach, I'd like to see if it is possible to achieve the performance of a model w/ distillation by simply replacing the objective with the proposed one.\n\nThank you for your insightful comments. We agree with you that the dynamics of distillation is similar to our approach. In sequence-level knowledge distillation, the student model is encouraged to imitate the output of the teacher model, which has a less diverse training dataset and thereby tends to predict a sharper distribution. This characteristic is similar to the effect of convex loss.\n\nTo compare the two methods, we used the autoregressive Transformer as the teacher and applied sequence-level knowledge distillation to train our models. The experiment results in the attached response PDF (Table 3, Figure 1) show that convex loss and knowledge distillation have similar effects on text generation models. Both methods lead to significant improvements in non-autoregressive models and bridge the performance gap between greedy and beam search of autoregressive models. It is worth noting that our method does not require training an additional teacher model and decoding the training set to achieve these improvements. Moreover, convex loss can be combined with knowledge distillation to further enhance the performance.\n\nIf you have any further thoughts, please feel free to bring them up, and we will be more than happy to engage in an in-depth discussion."}, {"review_id": "I6ZenaS3uv", "reviewer": "Reviewer_MNhM", "summary": "This paper proposed a new learning objective for language modeling, which is extended from Maximum likelihood estimation (MLE). More specifically, the author replaced the log function in log-likelihood with an arbitrary increasing function f. Then the author analyzed the properties of the loss under scenarios where f is convex and its relaxation. Besides, the author provides two practically feasible function forms of f for the learning objectives. Experiments with exponential function show empirical improvement of transformers on translation and summarization tasks.", "questions": "1. In equation 10， is x_i a sequence or a token? In my understanding, x_i represents a sequence. If x_i is a sequence, the difference between the proposed method and MLE is adding a multiplier to the gradient, as shown in equation 9?\n\n2. What are the empirical benefits of the theoretical analyses in the paper for further applications?", "limitations": "The author didn't mention any societal impact of the proposed method.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposed a variant of MLE loss for text generation, which is novel.\n\n2. The assumptions for theoretical analysis are reasonable and commonly appeared in real-world application scenarios.\n\n3. The paper is clearly stated and easy to follow.", "weaknesses": "1. About Baselines: The method should have been evaluated on more baselines. For AR model, the author only compared the proposed method with the vanilla transformer. For NAR, only two baselines (CMLM, CTC) are compared. The method needs to be applied on more powerful baselines (e.g. [1]) to show its effectiveness and universality.\n\n2. About the f function: The author proposed two practical forms for the f function. However, in the experiments, only the exponential form is discussed. \n\n3. About the theoretical analysis: The analyses seem trivial, and provided little instruction on the practical design of the loss function, or to guarantee the effectiveness.\n\nReference:\n[1]. Bridging the Gap between Training and Inference for Neural Machine Translation, ACL 2019\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We provide discussions and explanations about your concerns as follows.\n\n> About Baselines: The method should have been evaluated on more baselines. For AR model, the author only compared the proposed method with the vanilla transformer. For NAR, only two baselines (CMLM, CTC) are compared. The method needs to be applied on more powerful baselines (e.g. [1]) to show its effectiveness and universality.\n\nWe appreciate your suggestion to evaluate our method on more baselines. In Table 2 of the attached PDF document, we present the outcomes of implementing the proposed convex loss on data generated via sequence-level distillation. Upon applying the convex loss on the distilled training data, we observed a significant improvement in the NAR model, while the effect seems minor on the AR model. Particularly, NAR models benefit from convex loss by acquiring a more concentrated distribution to address their inherent expressive limitations. The simplification of the data distribution, although beneficial, might not suffice for NAR models to achieve this objective. The introduction of our proposed convex loss serves to further facilitate this desired outcome.\nMoreover, we would like to emphasize that our research is centered around the development of a theory-grounded algorithm to learn a concentrated distribution from a multi-modal data distribution. Therefore, we did not compare our methods with other training-augmented techniques, such as oracle-based schedule sampling in [1].\n\n[1]. Bridging the Gap between Training and Inference for Neural Machine Translation, ACL 2019\n\n\n> About the f function: The author proposed two practical forms for the f function. However, in the experiments, only the exponential form is discussed.\n\nThank you for your comment regarding the f function. We apologize for any confusion caused by the presentation. In our paper, we indeed explored both forms of the f function. However, due to space constraints, we included the results and discussion for the power form in the supplementary materials.\n\nWe found that the power form of the f function encountered some difficulties during training, leading to worse performance compared to the exponential form. In the revised version of the paper, we will make it clearer that the results for both forms of the f function are provided and ensure that the supplementary materials are more easily accessible for readers interested in the details of the power form.\n\n> About the theoretical analysis: The analyses seem trivial, and provided little instruction on the practical design of the loss function, or to guarantee the effectiveness. empirical benefits of the theoretical analyses\n\nThank you for your feedback regarding the theoretical analysis in our paper. We would like to emphasize that our analysis is non-trivial and contributes novel insights to the field. Our work presents four theorems accompanied by detailed proofs, which were previously unknown in the field.\n\nOur theoretical analysis provides valuable practical guidance for the design of loss functions. It demonstrates the effectiveness of convex functions in learning a one-hot distribution, while also highlighting the limitations of directly optimizing convex functions due to gradient-related challenges. This understanding informed the development of our composite loss function, a key step of our work.\n\nFurthermore, our theoretical analysis guarantees the effectiveness of the composite loss function by showing that it yields a distribution that assigns higher probability to outputs with greater likelihood under p_data. This property is particularly desirable for closed-ended text generation tasks, where generating the most appropriate response is the primary goal.\n\nWe hope that this response clarifies the importance and contributions of our theoretical analysis. We appreciate your feedback and will continue refining our paper to make our contribution clear.\n \n> is x_i a sequence or a token?  the difference between the proposed method and MLE is adding a multiplier to the gradient, as shown in equation 9?\n\nThank you for your question regarding Equation 10. We would like to confirm that x_i indeed represents a sequence, not a token. Your understanding of the difference between the proposed method and MLE is correct. In our method, we add a multiplier to the gradient, which can be interpreted as a weight for the loss.\n\nAs discussed in lines 242-247 and shown in Equation 13, the gradient of the convex-composition function is f'g·g'. The additional term f'g serves as a weight for the loss. Given that f is a convex function and g is an increasing function, the weight f'g is larger for more probable samples, thereby directing the model's focus towards generating outputs with high probabilities."}, {"review_id": "fUH7rg2RR2", "reviewer": "Reviewer_pbsn", "summary": "The objective of the authors in this paper is to develop a machine translation model that doesn't need to learn the complete output distribution given the input, but instead focuses on highly probable outputs based on p_data. To achieve this, the authors suggest replacing the logarithm in Maximum Likelihood Estimation (MLE) with a convex increasing function. They demonstrate that optimizing such a function leads to a desirable one-hot distribution. However, the authors acknowledge that training such a model can be challenging due to the gradient being proportional to the probability of the samples, which can be quite low for sequences.\n\nTo address this issue, the authors propose combining a concave function (e.g., log in MLE) with an increasing convex function. They illustrate that optimizing this composite function yields a distribution that assigns higher probability to outputs with greater likelihood under p_data.\n\nThe authors apply their loss function to train both autoregressive and non-autoregressive machine translation models. They demonstrate that the resulting greedy and beam outputs from these models surpass the performance of the greedy and beam outputs produced by MLE.\n\nUpdate: I have read the author's response. I am satisfied with the response and since I had already given an accept to the paper, I will keep my ranking.", "questions": "In line 149, the statement \"If Lf has multiple optimal distributions, we simply arrange them in alphabetical order and take the first one to be pf\" raises concerns. It is not feasible to arrange distributions in alphabetical order since the space of distributions is not countable, even if the sample space is countable.\n\nRegarding step number 3 in equation (18) of the Appendix, it is unclear how the optimality of pg implies its validity. Further clarification is needed to establish the reasoning behind this step and its connection to the optimality of pg.", "limitations": "The authors have discussed that convex functions of probability are hard to train. It will be good to have a similar discussion about the composition loss functions discussed in the paper.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The authors' approach to addressing the challenge of learning a peaky distribution is innovative. The novel convex-composition loss functions introduced in this paper are particularly interesting within the realm of sequence generation.\n\nThe paper is highly compelling and offers an intuitive reading experience. The authors' strategy of incorporating an increasing convex function to emphasize high-probability outputs under p_data is remarkable.\n\nThe theoretical concepts discussed in this paper hold immense potential impact, even if the direct application to machine translation does not yield substantial results.", "weaknesses": "Equation (5) may not hold true for non-autoregressive models as these models employ latent variables that interconnect the output tokens, as mentioned in [1] [2].\n\nThe term \"Convex loss functions\" in section 3.2 is misleading since the negative logarithm used in MLE is, in fact, a convex loss function. The authors actually refer to \"Loss functions with convex f.\"\n\nThere is an inaccuracy in line 200 where the authors state that the standard loss function in maximum likelihood estimation is the log-probability. This is incorrect. The loss function is actually the negative log-probability, which is convex and not concave.\n\nEquation (12) can only utilize odd values for the power function, as for even values, the function decreases within the range of g(p_theta(x)).\n\n[1] Gu, J., et al. \"Non-autoregressive neural machine translation.\" International Conference on Learning Representations (ICLR). 2018.\n[2] Gu, Jiatao, and Xiang Kong. \"Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade.\" Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We provide discussions and explanations about your concerns as follows.\n\n> Equation (5) may not hold true for non-autoregressive models as these models employ latent variables that interconnect the output tokens.\n\nThank you for your comment regarding Equation (5) and its applicability. We acknowledge that the equation may not hold true for latent models, as they employ latent variables that interconnect the output tokens. In section 2.2, we focus on introducing the background knowledge about text generation models, which is why we used the original form in Equation (5). We appreciate your feedback and will clarify this point in the revised version of the paper to avoid any confusion.\n\n> The term \"Convex loss functions\" in section 3.2 is misleading since the negative logarithm used in MLE is, in fact, a convex loss function. The authors actually refer to \"Loss functions with convex f\". There is an inaccuracy in line 200 where the authors state that the standard loss function in maximum likelihood estimation is the log-probability. This is incorrect. The loss function is actually the negative log-probability, which is convex and not concave.\n\nThank you for pointing out this issue. In the revised version of the paper, we will avoid using the term \"Convex loss functions\" and instead refer to them as \"Loss functions with convex f\" to prevent any confusion. Additionally, we will correct the statement in line 200 to \"The standard loss function in maximum likelihood estimation is the negative log-probability, where log-probability is a concave function.\"\n\n> Equation (12) can only utilize odd values for the power function, as for even values, the function decreases within the range of g(p_\\theta(x)).\n\nThank you for pointing out the issue with the power function in Equation (12). We acknowledge that there was a writing mistake in our paper, which has been fixed in Appendix D. As you correctly noted, the power function $f(x)=x^k$ is not suitable for our purpose due to its behavior within the range of $\\log(p)$. For even values of k, the function decreases, and for odd values of k, it is concave. \n\nThe correct form should be $f(x)=-(-x)^k$, with $0 < k < 1$, which is increasing and convex when $x < 0$. The corrected form and its experiment results are presented in Appendix D.\n\nWe apologize for the confusion caused by this mistake and appreciate your attention to detail. We hope this clarification resolves your concern.\n\n> In line 149, the statement \"If Lf has multiple optimal distributions, we simply arrange them in alphabetical order and take the first one to be pf\" raises concerns. It is not feasible to arrange distributions in alphabetical order since the space of distributions is not countable, even if the sample space is countable.\n\nWe appreciate your attention to detail and agree that our original statement was not appropriate. We will update the statement as follows: \"If Lf has multiple optimal distributions, we use pf to denote an arbitrary optimal distribution. This choice does not harm the generality of our analysis, as the subsequent discussion is applicable to all optimal distributions.\" \n\n> Regarding step number 3 in equation (18) of the Appendix, it is unclear how the optimality of pg implies its validity. Further clarification is needed to establish the reasoning behind this step and its connection to the optimality of p_g.\n\nThank you for your valuable question. We apologize for any confusion caused by our presentation, as we skipped too many steps in the derivation of Equation (18).\n\nWe can prove Equation (18) by contradiction, which is simialr to the proof of Theorem 2. If Equation (18) does not hold, then we can further reduce the loss, which contradicts the optimality of $p_g$. Specifically, we can let $p_g'(x_i)=p_g(x_i)-\\alpha$, and  $p_g'(x_j)=p_g(x_j)+\\alpha$. The gradient of loss with respect to $\\alpha=0$ is $p_{data}(x_i) \\cdot g'(p_{g}(x_i)) - p_{data}(x_j) \\cdot g'(p_{g}(x_j)) < 0$, so we can further reduce the loss with a positive $\\alpha$, proving Equation (18) by contradiction.\n\nWe hope this clarification resolves your concern. We will update the appendix in the revised version of the paper to include the missing steps and ensure a clearer presentation of the proof."}, {"review_id": "JR1bYzIZPj", "reviewer": "Reviewer_hk8n", "summary": "Authors have proposed the novel way set of loss functions to learn neural sequence model in both AR and NAR factorization. The core idea is to sharpen the model distribution so that is comes closer to the one-hot target distributions which are typically used for closed-form generation task such as NMT. They have theoretically confirmed how the usage of composition of convex and concave functions with model's probability will result in the aforementioned sharper model's induced distribution. \nThey performed set of experiments in both AR and NAR setting. AR settings shown that the proposed approach narrows down the gap between greedy and beam search for NMT and slightly improve scores on summarization task. NAR results shows improvements in translation quality.", "questions": "# tuning of k in AR vs NAR\n\n* you said you chose k = 1 for AR and tuned K for NAR experiments. And you only shown the ablation with k for NAR task. Why is so? I suspect there are some interesting pieces about k in AR setting. Could you share any insights? Also I believe it is crucial to tell that you tuned k for NAR experiments as it decreases the practicality of this method in real life. \n\n* from training details I see that you didnt use label smoothing for NAR modeling with convex objective. Why is so? I think that the answer \"it works better empirically\" is not good enough here as this method inherently related to making sharper distributions while smoothing is doing the opposite. So I tend to believe there is some interesting relationship there.", "limitations": "Authors did not discuss limitations in the main text of their submission.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed apprach is very intuitive and makes sense from the optimization point of view and given the motivation of getting a sharper distribution. It is also easy to implement.\n\n* The proposed framework of convex/concave composition allows community to extend this by examining other potential candidates for the loss function.\n\n* Assuming that the NAR NMT baseline results are strong baselines, the proposed approach substantially improves the NAR prediction quality.", "weaknesses": "# Experimental analysis\n\n* The resulting BLEU scores for AR setting are too close and not convincing. I think it is required to perform multiple random seed training or other way of estimating the translation performance variance to verify the improvement. In its current form I would be more convinced that AR modeling performance is nearly the same in terms of translation quality.\n\n# Related work\n\nThere is a recent work (I am not an author!) around modeling sharper distributions by reformulating the sequential random variables to be one-hot: https://arxiv.org/abs/2205.00704. I believe it is very similar in terms of the underlying motivation and observed improvements. Given that this was not mentioned in this work, I am not sure how well other related work is covered in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We provide discussions and explanations about your concerns as follows.\n\n> The resulting BLEU scores for AR setting are too close and not convincing. I think it is required to perform multiple random seed training or other way of estimating the translation performance variance to verify the improvement. In its current form I would be more convinced that AR modeling performance is nearly the same in terms of translation quality.\n\nThank you for your feedback regarding the evaluation on AR setting. Following your suggestion, we plan to perform multiple random seed training and report the results in the revised version of the paper. We would also like to emphasize that the current version of our paper has already demonstrated the positive effect of our method on autoregressive models:\n\n1. We conducted AR experiments on both machine translation and text summarization tasks. While the performance is close in the machine translation with beam search setting, our method shows improvements in other settings.\n\n2. Figure 2 demonstrates a consistent improvement of our method across different beam sizes. The gap between our method and the baseline narrows as the beam size increases, which complies with our theoretical analysis and suggests that the convex loss has a positive impact on translation quality.\n\n3. In the attached response PDF, we provide the AR performance under different values of k. We observe consistent improvements when k is close to 1, especially for greedy search.\n\nWe hope this response addresses your concern and demonstrates the effectiveness of our proposed method. If you have any further questions or concerns, please feel free to bring them up for discussion.\n\n> There is a recent work (I am not an author!) around modeling sharper distributions by reformulating the sequential random variables to be one-hot: https://arxiv.org/abs/2205.00704. I believe it is very similar in terms of the underlying motivation and observed improvements. Given that this was not mentioned in this work, I am not sure how well other related work is covered in this paper.\n\nWe appreciate your reference to the recent work (https://arxiv.org/abs/2205.00704) that shares a similar motivation with our paper in terms of modeling sharper distributions. We apologize for not mentioning this work in our paper and would like to express our gratitude for bringing it to our attention.\n\nUpon careful analysis of the method used in this paper, we found that it indeed shares a similar motivation with our work. However, the proposed method in the mentioned paper differs significantly from ours. The method in the paper reformulates the distribution of the model at the word level, which may not directly facilitate the model in finding high probability sentences at the sentence level. In contrast, our method directly trains the model to focus on highly probable sentences.\n\nNonetheless, we believe that the mentioned paper is an interesting and relevant work. In the revised version of our paper, we will include a discussion of this related work and highlight the differences between our method and the one proposed in the mentioned paper. Thank you once again for your valuable feedback.\n\n> you said you chose k = 1 for AR and tuned K for NAR experiments. And you only shown the ablation with k for NAR task. Why is so? I suspect there are some interesting pieces about k in AR setting. Could you share any insights? Also I believe it is crucial to tell that you tuned k for NAR experiments as it decreases the practicality of this method in real life.\n\nThank you for your question regarding the choice of k. The reason we did not show the ablation study with k for the AR setting is actually the space constraints. In the attached response PDF, we supplement the results for different values of k in the AR setting. Our results show that k=1 performs the best, while other choices of k, such as k=0.5 or 0.75, also bring improvements, especially in the greedy search setting.\n\nFor the NAR experiments, we tuned k from the set {1, 2, 3, 5, 8} on the validation set. We also provide the test set results for different values of k in Figure 2. In the revised version of the paper, we will include the results of different k in the AR setting to ensure a clearer presentation.\n\n> from training details I see that you didnt use label smoothing for NAR modeling with convex objective. Why is so? I think that the answer \"it works better empirically\" is not good enough here as this method inherently related to making sharper distributions while smoothing is doing the opposite. So I tend to believe there is some interesting relationship there.\n\nThank you for your insightful question regarding the relationship between label smoothing and convex objective. We have provided the experimental results in our attached response PDF, showing that autoregressive models perform better with label smoothing, while non-autoregressive models work better without it. As you mentioned, this experimental phenomenon is indeed interesting. Our interpretation is as follows:\n\nLabel smoothing serves as a regularization technique, helping the model generalize better on test data. Although we aim for a sharp distribution in autoregressive models to facilitate approximate search algorithms in identifying the most likely output, the importance of regularization may outweigh the need for a sharp distribution, making label smoothing beneficial.\n\nHowever, for non-autoregressive models, they are theoretically unable to fit the data distribution, let alone overfit the dataset. In this case, finding a sharp distribution for NAR models to fit becomes more important than preventing overfitting with label smoothing. As a result, NAR models prefer not to use label smoothing."}, {"review_id": "dh4E6EILzX", "reviewer": "Reviewer_CKPv", "summary": "Based on the ideas of convex optimization, the paper proposes a new loss function for the training of text generation models. The experimental results show that it has certain advantages over MLE, whether in autoregressive models or non-autoregressive models.", "questions": "The original goal of Convex Loss is to learn one-hot, which is more consistent with the greedy/beam search decoding strategy. However, this contradicts gradient-based optimizers, so the final loss balances the goal of one-hot and gradient, which is formula 12. My questions are:\n\n1. Compared to the gradient of MLE, the gradient of formula 12, which is formula 13, seems to be smoother. This appears to make the model's output smoother rather than closer to one-hot. Doesn't this contradict the starting point of Convex Loss?\n\n2. Is it possible to propose a quantitative metric to measure the impact of these two aspects? Based on this metric, we might be able to directly obtain an optimal solution, instead of introducing an adjustable hyperparameter like in formula 12. Because as long as an adjustable hyperparameter is introduced, the experimental results are likely to improve to a certain extent, which is a very commonplace feature and seems to waste the previous detailed theoretical analysis.", "limitations": "I couldn't find any discussion about limitations in the main text and the appendix of the paper. Perhaps the authors should include it in the paper.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper has done a relatively detailed theoretical derivation;\n\n2. The final obtained loss is simple and practical;\n\n3. Experimental results show its superiority over MLE in both autoregressive and non-autoregressive models;\n\n4. It has narrowed the gap between greedy search and beam search.", "weaknesses": "1. Some key experimental hyperparameters were not provided, such as the value of T, which I could not find in the main text or the appendix;\n\n2. The experimental part only compared with MLE, without comparing other loss functions (such as other classification losses used to approximate accuracy);\n\n3. The notation is a bit confusing (for example, k is used in both formula 11 and theorem 3, but the meanings are not the same);\n\n4. Since the goal of the Convex Loss is one-hot rather than learning distribution, it seems that it can only be used to improve the greedy/beam search, and is not applicable to the decoding strategy of random sampling.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We provide discussions and explanations about your concerns as follows.\n\n> Some key experimental hyperparameters were not provided, such as the value of T, which I could not find in the main text or the appendix; The notation is a bit confusing (for example, k is used in both formula 11 and theorem 3, but the meanings are not the same);\n\nWe apologize for the confusion caused by our presentation. T represents the length of target sentence. In the revised version of the paper, we will ensure that all key experimental hyperparameters are adequately presented and explained. To avoid any ambiguity, we will change the notation of \"k\" to \"m\" in Theorem 3 and Theorem 4.\n\nWe appreciate your attention to detail and valuable feedback, which helps us improve the clarity of our paper.\n\n> Compared to the gradient of MLE, the gradient of formula 12, which is formula 13, seems to be smoother. This appears to make the model's output smoother rather than closer to one-hot. Doesn't this contradict the starting point of Convex Loss?\n\nThank you for your question regarding the gradient of our loss. We understand your concern and would like to clarify the misunderstanding.\n\nThe gradient of formula 12, $fg(p(x))$, is $f'g(p(x)) \\cdot g'(p(x))$ (line 242), where formula 13 represents only a part of the gradient. Compared to the gradient of MLE, $g'(p(x))$, our gradient has an additional term, $f'g(p(x))$ (formula 13), which can be understood as the weight for the MLE loss. With $f$ being a convex function, $f'$ is an increasing function, so $f'g(p(x))$ will be larger for samples with greater likelihood. This encourages the model to assign higher probability to the most probable outputs, which aligns with the starting point of Convex Loss.\n\nWe hope this clarification resolves your concern. If you have any further questions, please feel free to bring them up for discussion.\n\n> Since the goal of the Convex Loss is one-hot rather than learning distribution, it seems that it can only be used to improve the greedy/beam search, and is not applicable to the decoding strategy of random sampling.\n\nThank you for your comment regarding the applicability of our approach. You are correct that our method primarily focuses on closed-ended generation tasks, such as machine translation and text summarization, where convex loss can help greedy/beam decoding find high-probability outputs. While our approach ensures a sharper optimal distribution, it currently cannot accurately control the shape of the distribution, which limits its applicability to open-ended generation tasks. \n\nNevertheless, we believe that further research in this direction could lead to more precise control over the shape of the optimal distribution. This could potentially improve commonly used sampling methods (e.g., top-k sampling, nucleus sampling, temperature sampling) in open-ended generation by enabling more controllable and effective generation of desired results.\n\n> Is it possible to propose a quantitative metric to measure the impact of these two aspects? Based on this metric, we might be able to directly obtain an optimal solution, instead of introducing an adjustable hyperparameter like in formula 12. Because as long as an adjustable hyperparameter is introduced, the experimental results are likely to improve to a certain extent, which is a very commonplace feature and seems to waste the previous detailed theoretical analysis.\n\nThank you for your very insightful question regarding the possibility of proposing a quantitative metric to measure the impact of the two aspects you mentioned, which are indeed our major principles of loss designing. Your question highlights the subtle trade-off between guiding the model to learn the desirable one-hot distribution and ensuring that the loss is not difficult to minimize.\n\nFinding a quantitative metric to balance these two aspects and obtain an optimal solution would indeed be a very valuable contribution to the field. However, due to the complexity and challenges associated with this task, we are unable to provide a solution within a short period of time. We consider this an important direction for future work and will explore this idea in our ongoing research."}]

Author rebuttal:
We thank reviewers for their valuable feedback and insightful comments on our paper. In response to the concerns and recommendations, we have conducted additional experiments and provided supplementary results in the attached response PDF. In Table 1, we present the results of tuning the hyper-parameter k for autoregressive models. In Table 2, we show the results of models trained with and without knowledge distillation. In Table 3, we report the outcomes of combining our method with text VAEs. In Table 4, we provide the results of models trained with and without label smoothing. Additionally, in Figure 1, we illustrate the performance of autoregressive models trained with and without knowledge distillation across different beam sizes.



Meta-review: The paper studies some constructions for loss functions that mitigate some of the downsides of standard MLE losses in text generation. Through careful function composition, useful loss functions are obtained.

Reviewers find the construction novel and interesting both in theory and in practice. The experimental section has some limitations that the authors have improved with some additional experiments, including one on VAEs; overall I believe the evaluation is relatively convincing. The writing is mostly clear, but I strongly urge the authors to clarify the ambiguity of convexity/concavity wrt common use in the context of classification losses, as pointed out by the reviewers. (The MLE / "cross-entropy" loss is convex w.r.t. the scores).

Overall, this work is of interest to the community and I recommend acceptance. Please take into account all feedback.
Decision: Accept (poster)