Title: Beyond MLE: Convex Learning for Text Generation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Maximum likelihood estimation

Convex function

Text generation
Abstract: Maximum likelihood estimation (MLE) is a statistical method used to estimate the parameters of a probability distribution that best explain the observed data. In the context of text generation, MLE is often used to train generative language models, which can then be used to generate new text. However, we argue that MLE is not always necessary and optimal, especially for closed-ended text generation tasks like machine translation. In these tasks, the goal of model is to generate the most appropriate response, which does not necessarily require it to estimate the entire data distribution with MLE. To this end, we propose a novel class of training objectives based on convex functions, which enables text generation models to focus on highly probable outputs without having to estimate the entire data distribution. We investigate the theoretical properties of the optimal predicted distribution when applying convex functions to the loss, demonstrating that convex functions can sharpen the optimal distribution, thereby enabling the model to better capture outputs with high probabilities. Experiments on various text generation tasks and models show the effectiveness of our approach. It enables autoregressive models to bridge the gap between greedy and beam search, and facilitates the learning of non-autoregressive models with a maximum improvement of 9+ BLEU points. Moreover, our approach also exhibits significant impact on large language models (LLMs), substantially enhancing their generative capability on various tasks. Source code is available at \url{https://github.com/ictnlp/Convex-Learning}.

Beyond MLE: Convex Learning for Text Generation

Abstract

Maximum likelihood estimation (MLE) is a statistical method used to estimate the parameters of a probability distribution that best explain the observed data. In the context of text generation, MLE is often used to train generative language models, which can then be used to generate new text. However, we argue that MLE is not always necessary and optimal, especially for closed-ended text generation tasks like machine translation. In these tasks, the goal of model is to generate the most appropriate response, which does not necessarily require it to estimate the entire data distribution with MLE. To this end, we propose a novel class of training objectives based on convex functions, which enables text generation models to focus on highly probable outputs without having to estimate the entire data distribution. We investigate the theoretical properties of the optimal predicted distribution when applying convex functions to the loss, demonstrating that convex functions can sharpen the optimal distribution, thereby enabling the model to better capture outputs with high probabilities. Experiments on various text generation tasks and models show the effectiveness of our approach. It enables autoregressive models to bridge the gap between greedy and beam search, and facilitates the learning of non-autoregressive models with a maximum improvement of 9+ BLEU points. Moreover, our approach also exhibits significant impact on large language models (LLMs), substantially enhancing their generative capability on various tasks. Source code is available at https://github.com/ictnlp/Convex-Learning .

1 Introduction

Text generation is an important field within natural language processing that aims to generate human-like texts for specific tasks. It can be broadly divided into two categories: open-ended and closed-ended text generation. Open-ended tasks encourage the model to produce novel and diverse outputs without a specific expected outcome or structure. Representative tasks in this category include language modeling [41, 7], chatbot [64], storytelling [13], etc. In contrast, closed-ended tasks are more constrained and adhere to specific rules or formats. Representative tasks in this category include machine translation [8, 2], text summarization [45], etc.

In recent years, learning neural probabilistic models with maximum likelihood estimation has become the dominant approach for both open-ended and closed-ended text generation [5, 2, 7]. Maximum likelihood estimation (MLE) is a statistical method used to estimate the parameters of a probability distribution that maximize the likelihood of the observed data [33]. Since directly maximizing the

likelihood can be numerically unstable, it is common to minimize the negative log-likelihood loss function, which is also referred to as cross-entropy loss. It is equivalent to minimizing KullbackLeibler (KL) divergence [25, 1] between the true distribution and the predicted distribution, which ensures that the optimal predicted distribution is the true data distribution.

While MLE has gained widespread adoption, it does not always align with the objective of text generation, especially for closed-ended text generation tasks such as translation and summarization. In these tasks, the goal of the model is to generate the most appropriate response, rather than producing diverse outputs. For example, in the task of machine translation, though there may exist multiple translations for the same input sentence, we usually want the most accurate and commonly used translation result. Generally speaking, the desired output can be mathematically defined as the output with the maximum probability in the true data distribution, which does not necessarily require the model to estimate the entire data distribution with MLE.

In terms of generating the most probable output, MLE is also suboptimal for current neural text generation models. For autoregressive models, even if the model can perfectly fit the data distribution, it still requires decoding algorithms like greedy or beam search to generate the output, which do not guarantee the exact result with the maximum probability. To our knowledge, only Stahlberg and Byrne [55] proposed an exact decoding algorithm for autoregressive models, but it is too slow for practical applications. The limitation in exact decoding can be overcome by non-autoregressive models [17, 14], which independently predict the output at each position. However, fitting the data distribution by MLE is theoretically beyond the ability of non-autoregressive models [21]. In light of these issues, alternative training objectives should be considered to better address the specific requirements of text generation without incurring the shortcomings associated with MLE.

Based on the analysis above, MLE is suboptimal that it trains the model to estimate the data distribution, which complicates the training and decoding of text generation models. It would be advantageous if the model could converge to a sharper optimal distribution under an alternative loss function, as this would enable autoregressive models to easily find high probability outputs and also allow non-autoregressive models to converge to a better distribution. Exploring loss functions with this characteristic could lead to improved performance and efficiency of neural text generation models, particularly for closed-ended tasks.

In this paper, we propose a novel class of training objectives based on convex functions, which help text generation models capture highly likely outputs without estimating the entire data distribution. Intuitively, the concave shape of log-probability discourages the model from assigning a large prediction probability to a single sample, as the marginal benefit diminishes with increasing probability. If the learning criterion is convex or less concave, then intuitively the model would converge to a sharper distribution, which is the motivation of this work. We further investigate the theoretical properties of the optimal predicted distribution when applying convex functions to the loss. Our findings demonstrate that convex functions can sharpen the optimal distribution, allowing the model to better capture outputs with high probabilities.

Experiments on various closed-ended text generation tasks and models show the effectiveness of our approach. Specifically, it enables autoregressive models to bridge the gap between greedy and beam search, and facilitates the learning of non-autoregressive models with a maximum improvement of 9+ BLEU points. Moreover, our approach also exhibits significant impact on large language models, substantially enhancing their generative capability on various tasks.

2 Preliminaries

2.1 Maximum Likelihood Estimation

Maximum likelihood estimation (MLE) is a statistical method used to estimate the parameters of a probability distribution that best explain the observed data. This is achieved by maximizing a likelihood function so that the observed data is most probable. Since directly maximizing the likelihood can be numerically unstable, it is common to minimize the negative log-likelihood loss function, also referred to as cross-entropy loss. Given the data distribution p data and a parametric model with parameters θ , MLE training minimizes:

MLEcan be viewed as an attempt to minimize KL divergence between the true underlying distribution of the data p data and the estimated distribution p θ provided by the model [1]. The following equation reveals the relationship between MLE loss and KL divergence:

where H data is the Shannon entropy of the data distribution, which remains constant with respect to the model parameter θ . Therefore, the MLE loss and KL divergence share the same minimizer that the estimated distribution p θ equals to the true distribution p data . By minimizing the MLE loss, the predicted distribution is encouraged to be as close as possible to the true data distribution. In the context of text generation, this ensures that the model learns to generate text that closely resembles the text in the training data.

The above discussion can be extended to conditional scenarios. In such cases, the log-likelihood loss can be expressed as:

where c represents the input context. This extension allows the MLE framework to accommodate a wide range of text generation tasks such as machine translation, summarization, dialogue system, etc.

2.2 Text Generation Models

Based on how the sequence probability is factorized, neural text generation models can be broadly categorized into two types: autoregressive (AR) models and non-autoregressive (NAR) models. Autoregressive models generate text sequentially, predicting one token at a time based on the previously generated tokens. In AR models, the probability of generating a sequence x = ( x 1 , x 2 , ..., x T ) is factorized as:

where c represents the input context. With the autoregressive decomposition, AR models can perfectly fit the data distribution if it satisfies p θ ( x t | x <t , c ) = p data ( x t | x <t , c ) for every x, c, t . In inference, ARmodels can perform deterministic decoding like greedy/beam search to generate a high probability output, or sample from the model distribution to generate diverse outputs.

In contrast to autoregressive models, non-autoregressive models [17, 14] generate text in parallel, predicting all tokens simultaneously without conditioning on previously generated tokens. This approach can significantly speed up the generation process, as it removes the sequential dependency between tokens. In NAR models, the generation probability is factorized as:

Unlike AR models, NAR models can efficiently find the most likely output by using argmax decoding at each step. However, MLE is beyond the ability of NAR models since they are theoretically unable to fit the data distribution. Huang et al. [21] showed that KL divergence from p θ to p data is bounded by a non-negative constant:

The MLE loss is minimized when NAR models achieve the equality by ignoring sequential dependency and predicting p θ ( x t | c ) = p data ( x t | c ) . Therefore, NAR models trained with MLE often suffer from reduced performance, as they lack the ability to model dependencies between tokens.

3 Approach

In this section, we will explore alternative loss functions for the learning of text generation models, which overcomes the limitations of MLE. We begin by introducing a general learning framework that allows arbitrary loss functions. Next, we discuss the benefits of applying convex functions to the loss within this framework. Finally, we use convex functions to construct composite loss functions, which can be used in practical text generation scenarios.

3.1 General Learning Framework

For simplicity of notation, we omit condition c in the probabilities, with the data distribution represented as p data ( x ) and the model predicting the distribution p θ ( x ) . The derived theoretical results hold in both unconditional and conditional settings.

First, we introduce the general learning framework for text generation, characterized by the following loss function:

where f is an arbitrary function of the prediction probability p θ ( x ) . We impose some basic requirements on f : (1) The domain of function f should contain the interval (0 , 1] ; (2) f must be differentiable on the interval (0 , 1] since we need to compute its gradient; and (3) f should be an increasing function on (0 , 1] to encourage the model to generate the current sample. Under this framework, we can explain maximum likelihood estimation as a special case of f = log , which is a differentiable and increasing function within the interval (0 , 1] . We also establish some reasonable assumptions:

Assumption 1 (Countability of Sample Space) . The sample space X is countable, which allows us to enumerate all samples in a systematic way. Note that |X| can be either finite or infinite.

Assumption 2 (Distinctness of Sample Probabilities) . In the data distribution p data , the probabilities of all samples are distinct, which allows us to arrange samples in a strictly descending order of sample probabilities. 3

Assumption 1 naturally holds in text generation tasks due to the inherent discreteness of textual data. With a countable sample space and probabilities lying in a dense subspace of real number, it is reasonable to assume the distinctness of sample probabilities. While Assumption 2 is not strictly necessary, removing it would introduce many corner cases that would complicate the subsequent analysis. In the following, we will assume that Assumptions 1-2 always hold, and we arrange the samples such that p data ( x 1 ) > p data ( x 2 ) > · · · > p data ( x i ) > · · · . Since the sample space X is countable, the loss function in Equation 7 can be reformulated as follows:

In this framework, our primary focus is to analyze the probability distribution p θ that the model is inclined to predict when the loss function is L f . We use p f to denote the optimal distribution that minimizes the loss L f , which represents the expected outcome of the model. If L f has multiple optimal distributions, we use p f to denote an arbitrary optimal distribution. This choice does not harm the generality of our analysis, as the subsequent discussion is applicable to all optimal distributions. Currently, it is only established that the optimal distribution for the MLE loss L log is the data distribution p log = p data . For other loss functions, the following theorem reveals a general property of the optimal distribution. With samples organized in descending order of their probabilities in the data distribution, i.e., p data ( x 1 ) > p data ( x 2 ) > · · · > p data ( x i ) > · · · , the optimal distribution of an arbitrary function f maintains this order as p f ( x 1 ) ≥ p f ( x 2 ) ≥ · · · ≥ p f ( x i ) ≥ · · · . The proofs for the theorems presented in this paper can be found in Appendix A.

Theorem 1. Given an arbitrary differentiable and increasing function f , the optimal distribution p f satisfies p f ( x 1 ) ≥ p f ( x 2 ) ≥ · · · ≥ p f ( x i ) ≥ · · · .

In the following, we will further explore the properties of optimal distributions associated with specific loss functions.

3.2 Loss with Convex Function

In certain text generation scenarios that require precise and deterministic outputs, it is beneficial for the model to converge to an optimal distribution that is sharper than the data distribution. In

this section, we demonstrate that this objective can be achieved by employing convex functions as learning criterion.

The MLE loss function is based on log-probability, which is a concave function whose gradient decreases as the probability increases. The concave shape of the learning criterion prevents the model from assigning a large prediction probability to a single sample, since the marginal benefit diminishes as the probability increases. If function f is convex, then intuitively the model would converge to a sharper distribution. The following theorem validates this intuition, which shows that the optimal distribution p f is a one-hot distribution when f is convex.

Theorem 2. If f is an increasing convex function on [0 , 1] , then the optimal distribution p f is a one-hot distribution that p f ( x 1 ) = 1 and p f ( x i ) = 0 , i > 1 .

The one-hot characteristic of the optimal distribution is advantageous for text generation models seeking precise and deterministic outputs. For autoregressive models, they do not need the computationally expensive beam search decoding any more if the model distribution is nearly one-hot. For non-autoregressive models, they suffer from reduced performance under MLE due to their inability to fit the data distribution. However, fitting a one-hot optimal distribution is well within their capabilities, allowing these models to generate high-quality outputs.

However, the direct application of loss with convex functions in training text generation models comes with an inherent limitation, impeding its practical utility. Specifically, the gradient of the parameter θ tends to be very small when the prediction probability approaches 0 , thereby rendering the training process inefficient. The gradient of θ can be formulated as follows:

where we have omitted the autoregressive history condition of p θ ( x t ) for simplicity. The equation above indicates that the gradient is proportional to the sentence probability p θ ( x ) . In text generation models, the sentence probability p θ ( x ) is the product of token probabilities p θ ( x t ) , which causes p θ ( x ) to be typically close to 0 , especially when the model is newly initialized.

To counter this effect, the gradient f ′ ( p θ ( x )) would need to approach infinity as p θ ( x ) approaches 0 . For instance, the log-probability function has the gradient 1 p θ ( x ) , which offsets the impact of p θ ( x ) such that f ′ ( p θ ( x )) · p θ ( x ) = 1 . However, for an increasing convex function f ( p θ ( x )) whose gradient increases with p θ ( x ) , its gradient must be bounded when p θ ( x ) approaches 0 , leading to an extremely small gradient update for the parameter θ during training. This inherent limitation of loss with convex functions poses a significant hurdle to their practical applications.

3.3 Loss with Convex-composition Function

3.3.1 Theoretical Analysis

In the preceding discussion, we illustrate that while convex functions can induce a desirable onehot optimal distribution, their inherent limitations during training pose significant impediments to practical applications. Consequently, we consider a relaxation of the convexity requirement, with the objective of rendering the function f less concave. This approach aims to obtain an optimal distribution that is sharper than p data , thereby providing a practical solution that augments model performance without sacrificing training feasibility.

The standard loss function in maximum likelihood estimation is the negative log-probability, where log-probability is a concave function that yields a smooth optimal distribution. To render the learning criterion less concave, we propose a convex-composition approach that combines a convex function f with the original concave function g . This composition yields the following loss function:

where f is an increasing convex function and g is an increasing concave function. The objective of this composition is to moderate the concavity of the overall loss function, thereby allowing for a sharper optimal distribution. The subsequent theorem and corollaries outline the theoretical properties associated with the optimal distribution under this function composition framework.

Theorem 3. Let f be an increasing convex function and g be an increasing concave function. Then, there exists a positive integer m such that the following inequalities hold:

Corollary 1. The Shannon entropy of p fg is less than or equal to the Shannon entropy of p g .

Corollary 2. For any n ∈ { 1 , 2 , ... } , the sum of the probabilities of the n most probable samples increases: ∑ n i =1 p fg ( x i ) ≥ ∑ n i =1 p g ( x i ) .

Theorem 3 indicates that the convex-composition loss function tends to allocate higher probabilities to the more probable samples, while simultaneously diminishing the probabilities assigned to less probable ones, resulting in a sharper optimal distribution. Corollary 1 quantitatively establishes this observation, demonstrating that the incorporation of a convex function into the loss function effectively sharpens the optimal distribution, as evidenced by a reduction in the Shannon entropy of p fg compared with p g . Furthermore, Corollary 2 reveals an increase in the cumulative probability of the n most probable samples. Consequently, text generation models are better equipped to capture the highly probable outputs without explicitly modeling the data distribution.

In the above analysis, we only assume the original loss g to be an increasing concave function. By imposing specific conditions on g , we can derive more desirable properties from the optimal distribution p fg , as demonstrated in the subsequent theorem:

Theorem 4. Let f be an increasing convex function and g be an increasing concave function. If g satisfies g ′′′ ( x ) · g ′ ( x ) ≥ g ′′ ( x ) 2 > 0 for all x ∈ (0 , 1) , then the difference between p fg and p g exhibits a monotonic order: p fg ( x 1 ) -p g ( x 1 ) ≥ p fg ( x 2 ) -p g ( x 2 ) ≥ ... ≥ p fg ( x m -1 ) -p g ( x m -1 ) ≥ 0 , where m is the positive integer described in Theorem 3.

This theorem provides a more granular description of the relative difference between p fg and p g . When p fg decreases the probabilities assigned to less probable samples, it tends to reallocate this probability mass to the most probable samples. This enables text generation models to more accurately capture the most probable outputs. Note that the condition g ′′′ ( x ) · g ′ ( x ) ≥ g ′′ ( x ) 2 > 0 is not overly restrictive. For instance, the loss function g = log in MLE readily fulfills this condition:

3.3.2 Practical Applications

The preceding theoretical analysis highlights the effectiveness of function composition. Here we turn to its practical applications and give some examples of convex-composition loss functions. The loss function in maximum likelihood estimation is typically the log-probability, and length normalization is often applied in practical usage, resulting in the loss g ( p θ ( x )) = log( p θ ( x )) T , where T denotes the sentence length. Common choices for increasing convex functions on ( -∞ , 0] include the exponential function f ( x ) = e kx , k ≥ 0 and the power function f ( x ) = -( -x ) k , 0 ≤ k ≤ 1 . Through function composition, we can derive the following losses:

The gradient of the convex-composition function is f ′ ( g ( p θ ( x ))) · g ′ ( p θ ( x )) . Compared to the gradient of the original loss g ′ ( p θ ( x )) , it has an additional term f ′ ( g ( p θ ( x ))) that can be interpreted as a weight for the loss. Given that f is a convex function and g is an increasing function, the weight f ′ ( g ( p θ ( x ))) is larger for more probable samples, thereby directing the model's focus towards

1

2

3

5

8

beam size

26.0

26.4

26.8

27.2

27.6

28.0

BLEU

EN-DE

1

2

3

5

8

beam size

29.5

29.9

30.3

30.7

31.1

31.5

DE-EN

BLEU (MLE)

BLEU (+Convex)

1

2

3

5

8

k

-th Power

9

11

13

15

17

19

21

23

25

BLEU

Vanilla-NAT

1

2

3

5

8

k

-th Power

9

11

13

15

17

19

21

23

25

CMLM

1

2

3

5

8

k

-th Power

9

11

13

15

17

19

21

23

25

CTC

0

5

10

15

20

25

30

35

40

0

5

10

15

20

25

30

35

40

0

5

10

15

20

25

30

35

40

Output NLL

BLEU (+Convex)

Output NLL (+Convex)

BLEU (MLE)

Output NLL (MLE)

generating outputs with high probabilities. Specifically, the loss weights f ′ ( g ( p θ ( x ))) associated with Equation 12 are:

where the exponential function weights the sample by the prediction probability, and the power function weights the sample by the log-probability.

In practical applications, label smoothing [59, 63] is a widely used regularization technique for text generation models. The smoothing loss and log-probability loss are typically combined using a fixed hyperparameter ϵ ls . To preserve the ratio of smoothing loss to log-probability loss, we also apply the weight f ′ ( g ( p θ ( x ))) to the smoothing loss before interpolating it with the convex-composition loss.

4 Experiments

To validate the practical advantages of loss functions with sharper optimal distributions, we conduct experiments on basic autoregressive (AR) models, non-autoregressive (NAR) models, and large language models (LLMs). We evaluate their performance on two representative closed-ended text generation tasks, including neural machine translation and text summarization. Following the theoretical analysis in previous sections, we combine the exponential function with standard log-probability, i.e. L f ( θ ) = -E x ∼ p data ( x ) [ p θ ( x ) k T ] , as our training objective in the following experiments. We have also attempted to combine the power function with log-probability as training objective. We found that the power form encountered some difficulties during training, leading to worse performance compared to the exponential form. Due to the space limit, we leave the results under this setting in Appendix E.

Our theoretical analysis suggests that the model trained by convex-composition loss tends to predict a sharper distribution, in which the probability mass is more heavily allocated to the most probable samples. Such property leads the model becoming more confident about its prediction and facilitates the de-facto maximum a posteriori (MAP) decoding framework in closed-ended text generation tasks. In the following, we will discuss and validate the effects of convexity in the context of AR models, NAR models, and LLMs respectively. More details of settings can be found in Appendix B.

4.1 Effects of Convexity on Autoregressive Models

In the context of autoregressive models, a model distribution trained with a convex-composition loss tends to exhibit fewer modes and a sharper distribution, thereby facilitating the task of approximate search algorithms in identifying the most likely output. We validate this conjecture by investigating the performance of greedy and beam search when trained with standard MLE and convex-composition loss in translation and summarization tasks. For translation task, we vary the beam size from { 1 , 2 , 3 , 5 , 8 } , where beam size 1 can be considered as greedy search.

Figure 1 visualizes the results in terms of BLEU [38], with precise numerical values given in Table 1. We observe a consistent improvement in translation quality when using convex-composition losses compared to MLE, and a similar trend is observed in summarization tasks as detailed in Table 2. These results provide experimental support that the composition with convex function promotes those approximate searching algorithms to perform argmax decoding. Meanwhile, Table 1 exhibits a diminishing gap between greedy search and beam search when equipped with convex-composition loss. This outcome can be attributed to the efficacy of the convex function in reducing the complexity of the model distribution, as described in Theorem 3. Such property amplifies the potential of lightweight approximate decoding algorithms within the autoregressive structure, a desirable trait in the context of modern, computation-intensive autoregressive neural networks.

4.2 Effects of Convexity on Non-autoregressive Models

Non-autoregressive models face the challenge of multi-modality, where fitting a data distribution with multiple target modes exceeds the capabilities of NAR models. Therefore, the mode collapse property of convex-composition loss would be beneficial to NAR models. Likelihood training will force the model to ignore sequential dependency, resulting in disfluency in its output (e.g., token repetition and omission). In contrast, convex-composition loss would encourage model to allocate most of its probability mass to the best among all proper candidates. Such property is able to help NAR model avoid generating a mixture of modes, thereby alleviating disfluency issues.

To demonstrate its effectiveness, we investigate the performance of convex-composition loss on three representative NAR models, including Vanilla-NAT [17], CMLM [14] and CTC [46]. Considering most of the NAR researches are restricted in the field of translation, we only conduct experiments on translation dataset. In addition to translation quality, we also assess the prediction confidence and generation fluency of NAR outputs. The prediction confidence is measured with negative loglikelihood of its generation and the fluency is measured by an external pre-trained language model 4 . We use the PPL value reported by the language model to quantify the fluency of generation. The exponent hyperparameter k is manipulated to adjust the convexity of our composite loss function.

The results are shown in Figure 2 and Table 3. We observe a consistent improvement in translation quality across all NAT models with a maximum improvement of 9+ BLEU points on CMLM. Meanwhile, Figure 2 implies that prediction confidence significantly gains and the gain increases as k gets larger. Such phenomenon reveals a descending trend of model entropy as applying convex function on loss, which is consistent with Corollary 1. More importantly, we note a strong correlation between model entropy and generation fluency in Table 4, providing clear evidence that the mode collapse property of convex function indeed relieves NAR model from multi-modality problem.

4.3 Effects of Convexity on Large Language Models

Large language models have demonstrated remarkable capabilities in various applications, including both open-ended and closed-ended text generation tasks. For open-ended tasks, stochastic decoding methods such as temperature sampling are commonly employed to produce responses. In contrast, deterministic decoding methods like beam search are favored for closed-ended tasks like machine translation [22, 69, 30]. Given that the convex-composition loss enhances the model's ability to identify highly probable sentences, incorporating this loss function into the LLMs' training process would be beneficial to closed-ended generation tasks.

To demonstrate its effectiveness, we assess the performance of LLMs in machine translation (Table 5) and summarization (Table 6). Table 5 reveals that the LLaMA-7B model, incorporating convexcomposition loss, surpasses the baseline model across all language pairs, achieving an average improvement of 1.84 BLEU. Likewise, the LLaMA-13B model with convex-composition loss outperforms the baseline model in three out of four language pairs. Table 6 further demonstrates the effectiveness of our method in text summarization. Due to memory limitations, we are only able to decode the text summarization dataset using the LLaMA-7B model.

5 Related Work

Alternative Loss Functions Maximum likelihood estimation has become the dominant approach for learning text generation models, but it also comes with certain limitations. Various alternative loss functions have been proposed to improve the training process from different perspectives. Regarding the exposure bias problem [42] that autoregressive models are exposed to different distributions during training and inference, Bengio et al. [4], Mihaylova and Martins [32], Zhang et al. [72] proposed to reduce this gap by sampling from the model's own predictions during training. Another issue with text generation models is text degeneration: output text may be bland, incoherent, or gets stuck in repetitive loops [20]. To avoid text degeneration, Dieng et al. [9] proposed a learning criterion termed

reflective likelihood to penalize incoherent outputs, and Welleck et al. [66] proposed unlikelihood training that forces unlikely generations to be assigned lower probability by the model. Additionally, to address the discrepancy between likelihood training and evaluation metrics, loss functions that more directly optimize evaluation metrics are proposed. Ranzato et al. [42] utilized the reinforcement learning technique to train recurrent neural networks with sequence level objectives. Shen et al. [53] proposed to optimize evaluation metrics with minimum risk training. Norouzi et al. [35], Edunov et al. [12] incorporated evaluation metrics into the maximum likelihood training objective. There are also efforts on learning a more focused distribution for text generation models [37, 71, 56]. However, these approaches primarily reformulate the loss function at the word level, which is insufficient for guiding the model towards identifying high-probability sentences at the sentence level. In contrast, our method explicitly trains the model to concentrate on generating highly probable sentences.

Reinforcement Learning Our work aligns closely with reinforcement learning (RL) based training techniques for text generation [67, 58]. While RL techniques typically maximize the expected reward by concentrating the probability mass on the sequence with the highest reward, our approach strives to put all the probability mass on the most likely sequence. RL allows for text generation models to optimize discrete evaluation metrics, which have wide usage in text generation tasks, including machine translation [42, 3], text summarization [39], image captioning [44], dialogue generation [27], etc. Furthermore, RL can be integrated with Generative Adversarial Networks [68] and can leverage human feedback for training [57, 36].

Loss Functions for NAR Models The limitation of maximum likelihood estimation is amplified in non-autoregressive (NAR) models since they inherently lack the capability to fit the data distribution [21]. To address this issue, researchers have developed loss functions specifically designed for NAR models, guiding them towards generating coherent text. Shao et al. [50, 52], Ding et al. [10] proposed to train NAR models with sequence-level objective functions. Ghazvininejad et al. [15], Du et al. [11] relaxed the alignment restriction in the cross-entropy loss. Shao et al. [51], Shao and Feng [49], Ma et al. [31] proposed n-gram based differentiable training objectives to optimize n-gram prediction accuracy. However, these methods lack theoretical guarantees for the shape of optimal distribution.

6 Conclusion

This paper investigates the theoretical properties and practical applications of a novel class of training objectives based on convex functions. Our findings show that convex functions can sharpen the optimal distribution, enabling text generation models to focus on highly probable outputs without having to estimate the entire data distribution. Experiments on various text generation tasks and models verify our theoretical analysis and demonstrate the practical effectiveness of our approach.

References

H. Akaike. Information Theory and an Extension of the Maximum Likelihood Principle , pages 610-624. Springer New York, New York, NY, 1992. ISBN 978-1-4612-0919-5. doi: 10.1007/ 978-1-4612-0919-5_38. URL https://doi.org/10.1007/978-1-4612-0919-5_38 .

D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1409.0473 .

D. Bahdanau, P. Brakel, K. Xu, A. Goyal, R. Lowe, J. Pineau, A. Courville, and Y. Bengio. An actor-critic algorithm for sequence prediction. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id=SJDaqqveg .

S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper_files/paper/2015/file/ e995f98d56967d946471af29d7bf99f1-Paper.pdf .

Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin. A neural probabilistic language model. J. Mach. Learn. Res. , 3:1137-1155, 2003. URL http://jmlr.org/papers/v3/bengio03a. html .

M. Bhandari, P. N. Gour, A. Ashfaq, P. Liu, and G. Neubig. Re-evaluating evaluation in text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 9347-9359, Online, Nov. 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.751. URL https://aclanthology.org/2020.emnlp-main.751 .

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1724-1734, Doha, Qatar, Oct. 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-1179. URL https://www.aclweb.org/anthology/ D14-1179 .

A. B. Dieng, K. Cho, D. M. Blei, and Y. LeCun. Learning with reflective likelihoods, 2019. URL https://openreview.net/forum?id=SJlh2jR9FX .

L. Ding, L. Wang, X. Liu, D. F. Wong, D. Tao, and Z. Tu. Progressive multi-granularity training for non-autoregressive translation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 2797-2803, Online, Aug. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.findings-acl.247. URL https://aclanthology.org/ 2021.findings-acl.247 .

C. Du, Z. Tu, and J. Jiang. Order-agnostic cross entropy for non-autoregressive machine translation. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 2849-2859. PMLR, 2021. URL http://proceedings. mlr.press/v139/du21c.html .

S. Edunov, M. Ott, M. Auli, D. Grangier, and M. Ranzato. Classical structured prediction losses for sequence to sequence learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 355-364, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1033. URL https://aclanthology. org/N18-1033 .

A. Fan, M. Lewis, and Y. Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 889-898, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1082. URL https://aclanthology.org/P18-1082 .

M. Ghazvininejad, O. Levy, Y. Liu, and L. Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 6112-6121, 2019. URL https://www.aclweb.org/anthology/D19-1633 .

M. Ghazvininejad, V. Karpukhin, L. Zettlemoyer, and O. Levy. Aligned cross entropy for non-autoregressive machine translation. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 3515-3523. PMLR, 2020. URL http://proceedings. mlr.press/v119/ghazvininejad20a.html .

J. Gu and X. Kong. Fully non-autoregressive neural machine translation: Tricks of the trade. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 120133, Online, Aug. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021. findings-acl.11. URL https://aclanthology.org/2021.findings-acl.11 .

J. Gu, J. Bradbury, C. Xiong, V. O. K. Li, and R. Socher. Non-autoregressive neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings , 2018. URL https://openreview.net/forum?id=B1l8BtlCb .

K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper_files/paper/ 2015/file/afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf .

G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop , 2015. URL http://arxiv.org/ abs/1503.02531 .

A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration. In International Conference on Learning Representations , 2020. URL https: //openreview.net/forum?id=rygGQyrFvH .

F. Huang, T. Tao, H. Zhou, L. Li, and M. Huang. On the learning of non-autoregressive transformers. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 9356-9376. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/huang22k.html .

W. Jiao, J. tse Huang, W. Wang, X. Wang, S. Shi, and Z. Tu. Parrot: Translating during chat using large language models. arXiv preprint arXiv:2304.02426 , 2023.

Y. Kim and A. M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 1317-1327, Austin, Texas, Nov. 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1139. URL https://aclanthology.org/D16-1139 .

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http: //arxiv.org/abs/1412.6980 .

S. Kullback and R. A. Leibler. On information and sufficiency. The annals of mathematical statistics , 22(1):79-86, 1951.

M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. CoRR , abs/1910.13461, 2019. URL http:// arxiv.org/abs/1910.13461 .

J. Li, W. Monroe, A. Ritter, D. Jurafsky, M. Galley, and J. Gao. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 1192-1202, Austin, Texas, Nov. 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1127. URL https://aclanthology. org/D16-1127 .

C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out , pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013 .

D. Liu, Y. Yan, Y. Gong, W. Qi, H. Zhang, J. Jiao, W. Chen, J. Fu, L. Shou, M. Gong, P. Wang, J. Chen, D. Jiang, J. Lv, R. Zhang, W. Wu, M. Zhou, and N. Duan. GLGE: A new general language generation evaluation benchmark. In C. Zong, F. Xia, W. Li, and R. Navigli, editors, Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021 , volume ACL/IJCNLP 2021 of Findings of ACL , pages 408-420. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.findings-acl.36. URL https://doi.org/10.18653/v1/2021.findings-acl.36 .

Y. Liu, X. Zeng, F. Meng, and J. Zhou. Instruction position matters in sequence generation with large language models. arXiv preprint arXiv:2308.12097 , 2023.

Z. Ma, C. Shao, S. Gui, M. Zhang, and Y. Feng. Fuzzy alignments in directed acyclic graph for non-autoregressive machine translation. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=LSz-gQyd0zE .

T. Mihaylova and A. F. T. Martins. Scheduled sampling for transformers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop , pages 351-356, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-2049. URL https://aclanthology.org/P19-2049 .

I. J. Myung. Tutorial on maximum likelihood estimation. Journal of mathematical Psychology , 47(1):90-100, 2003.

S. Narayan, S. B. Cohen, and M. Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 17971807, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1206. URL https://aclanthology.org/D18-1206 .

M. Norouzi, S. Bengio, z. Chen, N. Jaitly, M. Schuster, Y. Wu, and D. Schuurmans. Reward augmented maximum likelihood for neural structured prediction. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/ paper_files/paper/2016/file/2f885d0fbe2e131bfc9d98363e55d1d4-Paper.pdf .

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Gray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id= TG8KACxEON .

R. Y. Pang and H. He. Text generation by learning from demonstrations. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum? id=RovX-uQ1Hua .

K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , pages 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/1073083.1073135. URL https://aclanthology.org/P02-1040 .

R. Paulus, C. Xiong, and R. Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations , 2018. URL https://openreview. net/forum?id=HkAClQgA-.

W. Qi, Y. Gong, J. Jiao, Y. Yan, W. Chen, D. Liu, K. Tang, H. Li, J. Chen, R. Zhang, M. Zhou, and N. Duan. BANG: bridging autoregressive and non-autoregressive generation with large scale pretraining. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 8630-8639. PMLR, 2021. URL http: //proceedings.mlr.press/v139/qi21a.html .

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. 2018.

M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. Sequence level training with recurrent neural networks. In Y. Bengio and Y. LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings , 2016. URL http://arxiv.org/abs/1511.06732 .

R. Rei, J. G. C. de Souza, D. Alves, C. Zerva, A. C. Farinha, T. Glushkova, A. Lavie, L. Coheur, and A. F. T. Martins. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT) , pages 578-585, Abu Dhabi, United Arab Emirates (Hybrid), Dec. 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.wmt-1.52 .

S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel. Self-critical sequence training for image captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1179-1195, 2017. doi: 10.1109/CVPR.2017.131.

A. M. Rush, S. Chopra, and J. Weston. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages 379-389, Lisbon, Portugal, Sept. 2015. Association for Computational Linguistics. doi: 10.18653/v1/D15-1044. URL https://aclanthology.org/D15-1044 .

C. Saharia, W. Chan, S. Saxena, and M. Norouzi. Non-autoregressive machine translation with latent alignments. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1098-1108, Online, Nov. 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.83. URL https://aclanthology.org/2020.emnlp-main.83 .

A. See, P. J. Liu, and C. D. Manning. Get to the point: Summarization with pointergenerator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1099. URL https://aclanthology.org/P17-1099 .

R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers . The Association for Computer Linguistics, 2016. doi: 10.18653/v1/p16-1162. URL https: //doi.org/10.18653/v1/p16-1162 .

C. Shao and Y. Feng. Non-monotonic latent alignments for ctc-based non-autoregressive machine translation. In NeurIPS , 2022. URL http://papers.nips.cc/paper_files/paper/ 2022/hash/35f805e65c77652efa731edc10c8e3a6-Abstract-Conference.html .

C. Shao, Y. Feng, J. Zhang, F. Meng, X. Chen, and J. Zhou. Retrieving sequential information for non-autoregressive neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 3013-3024, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1288. URL https: //www.aclweb.org/anthology/P19-1288 .

C. Shao, J. Zhang, Y. Feng, F. Meng, and J. Zhou. Minimizing the bag-of-ngrams difference for non-autoregressive neural machine translation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020 , pages 198205. AAAI Press, 2020. URL https://aaai.org/ojs/index.php/AAAI/article/view/ 5351 .

C. Shao, Y. Feng, J. Zhang, F. Meng, and J. Zhou. Sequence-Level Training for NonAutoregressive Neural Machine Translation. Computational Linguistics , pages 1-35, 10 2021. ISSN 0891-2017. doi: 10.1162/coli_a_00421. URL https://doi.org/10.1162/coli_a_ 00421 .

S. Shen, Y. Cheng, Z. He, W. He, H. Wu, M. Sun, and Y. Liu. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1683-1692, Berlin, Germany, Aug. 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1159. URL https://aclanthology.org/P16-1159 .

R. Shu, J. Lee, H. Nakayama, and K. Cho. Latent-variable non-autoregressive neural machine translation with deterministic inference using a delta posterior. 34:8846-8853, Apr. 2020. doi: 10.1609/aaai.v34i05.6413. URL https://ojs.aaai.org/index.php/AAAI/ article/view/6413 .

F. Stahlberg and B. Byrne. On NMT search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 3356-3362, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1331. URL https://aclanthology.org/D19-1331 .

F. Stahlberg and S. Kumar. Jam or cream first? modeling ambiguity in neural machine translation with SCONES. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 4950-4961, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/ 2022.naacl-main.365. URL https://aclanthology.org/2022.naacl-main.365 .

N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 3008-3021. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/ 1f89885d556929e98d3ef9b86448f951-Paper.pdf .

R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 12th International Conference on Neural Information Processing Systems , NIPS'99, pages 1057-1063, Cambridge, MA, USA, 1999. MIT Press.

C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2818-2826, 2016.

R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/ stanford_alpaca , 2023.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5998-6008, 2017. URL https://proceedings.neurips. cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html .

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, u. Kaiser, and I. Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, pages 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.

O. Vinyals and Q. Le. A neural conversational model. arXiv preprint arXiv:1506.05869 , 2015.

Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560 , 2022.

S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston. Neural text generation with unlikelihood training. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=SJeYe0NtvH .

R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. , 8(3-4):229-256, May 1992. ISSN 0885-6125. doi: 10.1007/BF00992696. URL https://doi.org/10.1007/BF00992696 .

L. Yu, W. Zhang, J. Wang, and Y. Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence , AAAI'17, pages 2852-2858. AAAI Press, 2017.

J. Zeng, F. Meng, Y. Yin, and J. Zhou. Tim: Teaching large language models to translate with comparison. arXiv preprint arXiv:2307.04408 , 2023.

S. Zhang, Q. Fang, Z. Zhang, Z. Ma, Y. Zhou, L. Huang, M. Bu, S. Gui, Y. Chen, X. Chen, and Y. Feng. Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models. arXiv preprint arXiv:2306.10968 , 2023.

S. Zhang, S. Wu, O. Irsoy, S. Lu, M. Bansal, M. Dredze, and D. Rosenberg. MixCE: Training autoregressive language models by mixing forward and reverse cross-entropies. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 9027-9050, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.502. URL https: //aclanthology.org/2023.acl-long.502 .

W. Zhang, Y. Feng, F. Meng, D. You, and Q. Liu. Bridging the gap between training and inference for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4334-4343, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1426. URL https:// aclanthology.org/P19-1426 .

A Proofs

A.1 Proof of Theorem 1

Theorem 1. Given an arbitrary differentiable and increasing function f , the optimal distribution p f satisfies p f ( x 1 ) ≥ p f ( x 2 ) ≥ · · · ≥ p f ( x i ) · · · .

Proof. We prove this theorem by contradiction. Suppose there exist an indice ( i, j ) such that i < j and p f ( x i ) < p f ( x j ) . In this case, we can construct a distribution p ′ f with a lower loss than p f , which contradicts the optimality of p f . Specifically, let p ′ f be identical to p f except for the changes p ′ f ( x i ) = p f ( x j ) and p ′ f ( x j ) = p f ( x i ) . We denote the loss of a model distribution p as L f ( p θ = p ) and show that p ′ f has a lower loss:

This inequality contradicts the assumption that p f minimizes L f , thereby proving the theorem.

A.2 Proof of Theorem 2

Theorem 2. If f is an increasing convex function on [0 , 1] , then the optimal distribution p f is a one-hot distribution that p f ( x 1 ) = 1 and p f ( x i ) = 0 , i > 1 .

Proof. We prove this theorem by contradiction. Suppose p f is not the one-hot distribution described above, then there must exist an index i > 1 such that p f ( x i ) > 0 . In this case, we can construct a distribution p ′ f with a lower loss than p f , which contradicts the optimality of p f . Specifically, let p ′ f be identical to p f except for the changes p ′ f ( x 1 ) = p f ( x 1 ) + α and p ′ f ( x i ) = p f ( x i ) -α , where 0 ≤ α ≤ p f ( x i ) . Then we can calculate the gradient of loss L f ( p θ = p ′ f ) with respect to α :

We can analyze the above equation via the following steps:

The above reasoning shows the loss can be further reduced, which contradicts the assumption that p f minimizes L f and proves the theorem.

A.3 Proof of Theorem 3

Theorem 3. Let f be an increasing convex function and g be an increasing concave function. Then, there exists a positive integer m such that the following inequalities hold:

Proof. We prove this theorem by contradiction. Assuming the theorem does not hold, there must exist an indice ( i, j ) with i < j , p fg ( x i ) < p g ( x i ) , and p fg ( x j ) > p g ( x j ) . In this case, we can construct a distribution p ′ fg with a lower loss than p fg , which contradicts the optimality of p fg .

First, we can establish the following inequality from the optimality of p g :

Assume the above inequality does not hold, then we can further reduce the loss, which contradicts the optimality of p g . Let p ′ g ( x i ) = p g ( x i ) -α , and p ′ g ( x j ) = p g ( x j ) + α . The gradient of loss L fg with respect to α = 0 is p data ( x i ) · g ′ ( p g ( x i )) -p data ( x j ) · g ′ ( p g ( x j )) < 0 , so we can further reduce the loss with a positive α , proving inequality 17 by contradiction.

Then, let p ′ fg be identical to p fg except for the changes p ′ fg ( x i ) = p fg ( x i ) + α and p ′ fg ( x j ) = p fg ( x j ) -α , where 0 ≤ α ≤ p fg ( x j ) . We can calculate the gradient of loss L fg ( p θ = p ′ fg ) with respect to α :

We can analyze the above equation via the following steps:

Combining steps 1-3, we obtain:

Further, the following steps shows that the gradient is less than 0:

The above reasoning shows the loss can be further reduced, which contradicts the assumption that p f minimizes L f and proves the theorem.

Corollary 1. The Shannon entropy of p fg is less than or equal to the Shannon entropy of p g .

Proof. The Shannon entropy of distribution p , denoted by H p , is defined as H p = -∑ x p ( x ) log p ( x ) . Consider a function h (∆ x ) = -( x 1 +∆ x ) log( x 1 +∆ x ) -( x 2 -∆ x ) log( x 2 -∆ x ) . It's first-order derivative h ′ (∆ x ) = log( x 2 -∆ x ) -log( x 1 +∆ x ) . Assuming x 1 ≥ x 2 , we observe that h ′ (∆ x ) < 0 when ∆ x > 0 , so the entropy decreases when we reduce x 2 and increase x 1 accordingly. The transformation from p g to p fg can be viewed as a series of such adjustments, which implies that the Shannon entropy of p fg is less than or equal to the Shannon entropy of p g .

Corollary 2. For any n ∈ { 1 , 2 , ... } , the sum of the probabilities of the n most probable samples increases: ∑ n i =1 p fg ( x i ) ≥ ∑ n i =1 p g ( x i ) .

Proof. The theorem guarantees the existence of a positive integer m such that p fg ( x i ) ≥ p g ( x i ) for all i < m and p fg ( x i ) ≤ p g ( x i ) for all i ≥ m . In the case where n < m , we have p fg ( x i ) ≥ p g ( x i ) for all i with 1 ≤ i ≤ n < m . This leads to the inequality ∑ n i =1 p fg ( x i ) ≥ ∑ n i =1 p g ( x i ) . In the case where n ≥ m , we have p fg ( x i ) ≤ p g ( x i ) for all i with m ≤ n < i . Therefore, we can write ∑ n i =1 ( p fg ( x i ) -p g ( x i )) = -∑ |X| i = n +1 ( p fg ( x i ) -p g ( x i )) ≥ 0 . Consequently, the inequality ∑ n i =1 p fg ( x i ) ≥ ∑ n i =1 p g ( x i ) also holds. Therefore, in both cases, the corollary is proved.

A.4 Proof of Theorem 4

Theorem 4. Let f be an increasing convex function and g be an increasing concave function. If g satisfies g ′′′ ( x ) · g ′ ( x ) ≥ g ′′ ( x ) 2 > 0 for all x ∈ (0 , 1) , then the difference between p fg and p g exhibits a monotonic order: p fg ( x 1 ) -p g ( x 1 ) ≥ p fg ( x 2 ) -p g ( x 2 ) ≥ ... ≥ p fg ( x m -1 ) -p g ( x m -1 ) ≥ 0 , where m is the positive integer described in Theorem 3.

.

Proof. We prove this theorem by contradiction. Assuming the theorem does not hold, there must exist an indice indice ( i, j ) with i < j and 0 ≤ p fg ( x i ) -p g ( x i ) < p fg ( x j ) -p g ( x j ) . In this case, we can construct a distribution p ′ fg with a lower loss than p fg , which contradicts the optimality of p fg . Specifically, let p ′ fg be identical to p fg except for the changes p ′ fg ( x i ) = p fg ( x i ) + α and p ′ fg ( x j ) = p fg ( x j ) -α , where 0 ≤ α ≤ p fg ( x j ) . Then we can calculate the gradient of loss L fg ( p θ = p ′ fg ) with respect to α :

Our goal is to demonstrate that ∂ L fg ( p θ = p ′ fg ) ∂α ∣ ∣ ∣ ∣ α =0 < 0 , which would contradict the assumption that p fg minimizes L fg , thereby proving the theorem.

Given the optimality of p g , we have p data ( x i ) · g ′ ( p g ( x i )) ≥ p data ( x j ) · g ′ ( p g ( x j )) , otherwise we can reduce p g ( x i ) to obtain a lower loss. From Theorem 1, we know that p fg ( x i ) ≥ p fg ( x j ) , and because f is convex and g is increasing, we have f ′ g ( p fg ( x i )) ≥ f ′ g ( p fg ( x j )) . Using these inequalities, we obtain a upperbound of equation 22:

To demonstrate that ∂ L fg ( pθ = p ′ fg ) ∂α ∣ ∣ ∣ ∣ α =0 < 0 , we only need to prove the following inequality: g ′ ( p fg ( x j )) g ′ ( p g ( x j )) -g ′ ( p fg ( x i )) g ′ ( p g ( x i )) < 0 . (24)

Let ∆ x = p fg ( x i ) -p g ( x i ) < p fg ( x j ) -p g ( x j ) . As a result, p fg ( x j ) > ∆ x + p g ( x j ) , and thus g ′ ( p fg ( x j )) < g ′ ( p g ( x j ) + ∆ x ) . This allows us to further simplify the inequality:

To establish that the right-hand side of the above inequality is non-positive, we can apply the logarithm transformation and show the following inequality instead:

Let's denote h ( x ) = log( g ′ ( x )) , x 1 = p g ( x i ) , and x 2 = p g ( x j ) . The above inequality can be simplified to:

where ∆ x ≥ 0 and x 2 ≤ x 1 according to Theorem 1. The above inequality holds when h ( x ) is a convex function. The second-order derivative of h ( x ) = log( g ′ ( x )) is:

Therefore, h ( x ) is a convex function under the condition g ′′′ ( x ) · g ′ ( x ) ≥ g ′′ ( x ) 2 . This verifies that ∂ L fg ( p θ = p ′ fg ) ∂α ∣ ∣ ∣ ∣ α =0 < 0 , completing the proof by contradiction.

B Experimental Settings

B.1 Machine Translation

B.1.1 Datasets and Metrics

Datasets We conduct experiments on widely used translation benchmark: WMT14 English-German (EN-DE, 4.5M), where the validation and test sets are newstest2013 and newstest2014 respectively. We apply BPE [48] with 32K merge operations to learn a joint vocabulary on the tokenized data. Considering the major topic of this research is how to learn from a real-world data distribution, we don't apply any tricks that may have an influence on the distribution, e.g., knowledge distillation.

Metrics The overall quality of translation is assessed using metrics BLEU [38] and COMET [43]. 5 In the case of non-autoregressive models, we additionally quantify the prediction confidence and translation fluency of the generated output. Prediction confidence is measured with negative log-likelihood (NLL) of model generation. A lower NLL value indicates a more focused model distribution and higher prediction confidence. To evaluate translation fluency, we utilize an external pre-trained autoregressive language model. The generated translation is fed to the language model using teacher forcing and the resulting perplexity (PPL) is calculated as a measure of fluency. 6 A lower external PPL score indicates a higher level of fluency.

B.1.2 Implementation Details

Architectures In order to validate the overall efficacy of convex-composition loss, we perform experiments using various model architectures. We adopt Transformerbase [63] as our autoregressive baseline and Vanilla-NAT [17], CMLM [14] and CTC [46] as our non-autoregressive baselines. We apply uniform copy to construct decoder inputs in Vanilla-NAT and CTC. The decoder length in CTC is set to 2× the source length.

Training Although training with convex-composition loss offers the desirable property of optimality, it can encounter gradient vanishing issues during initialization as analyzed previously. To mitigate this, we employ a two-step training approach: MLE pre-training followed by fine-tuning with convexcomposition loss. This approach allows us to avoid numerical gradient issues while still benefiting from the optimality achieved through convex composition. For training with convex-composition loss, we set the exponent hyperparameter k to 1 for the autoregressive model and tune it from {1,2,3,5,8} on the validation set for non-autoregressive models. Throughout both MLE and convex-composition training, all models are optimized using the Adam optimizer [24] with β = (0 . 9 , 0 . 98) and ϵ = 10 -8 . Detailed information regarding other training hyperparameters can be found in Table 7.

Decoding For the autoregressive model, we set the beam length to 5 by default and tune the length penalty on the validation set unless stated otherwise. For Vanilla-NAT and CTC, we utilize fully non-autoregressive argmax decoding. In the case of CMLM, we employ 5 length candidates and

disable iteration for inference. The decoding speedup is measured with a batch size of 1 on GeForce RTX 3090 GPUs.

B.2 Abstractive Summarization

B.2.1 Datasets and Metrics

We conduct experiments on two widely used summarization benchmarks: CNN/DailyMail [18] and Xsum [34]. CNN/DailyMail contains 220K articles from the Daily Mail newspaper and 93K articles from CNN. Each article contains a bullet point summary consisting of multiple sentences. We use the non-anonymized variant following [47, 29]. After the pre-processing, there are 311,971 〈 article, summary 〉 pairs. XSum consists of 227K online articles from the British Broadcasting Corporation (BBC), containing professionally written single-sentence summaries. After the preprocessing, there are 226,677 〈 article, summary 〉 data pairs. In order to maintain consistency with previous works [26, 40], we employ GPT-2 tokenizer to tokenize raw CNN/DailyMail data, and Berttokenizer to tokenize raw Xsum data. The summarization quality is measured with ROUGE-1, ROUGE-2 and ROUGE-L [28] as discussed in [6].

B.2.2 Implementation Details

In our summarization experiments, most of the implementation details of the Transformer align with those used in translation. However, there are a few modifications to ensure consistency with previous work [26]. We apply layer normalization to the embeddings. The attention dropout is set to 0.1, and the weight decay is set to 0.01. We utilize beam search with a size of 4 during decoding. The length penalty, max_len_b, and min_len are set to 2.0, 140, and 55, respectively on CNN/DailyMail dataset. We use a length penalty of 1.2 on Xsum dataset. For CNN/DailyMail dataset, we additionally employ a tri-gram repetition prevention trick.

B.3 Large Language Models

For the development of LLMs, we utilize LLaMA-7B and LLaMA-13B [61] as our foundation models. We conduct instruction tuning using the Alpaca dataset by GPT4 [65, 60], which comprises 52K instruction-following demonstrations. Instead of the standard cross-entropy loss employed during instruction tuning, we adopt the convex-composition loss of exponential form to fine-tune foundation models.

The generative capability of LLMs is also evaluated on the two representative closed-ended text generation tasks: machine translation and text summarization. For machine translation, we follow previous works [22, 70, 69, 30] to evaluate the translation capability on four WMT22 translation tasks (Chinese-to-English, English-to-Chinese, German-to-English, and English-to-German). For text summarization, we follow Liu et al. [30] to conduct the evaluation on CNN/DailyMail Dataset [18]. We employ beam search with a beam size of 4 for machine translation and 2 for summarization. The prompt for machine translation is "Translate the following sentences from [SRC] to [TGT]." The prompt for summarization is "Write a brief and focused summary of the passage that follows.".

C Effects of k on AR Models

We study the effects of exponent hyper-parameter k on autoregressive models. Table 8 presents the BLEUscores of autoregressive models as the exponent k varies, showing that the optimal performance is achieved when k = 1 . Other choices of k , such as k = 0 . 5 or 0 . 75 , also yield improvements, predominantly in the context of the greedy search setting.

D Correlations on Other NAR Models

In Section 4.2, we present compelling evidence in support of the mode collapse property of convex function effectively mitigating the multimodality issue in the NAR model. This evidence is derived from the strong correlation observed between model entropy and generation fluency in the CMLM model, as demonstrated in Table 4. In this section, we provide additional evidence for other NAR models to further support our findings in Table 9 and 10.

E Results on Alternative Choice of Convex Function

In addition to the exponential function, we have also explored another choice of convex function in our framework of convex-composition loss. In this section, we discuss the results of the choice of power function, i.e., L f ( θ ) = -E x ∼ p data ( x ) [ -( -log( p θ ( x )) T ) k ] , 0 ≤ k ≤ 1 . The results obtained from applying the power function in convex-composition loss are presented in Table 11.

We have observed that the benefits of applying the power function within the convex composition framework are significantly marginal compared to the exponential function, especially in the case of Vanilla-NAT. In addition, we have found the training process may encounter difficulties or failure when k is approaching 0. We attribute such problem to the shape of f ′ ( g ( p θ ( x ))) when power function is applied, i.e., k · ( -log( p θ ( x )) T ) k -1 .

As shown in Figure 3, the value of f ′ ( g ( p θ ( x ))) will approach a constant 1 as k approaches 1. This phenomenon arises due to the reduction in the convexity of function f , resulting in a decrease in gain. In case of k approaching 0, the situation is even worse where f ′ ( g ( p θ ( x ))) will experience a sudden increase from an extremely small value near 0. These factors result in an unstable training process and contribute to the power function being less suitable within the framework of convex-composition loss.

0.6

0.5

0.4

0.3

0.2

0.1

0.0

logp

T

0

2

4

6

8

10

12

14

f

(

logp

T

)

k

=0.1,

power

k

=0.7,

power

k

=0.99,

power

k

=3,

exponential

F Results on Diverse Generation

We study the effects of convex functions on VAE-based text generation models by replacing the log-probability-based reconstruction loss in ELBO with the convex-composition loss. Formally, we train the model using the following loss:

where we opt for the convex function f to be e kx , k ≥ 0 . We perform experiments within the context of conditional generation, utilizing a V AE-based non-autoregressive model [54, 16] for the task of machine translation. During inference, we randomly sample the latent variable 3 times to generate diverse texts. We assess the quality with BLEU score computed against reference (reference-BLEU) and measure the diversity with BLEU score computed against each other (pairwise-BLEU). The average value and standard derivation are reported in Table 12.

Table 12: Reference-BLEU and Pairwise-BLEU scores of VAE-based NAT models trained with different objectives on WMT14 EN-DE test set. The texts are generated by sampling the latent distribution 3 times.

During the training process, we have observed that KL divergence tends to vanish more readily when the convex functions are applied. We attribute this phenomenon to the smaller norms of gradients associated with the convex-composition loss. As a result, the gradient of the KL divergence dominates the model update, leading to the KL divergence vanishing. We note VAE-based text generation models trained using the convex-composition loss exhibit a higher generation quality while suffering from poor diversity, which is consistent with the mode collapse property of convex function.

G Analysis of Convex Learning and Knowledge Distillation

With the ability to capture a concentrated distribution from datasets exhibiting a multi-modal distribution, the proposed convex learning approach shows similar dynamics to knowledge distillation [19], a technique which encourages the student model to imitate the output of the teacher model. To compare the two methods, we utilize autoregressive Transformer as the teacher and apply sequence-level knowledge distillation [23] to construct a dataset of lower complexity, and train the models using different losses.

1

2

3

5

8

beam size

26.0

26.4

26.8

27.2

27.6

28.0

BLEU

MLE

MLE + KD

The results in Table 13 and Figure 4 demonstrate that convex learning and knowledge distillation have similar effects on text generation models. Both methods lead to significant improvements on non-autoregressive models and bridge the performance gap between greedy and beam search of autoregressive models. It is worth noting that training with the convex-composition loss avoids the intricate process of training an additional teacher model and decoding the whole training set to achieve the improvements. Moreover, convex-composition loss can be combined with knowledge distillation to further enhance the performance.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBased on the ideas of convex optimization, the paper proposes a new loss function for the training of text generation models. The experimental results show that it has certain advantages over MLE, whether in autoregressive models or non-autoregressive models.\n\nSTRENGTHS:\n1. The paper has done a relatively detailed theoretical derivation;\n\n2. The final obtained loss is simple and practical;\n\n3. Experimental results show its superiority over MLE in both autoregressive and non-autoregressive models;\n\n4. It has narrowed the gap between greedy search and beam search.\n\nWEAKNESSES:\n1. Some key experimental hyperparameters were not provided, such as the value of T, which I could not find in the main text or the appendix;\n\n2. The experimental part only compared with MLE, without comparing other loss functions (such as other classification losses used to approximate accuracy);\n\n3. The notation is a bit confusing (for example, k is used in both formula 11 and theorem 3, but the meanings are not the same);\n\n4. Since the goal of the Convex Loss is one-hot rather than learning distribution, it seems that it can only be used to improve the greedy/beam search, and is not applicable to the decoding strategy of random sampling.\n\nQUESTIONS:\nThe original goal of Convex Loss is to learn one-hot, which is more consistent with the greedy/beam search decoding strategy. However, this contradicts gradient-based optimizers, so the final loss balances the goal of one-hot and gradient, which is formula 12. My questions are:\n\n1. Compared to the gradient of MLE, the gradient of formula 12, which is formula 13, seems to be smoother. This appears to make the model's output smoother rather than closer to one-hot. Doesn't this contradict the starting point of Convex Loss?\n\n2. Is it possible to propose a quantitative metric to measure the impact of these two aspects? Based on this metric, we might be able to directly obtain an optimal solution, instead of introducing an adjustable hyperparameter like in formula 12. Because as long as an adjustable hyperparameter is introduced, the experimental results are likely to improve to a certain extent, which is a very commonplace feature and seems to waste the previous detailed theoretical analysis.\n\nLIMITATIONS:\nI couldn't find any discussion about limitations in the main text and the appendix of the paper. Perhaps the authors should include it in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors have proposed the novel way set of loss functions to learn neural sequence model in both AR and NAR factorization. The core idea is to sharpen the model distribution so that is comes closer to the one-hot target distributions which are typically used for closed-form generation task such as NMT. They have theoretically confirmed how the usage of composition of convex and concave functions with model's probability will result in the aforementioned sharper model's induced distribution. \nThey performed set of experiments in both AR and NAR setting. AR settings shown that the proposed approach narrows down the gap between greedy and beam search for NMT and slightly improve scores on summarization task. NAR results shows improvements in translation quality.\n\nSTRENGTHS:\n* The proposed apprach is very intuitive and makes sense from the optimization point of view and given the motivation of getting a sharper distribution. It is also easy to implement.\n\n* The proposed framework of convex/concave composition allows community to extend this by examining other potential candidates for the loss function.\n\n* Assuming that the NAR NMT baseline results are strong baselines, the proposed approach substantially improves the NAR prediction quality.\n\nWEAKNESSES:\n# Experimental analysis\n\n* The resulting BLEU scores for AR setting are too close and not convincing. I think it is required to perform multiple random seed training or other way of estimating the translation performance variance to verify the improvement. In its current form I would be more convinced that AR modeling performance is nearly the same in terms of translation quality.\n\n# Related work\n\nThere is a recent work (I am not an author!) around modeling sharper distributions by reformulating the sequential random variables to be one-hot: https://arxiv.org/abs/2205.00704. I believe it is very similar in terms of the underlying motivation and observed improvements. Given that this was not mentioned in this work, I am not sure how well other related work is covered in this paper.\n\nQUESTIONS:\n# tuning of k in AR vs NAR\n\n* you said you chose k = 1 for AR and tuned K for NAR experiments. And you only shown the ablation with k for NAR task. Why is so? I suspect there are some interesting pieces about k in AR setting. Could you share any insights? Also I believe it is crucial to tell that you tuned k for NAR experiments as it decreases the practicality of this method in real life. \n\n* from training details I see that you didnt use label smoothing for NAR modeling with convex objective. Why is so? I think that the answer \"it works better empirically\" is not good enough here as this method inherently related to making sharper distributions while smoothing is doing the opposite. So I tend to believe there is some interesting relationship there.\n\nLIMITATIONS:\nAuthors did not discuss limitations in the main text of their submission.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe objective of the authors in this paper is to develop a machine translation model that doesn't need to learn the complete output distribution given the input, but instead focuses on highly probable outputs based on p_data. To achieve this, the authors suggest replacing the logarithm in Maximum Likelihood Estimation (MLE) with a convex increasing function. They demonstrate that optimizing such a function leads to a desirable one-hot distribution. However, the authors acknowledge that training such a model can be challenging due to the gradient being proportional to the probability of the samples, which can be quite low for sequences.\n\nTo address this issue, the authors propose combining a concave function (e.g., log in MLE) with an increasing convex function. They illustrate that optimizing this composite function yields a distribution that assigns higher probability to outputs with greater likelihood under p_data.\n\nThe authors apply their loss function to train both autoregressive and non-autoregressive machine translation models. They demonstrate that the resulting greedy and beam outputs from these models surpass the performance of the greedy and beam outputs produced by MLE.\n\nUpdate: I have read the author's response. I am satisfied with the response and since I had already given an accept to the paper, I will keep my ranking.\n\nSTRENGTHS:\nThe authors' approach to addressing the challenge of learning a peaky distribution is innovative. The novel convex-composition loss functions introduced in this paper are particularly interesting within the realm of sequence generation.\n\nThe paper is highly compelling and offers an intuitive reading experience. The authors' strategy of incorporating an increasing convex function to emphasize high-probability outputs under p_data is remarkable.\n\nThe theoretical concepts discussed in this paper hold immense potential impact, even if the direct application to machine translation does not yield substantial results.\n\nWEAKNESSES:\nEquation (5) may not hold true for non-autoregressive models as these models employ latent variables that interconnect the output tokens, as mentioned in [1] [2].\n\nThe term \"Convex loss functions\" in section 3.2 is misleading since the negative logarithm used in MLE is, in fact, a convex loss function. The authors actually refer to \"Loss functions with convex f.\"\n\nThere is an inaccuracy in line 200 where the authors state that the standard loss function in maximum likelihood estimation is the log-probability. This is incorrect. The loss function is actually the negative log-probability, which is convex and not concave.\n\nEquation (12) can only utilize odd values for the power function, as for even values, the function decreases within the range of g(p_theta(x)).\n\n[1] Gu, J., et al. \"Non-autoregressive neural machine translation.\" International Conference on Learning Representations (ICLR). 2018.\n[2] Gu, Jiatao, and Xiang Kong. \"Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade.\" Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. 2021.\n\nQUESTIONS:\nIn line 149, the statement \"If Lf has multiple optimal distributions, we simply arrange them in alphabetical order and take the first one to be pf\" raises concerns. It is not feasible to arrange distributions in alphabetical order since the space of distributions is not countable, even if the sample space is countable.\n\nRegarding step number 3 in equation (18) of the Appendix, it is unclear how the optimality of pg implies its validity. Further clarification is needed to establish the reasoning behind this step and its connection to the optimality of pg.\n\nLIMITATIONS:\nThe authors have discussed that convex functions of probability are hard to train. It will be good to have a similar discussion about the composition loss functions discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new learning objective for language modeling, which is extended from Maximum likelihood estimation (MLE). More specifically, the author replaced the log function in log-likelihood with an arbitrary increasing function f. Then the author analyzed the properties of the loss under scenarios where f is convex and its relaxation. Besides, the author provides two practically feasible function forms of f for the learning objectives. Experiments with exponential function show empirical improvement of transformers on translation and summarization tasks.\n\nSTRENGTHS:\n1. The paper proposed a variant of MLE loss for text generation, which is novel.\n\n2. The assumptions for theoretical analysis are reasonable and commonly appeared in real-world application scenarios.\n\n3. The paper is clearly stated and easy to follow.\n\nWEAKNESSES:\n1. About Baselines: The method should have been evaluated on more baselines. For AR model, the author only compared the proposed method with the vanilla transformer. For NAR, only two baselines (CMLM, CTC) are compared. The method needs to be applied on more powerful baselines (e.g. [1]) to show its effectiveness and universality.\n\n2. About the f function: The author proposed two practical forms for the f function. However, in the experiments, only the exponential form is discussed. \n\n3. About the theoretical analysis: The analyses seem trivial, and provided little instruction on the practical design of the loss function, or to guarantee the effectiveness.\n\nReference:\n[1]. Bridging the Gap between Training and Inference for Neural Machine Translation, ACL 2019\n\nQUESTIONS:\n1. In equation 10， is x_i a sequence or a token? In my understanding, x_i represents a sequence. If x_i is a sequence, the difference between the proposed method and MLE is adding a multiplier to the gradient, as shown in equation 9?\n\n2. What are the empirical benefits of the theoretical analyses in the paper for further applications?\n\nLIMITATIONS:\nThe author didn't mention any societal impact of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses the potential limitations of likelihood/KL-guided training for natural language generators from a measure theoretical perspective. It analyses the discrepancy between maximizing the likelihood (thus recall-friendly) and producing high-quality samples (thus precision-oriented) in autoregressive and non-autoregressive models, and proposes an alternative to MLE as the concrete solution. Experiments on enhancing autoregressive/non-autoregressive language models on machine translation validate the effectiveness of the proposed objective.\n\nSTRENGTHS:\n1. The theoretical analysis is insightful, solid, and well-presented. It makes a lot of sense and the corresponding experiment results comply with the analysis, especially the ones on non-autoregressive models.\n\n2. The measure theoretical problem it studies has been a critical yet long-neglected one since the falling short of language GANs. This paper is a solid step towards the ultimate solution and I would really hope the publication of this paper can bring the community's focus back to this area.\n\nWEAKNESSES:\n1. It's less analyzed in this paper about the proposed objective on some potential (important and necessary) scenarios. For example, for text VAE/other types of latent-variable text generators, it's long been a dilemma to improve the generation quality while preventing the latent collapse problem.\n\nQUESTIONS:\n1. As stated in Weaknesses, is it possible to include additional experiments on diverse text generation with VAEs? You can try interpolation and I'm very curious if the proposed approach is eventually capable of mitigating the latent collapse of text VAEs.\n\n2. I would appreciate it if you can explicitly compare against models under setups with/without sample-based distillation from teacher models. As far as I'm concerned the dynamics of distillation (especially in machine translation) would be somehow similar to the proposed approach, I'd like to see if it is possible to achieve the performance of a model w/ distillation by simply replacing the objective with the proposed one. \n\nWould be more than happy to raise my scores if these two aspects are addressed in the revision period.\n\nLIMITATIONS:\nI don't observe any potential societal limitations of this work. This work is more of an algorithmic/theoretical contribution to the community.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}