SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This theorem provides a more granular description of the relative difference between  $p_{fg}$  and  $p_g$ . When  $p_{fg}$  decreases the probabilities assigned to less probable samples, it tends to reallocate this probability mass to the most probable samples. This enables text generation models to more accurately capture the most probable outputs. Note that the condition  $g^{\prime \prime \prime}(x)\cdot g^{\prime}(x)\geq g^{\prime \prime}(x)^{2} > 0$  is not overly restrictive. For instance, the loss function  $g = \log$  in MLE readily fulfills this condition:
$$
\log^ {\prime \prime \prime} (x) \cdot \log^ {\prime} (x) - \log^ {\prime \prime} (x) ^ {2} = \frac {2}{x ^ {3}} \cdot \frac {1}{x} - (- \frac {1}{x ^ {2}}) ^ {2} = \frac {1}{x ^ {4}} > 0. \tag {11}
$$
3.3.2 Practical Applications
The preceding theoretical analysis highlights the effectiveness of function composition. Here we turn to its practical applications and give some examples of convex-composition loss functions. The loss function in maximum likelihood estimation is typically the log-probability, and length normalization is often applied in practical usage, resulting in the loss  $g(p_{\theta}(x)) = \frac{\log(p_{\theta}(x))}{T}$ , where  $T$  denotes the sentence length. Common choices for increasing convex functions on  $(-\infty, 0]$  include the exponential function  $f(x) = e^{kx}, k \geq 0$  and the power function  $f(x) = -(-x)^{k}, 0 \leq k \leq 1$ . Through function composition, we can derive the following losses:
$$
f g \left(p _ {\theta} (x)\right) = \left\{ \begin{array}{l l} p _ {\theta} (x) ^ {\frac {k}{T}}, & f (x) = e ^ {k x} \\ - \left(- \frac {\log \left(p _ {\theta} (x)\right)}{T}\right) ^ {k}, & f (x) = - (- x) ^ {k}. \end{array} \right. \tag {12}
$$
The gradient of the convex-composition function is  $f'(g(p_{\theta}(x))) \cdot g'(p_{\theta}(x))$ . Compared to the gradient of the original loss  $g'(p_{\theta}(x))$ , it has an additional term  $f'(g(p_{\theta}(x)))$  that can be interpreted as a weight for the loss. Given that  $f$  is a convex function and  $g$  is an increasing function, the weight  $f'(g(p_{\theta}(x)))$  is larger for more probable samples, thereby directing the model's focus towards
Figure 1: Translation quality (BLEU) of autoregressive model as beam size varies on WMT14 EN  $\leftrightarrow$  DE test set.
Figure 2: Translation quality (BLEU) and prediction confidence (Output NLL) of different NAT models as the exponent  $k$  varies on WMT14 EN-DE test set.
generating outputs with high probabilities. Specifically, the loss weights  $f'(g(p_{\theta}(x)))$  associated with Equation 12 are:
$$
f ^ {\prime} \left(g \left(p _ {\theta} (x)\right)\right) = \left\{ \begin{array}{l l} k \cdot p _ {\theta} (x) ^ {\frac {k}{T}}, & f (x) = e ^ {k x} \\ k \cdot \left(- \frac {\log \left(p _ {\theta} (x)\right)}{T}\right) ^ {k - 1}, & f (x) = - (- x) ^ {k}, \end{array} \right. \tag {13}
$$
where the exponential function weights the sample by the prediction probability, and the power function weights the sample by the log-probability.
In practical applications, label smoothing [59, 63] is a widely used regularization technique for text generation models. The smoothing loss and log-probability loss are typically combined using a fixed hyperparameter  $\epsilon_{ls}$ . To preserve the ratio of smoothing loss to log-probability loss, we also apply the weight  $f'(g(p_{\theta}(x)))$  to the smoothing loss before interpolating it with the convex-composition loss.
To validate the practical advantages of loss functions with sharper optimal distributions, we conduct experiments on basic autoregressive (AR) models, non-autoregressive (NAR) models, and large language models (LLMs). We evaluate their performance on two representative closed-ended text generation tasks, including neural machine translation and text summarization. Following the theoretical analysis in previous sections, we combine the exponential function with standard log-probability, i.e.  $\mathcal{L}_f(\theta) = -\mathbb{E}_{x\sim p_{data}(x)}[p_\theta (x)^{\frac{k}{T}}]$ , as our training objective in the following experiments. We have also attempted to combine the power function with log-probability as training objective. We found that the power form encountered some difficulties during training, leading to worse performance compared to the exponential form. Due to the space limit, we leave the results under this setting in Appendix E.
Our theoretical analysis suggests that the model trained by convex-composition loss tends to predict a sharper distribution, in which the probability mass is more heavily allocated to the most probable samples. Such property leads the model becoming more confident about its prediction and facilitates the de-facto maximum a posteriori (MAP) decoding framework in closed-ended text generation tasks. In the following, we will discuss and validate the effects of convexity in the context of AR models, NAR models, and LLMs respectively. More details of settings can be found in Appendix B.
Table 1: BLEU scores of autoregressive models on WMT14 EN  $\leftrightarrow$  DE test set with different decoding strategies.
Model	EN-DE	DE-EN
greedy	beam5	Δ	greedy	beam5	Δ
Transformer [62]	26.48	27.57	1.09	29.78	31.21	1.43
Transformer + Convex	26.92	27.78	0.86	30.32	31.33	1.01
Table 2: ROUGE scores on CNN/DailyMail and XSum test sets. RG-1, RG-2, RG-L stand for ROUGE-1, ROUGE-2 and ROUGE-L scores.
Model	CNN/DM	XSUM
RG-1	RG-2	RG-L	RG-1	RG-2	RG-L
Transformer [62]	39.03	15.98	35.88	31.04	10.68	24.77
Transformer + Convex	39.56	16.84	36.26	31.55	11.13	25.09
4.1 Effects of Convexity on Autoregressive Models
In the context of autoregressive models, a model distribution trained with a convex-composition loss tends to exhibit fewer modes and a sharper distribution, thereby facilitating the task of approximate search algorithms in identifying the most likely output. We validate this conjecture by investigating the performance of greedy and beam search when trained with standard MLE and convex-composition loss in translation and summarization tasks. For translation task, we vary the beam size from  $\{1,2,3,5,8\}$ , where beam size 1 can be considered as greedy search.
Figure 1 visualizes the results in terms of BLEU [38], with precise numerical values given in Table 1. We observe a consistent improvement in translation quality when using convex-composition losses compared to MLE, and a similar trend is observed in summarization tasks as detailed in Table 2. These results provide experimental support that the composition with convex function promotes those approximate searching algorithms to perform argmax decoding. Meanwhile, Table 1 exhibits a diminishing gap between greedy search and beam search when equipped with convex-composition loss. This outcome can be attributed to the efficacy of the convex function in reducing the complexity of the model distribution, as described in Theorem 3. Such property amplifies the potential of lightweight approximate decoding algorithms within the autoregressive structure, a desirable trait in the context of modern, computation-intensive autoregressive neural networks.
4.2 Effects of Convexity on Non-autoregressive Models
Non-autoregressive models face the challenge of multi-modality, where fitting a data distribution with multiple target modes exceeds the capabilities of NAR models. Therefore, the mode collapse property of convex-composition loss would be beneficial to NAR models. Likelihood training will force the model to ignore sequential dependency, resulting in disfluency in its output (e.g., token repetition and omission). In contrast, convex-composition loss would encourage model to allocate most of its probability mass to the best among all proper candidates. Such property is able to help NAR model avoid generating a mixture of modes, thereby alleviating disfluency issues.
To demonstrate its effectiveness, we investigate the performance of convex-composition loss on three representative NAR models, including Vanilla-NAT [17], CMLM [14] and CTC [46]. Considering most of the NAR researches are restricted in the field of translation, we only conduct experiments on translation dataset. In addition to translation quality, we also assess the prediction confidence and generation fluency of NAR outputs. The prediction confidence is measured with negative log-likelihood of its generation and the fluency is measured by an external pre-trained language model  $^4$ . We use the PPL value reported by the language model to quantify the fluency of generation. The exponent hyperparameter  $k$  is manipulated to adjust the convexity of our composite loss function.
<sup>4</sup>https://github.com/facebookresearch/fairseq/tree/main/examples/language_model
Table 3: BLEU and COMET scores on WMT14 EN  $\leftrightarrow$  DE test set.
Model	Speedup	EN-DE	DE-EN
BLEU	COMET	BLEU	COMET
Transformer [62]	1.0×	27.57	82.76	31.21	82.98
Vanilla-NAT [17]	15.6×	10.41	40.69	16.01	56.03
Vanilla-NAT + Convex	15.6×	16.74	57.25	22.63	68.83
CMLM [14]	15.0×	11.22	43.62	15.26	56.63
CMLM + Convex	15.0×	20.45	65.99	19.11	63.54
CTC [46]	14.7×	16.98	54.77	20.53	66.26
CTC + Convex	14.7×	23.34	67.38	26.68	74.75
Table 4: Prediction confidence (Output NLL) and generation fluency (External PPL) of CMLM on WMT14 EN-DE test set.
k-th Power	1	2	3	5	8
Confidence (Output NLL) ↓	20.57	13.72	10.09	6.85	4.88
Fluency (External PPL) ↓	939.34	481.08	315.54	213.84	218.68
The results are shown in Figure 2 and Table 3. We observe a consistent improvement in translation quality across all NAT models with a maximum improvement of  $9+$  BLEU points on CMLM. Meanwhile, Figure 2 implies that prediction confidence significantly gains and the gain increases as  $k$  gets larger. Such phenomenon reveals a descending trend of model entropy as applying convex function on loss, which is consistent with Corollary 1. More importantly, we note a strong correlation between model entropy and generation fluency in Table 4, providing clear evidence that the mode collapse property of convex function indeed relieves NAR model from multi-modality problem.
4.3 Effects of Convexity on Large Language Models
Large language models have demonstrated remarkable capabilities in various applications, including both open-ended and closed-ended text generation tasks. For open-ended tasks, stochastic decoding methods such as temperature sampling are commonly employed to produce responses. In contrast, deterministic decoding methods like beam search are favored for closed-ended tasks like machine translation [22, 69, 30]. Given that the convex-composition loss enhances the model's ability to identify highly probable sentences, incorporating this loss function into the LLMs' training process would be beneficial to closed-ended generation tasks.
To demonstrate its effectiveness, we assess the performance of LLMs in machine translation (Table 5) and summarization (Table 6). Table 5 reveals that the LLaMA-7B model, incorporating convex-composition loss, surpasses the baseline model across all language pairs, achieving an average improvement of 1.84 BLEU. Likewise, the LLaMA-13B model with convex-composition loss outperforms the baseline model in three out of four language pairs. Table 6 further demonstrates the effectiveness of our method in text summarization. Due to memory limitations, we are only able to decode the text summarization dataset using the LLaMA-7B model.
Alternative Loss Functions Maximum likelihood estimation has become the dominant approach for learning text generation models, but it also comes with certain limitations. Various alternative loss functions have been proposed to improve the training process from different perspectives. Regarding the exposure bias problem [42] that autoregressive models are exposed to different distributions during training and inference, Bengio et al. [4], Mihaylova and Martins [32], Zhang et al. [72] proposed to reduce this gap by sampling from the model's own predictions during training. Another issue with text generation models is text degeneration: output text may be bland, incoherent, or gets stuck in repetitive loops [20]. To avoid text degeneration, Dieng et al. [9] proposed a learning criterion termed
Table 5: BLEU scores of Alpaca fine-tuned large language models on WMT22 test sets.
Model	EN-DE	DE-EN	EN-ZH	ZH-EN	AVG
LLaMA-7B	25.42	17.93	13.86	13.17	17.59
LLaMA-7B + Convex	27.57	19.88	15.00	15.28	19.43
LLaMA-13B	29.35	21.74	15.58	16.27	20.74
LLaMA-13B + Convex	28.75	22.20	16.25	20.08	21.82
Table 6: ROUGE scores of Alpaca fine-tuned large language models on CNN/DailyMail.
Model	RG-1	RG-2	RG-L	AVG
LLaMA-7B	28.66	12.49	26.37	22.51
LLaMA-7B + Convex	32.76	14.67	30.00	25.81
reflective likelihood to penalize incoherent outputs, and Welleck et al. [66] proposed unlikelihood training that forces unlikely generations to be assigned lower probability by the model. Additionally, to address the discrepancy between likelihood training and evaluation metrics, loss functions that more directly optimize evaluation metrics are proposed. Ranzato et al. [42] utilized the reinforcement learning technique to train recurrent neural networks with sequence level objectives. Shen et al. [53] proposed to optimize evaluation metrics with minimum risk training. Norouzi et al. [35], Edunov et al. [12] incorporated evaluation metrics into the maximum likelihood training objective. There are also efforts on learning a more focused distribution for text generation models [37, 71, 56]. However, these approaches primarily reformulate the loss function at the word level, which is insufficient for guiding the model towards identifying high-probability sentences at the sentence level. In contrast, our method explicitly trains the model to concentrate on generating highly probable sentences.
Reinforcement Learning Our work aligns closely with reinforcement learning (RL) based training techniques for text generation [67, 58]. While RL techniques typically maximize the expected reward by concentrating the probability mass on the sequence with the highest reward, our approach strives to put all the probability mass on the most likely sequence. RL allows for text generation models to optimize discrete evaluation metrics, which have wide usage in text generation tasks, including machine translation [42, 3], text summarization [39], image captioning [44], dialogue generation [27], etc. Furthermore, RL can be integrated with Generative Adversarial Networks [68] and can leverage human feedback for training [57, 36].
Loss Functions for NAR Models The limitation of maximum likelihood estimation is amplified in non-autoregressive (NAR) models since they inherently lack the capability to fit the data distribution [21]. To address this issue, researchers have developed loss functions specifically designed for NAR models, guiding them towards generating coherent text. Shao et al. [50, 52], Ding et al. [10] proposed to train NAR models with sequence-level objective functions. Ghazvininejad et al. [15], Du et al. [11] relaxed the alignment restriction in the cross-entropy loss. Shao et al. [51], Shao and Feng [49], Ma et al. [31] proposed n-gram based differentiable training objectives to optimize n-gram prediction accuracy. However, these methods lack theoretical guarantees for the shape of optimal distribution.
This paper investigates the theoretical properties and practical applications of a novel class of training objectives based on convex functions. Our findings show that convex functions can sharpen the optimal distribution, enabling text generation models to focus on highly probable outputs without having to estimate the entire data distribution. Experiments on various text generation tasks and models verify our theoretical analysis and demonstrate the practical effectiveness of our approach.
We thank the anonymous reviewers for their insightful comments.
[62] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
[63] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, u. Kaiser, and I. Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pages 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
[64] O. Vinyals and Q. Le. A neural conversational model. arXiv preprint arXiv:1506.05869, 2015.
[65] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022.
[66] S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston. Neural text generation with unlikelihood training. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SJeYeONtvH.
[67] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8(3-4):229-256, May 1992. ISSN 0885-6125. doi: 10.1007/BF00992696. URL https://doi.org/10.1007/BF00992696.
[68] L. Yu, W. Zhang, J. Wang, and Y. Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, pages 2852-2858. AAAI Press, 2017.
[69] J. Zeng, F. Meng, Y. Yin, and J. Zhou. Tim: Teaching large language models to translate with comparison. arXiv preprint arXiv:2307.04408, 2023.
[70] S. Zhang, Q. Fang, Z. Zhang, Z. Ma, Y. Zhou, L. Huang, M. Bu, S. Gui, Y. Chen, X. Chen, and Y. Feng. Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models. arXiv preprint arXiv:2306.10968, 2023.
[71] S. Zhang, S. Wu, O. Irsoy, S. Lu, M. Bansal, M. Dredze, and D. Rosenberg. MixCE: Training autoregressive language models by mixing forward and reverse cross-entropies. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9027–9050, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.502. URL https://aclanthology.org/2023.acl-long.502.
[72] W. Zhang, Y. Feng, F. Meng, D. You, and Q. Liu. Bridging the gap between training and inference for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4334-4343, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1426. URL https://aclanthology.org/P19-1426.
Theorem 1. Given an arbitrary differentiable and increasing function  $f$ , the optimal distribution  $p_f$  satisfies  $p_f(x_1) \geq p_f(x_2) \geq \dots \geq p_f(x_i) \dots$ .
Proof. We prove this theorem by contradiction. Suppose there exist an index  $(i,j)$  such that  $i < j$  and  $p_f(x_i) < p_f(x_j)$ . In this case, we can construct a distribution  $p_f'$  with a lower loss than  $p_f$ , which contradicts the optimality of  $p_f$ . Specifically, let  $p_f'$  be identical to  $p_f$  except for the changes  $p_f'(x_i) = p_f(x_j)$  and  $p_f'(x_j) = p_f(x_i)$ . We denote the loss of a model distribution  $p$  as  $\mathcal{L}_f(p_\theta = p)$  and show that  $p_f'$  has a lower loss:
$$
\mathcal {L} _ {f} \left(p _ {\theta} = p _ {f} ^ {\prime}\right) = \mathcal {L} _ {f} \left(p _ {\theta} = p _ {f}\right) + \left(p _ {d a t a} \left(x _ {j}\right) - p _ {d a t a} \left(x _ {i}\right)\right) \cdot \left(f \left(p _ {f} \left(x _ {j}\right)\right) - f \left(p _ {f} \left(x _ {i}\right)\right)\right) <   \mathcal {L} _ {f} \left(p _ {\theta} = p _ {f}\right). \tag {14}
$$
This inequality contradicts the assumption that  $p_f$  minimizes  $\mathcal{L}_f$ , thereby proving the theorem.
A.2 Proof of Theorem 2
Proof. We prove this theorem by contradiction. Suppose  $p_f$  is not the one-hot distribution described above, then there must exist an index  $i > 1$  such that  $p_f(x_i) > 0$ . In this case, we can construct a distribution  $p_f'$  with a lower loss than  $p_f$ , which contradicts the optimality of  $p_f$ . Specifically, let  $p_f'$  be identical to  $p_f$  except for the changes  $p_f'(x_1) = p_f(x_1) + \alpha$  and  $p_f'(x_i) = p_f(x_i) - \alpha$ , where  $0 \leq \alpha \leq p_f(x_i)$ . Then we can calculate the gradient of loss  $\mathcal{L}_f(p_\theta = p_f')$  with respect to  $\alpha$ :
$$
\left. \frac {\partial \mathcal {L} _ {f} \left(p _ {\theta} = p _ {f} ^ {\prime}\right)}{\partial \alpha} \right| _ {\alpha = 0} = p _ {d a t a} \left(x _ {i}\right) \cdot f ^ {\prime} \left(p _ {f} \left(x _ {i}\right)\right) - p _ {d a t a} \left(x _ {1}\right) \cdot f ^ {\prime} \left(p _ {f} \left(x _ {1}\right)\right). \tag {15}
$$
We can analyze the above equation via the following steps:
$$
\left\{ \begin{array}{l} 1. p _ {f} \left(x _ {1}\right) \geq p _ {f} \left(x _ {i}\right), \text {f r o m t h e o r e m 1} \\ 2. f ^ {\prime} \left(p _ {f} \left(x _ {1}\right)\right) \geq f ^ {\prime} \left(p _ {f} \left(x _ {i}\right)\right) > 0, \text {f r o m s t e p 1 a n d t h e c o n v e x i t y o f} f \\ 3. p _ {d a t a} \left(x _ {1}\right) > p _ {d a t a} \left(x _ {i}\right) > 0 \\ 4. \frac {\partial \mathcal {L} _ {f} \left(p _ {\theta} = p _ {f} ^ {\prime}\right)}{\partial \alpha} \Bigg | _ {\alpha = 0} <   0, \text {f r o m s t e p s 2 , 3} \end{array} . \right. \tag {16}
$$
The above reasoning shows the loss can be further reduced, which contradicts the assumption that  $p_f$  minimizes  $\mathcal{L}_f$  and proves the theorem.
A.3 Proof of Theorem 3
1.  $p_{fg}(x_i) \geq p_g(x_i)$  for all  $i < m$ ,
2.  $p_{fg}(x_i)\leq p_g(x_i)$  forall  $i\geq m$
Proof. We prove this theorem by contradiction. Assuming the theorem does not hold, there must exist an index  $(i,j)$  with  $i < j$ ,  $p_{fg}(x_i) < p_g(x_i)$ , and  $p_{fg}(x_j) > p_g(x_j)$ . In this case, we can construct a distribution  $p_{fg}'$  with a lower loss than  $p_{fg}$ , which contradicts the optimality of  $p_{fg}$ .
First, we can establish the following inequality from the optimality of  $p_g$ :
$$
p _ {d a t a} \left(x _ {i}\right) \cdot g ^ {\prime} \left(p _ {g} \left(x _ {i}\right)\right) \geq p _ {d a t a} \left(x _ {j}\right) \cdot g ^ {\prime} \left(p _ {g} \left(x _ {j}\right)\right). \tag {17}
$$
Assume the above inequality does not hold, then we can further reduce the loss, which contradicts the optimality of  $p_g$ . Let  $p_g'(x_i) = p_g(x_i) - \alpha$ , and  $p_g'(x_j) = p_g(x_j) + \alpha$ . The gradient of loss  $\mathcal{L}_{fg}$  with respect to  $\alpha = 0$  is  $p_{data}(x_i) \cdot g'(p_g(x_i)) - p_{data}(x_j) \cdot g'(p_g(x_j)) < 0$ , so we can further reduce the loss with a positive  $\alpha$ , proving inequality 17 by contradiction.
Then, let  $p_{fg}'$  be identical to  $p_{fg}$  except for the changes  $p_{fg}'(x_i) = p_{fg}(x_i) + \alpha$  and  $p_{fg}'(x_j) = p_{fg}(x_j) - \alpha$ , where  $0 \leq \alpha \leq p_{fg}(x_j)$ . We can calculate the gradient of loss  $\mathcal{L}_{fg}(p_\theta = p_{fg}')$  with respect to  $\alpha$ :
$$
\left. \frac {\partial \mathcal {L} _ {f g} \left(p _ {\theta} = p _ {f g} ^ {\prime}\right)}{\partial \alpha} \right| _ {\alpha = 0} = p _ {d a t a} \left(x _ {j}\right) \cdot f ^ {\prime} g \left(p _ {f g} \left(x _ {j}\right)\right) \cdot g ^ {\prime} \left(p _ {f g} \left(x _ {j}\right)\right) - p _ {d a t a} \left(x _ {i}\right) \cdot f ^ {\prime} g \left(p _ {f g} \left(x _ {i}\right)\right) \cdot g ^ {\prime} \left(p _ {f g} \left(x _ {i}\right)\right). \tag {18}
$$
$\left\{ \begin{array}{ll}1,g^{\prime}(p_{fg}(x_i))\geq g^{\prime}(p_g(x_i)),\mathrm{from}p_{fg}(x_i) <   p_g(x_i)\mathrm{and the concavity of}g\\ 2,g^{\prime}(p_{fg}(x_j))\leq g^{\prime}(p_g(x_j)),\mathrm{from}p_{fg}(x_j) > p_g(x_j)\mathrm{and the concavity of}g \end{array} \right.$  (19)  $3.p_{data}(x_i)\cdot g'(p_g(x_i))\geq p_{data}(x_j)\cdot g'(p_g(x_j))$  , from inequality 17