The summary generated by the proposed method is more comprehensive than the reference, while the reference only includes the facts in the last sentence of the source content. In short, the generated summary of the proposed method is more consistent with the source content. It also exhibits the necessity of the proposed human evaluation. Because when the generated summary is evaluated against the reference, it may seem redundant or wrong, but it is actually true to the source content. While it is arguable that the generated summary is better than the reference, there is no doubt that the generated summary of the proposed method is better than the baseline. However, the improvement cannot be properly shown by the existing evaluation methods. Furthermore, the examples suggest that the proposed method does learn better correspondence. The highlighted words in each example in Table TABREF28 share almost the same previous words. However, in the first one, the baseline considers “UTF8gbsn停” (stop) as the most related words, which is a sign of noisy word relations learned from other training examples, while the proposed method generates “UTF8gbsn进站” (to the platform), which is more related to what a human thinks.

Regularizing Output Distribution of Abstractive Chinese Social Media Text Summarization for Improved Semantic Consistency	Abstractive text summarization is a highly difficult problem, and the sequence-to-sequence model has shown success in improving the performance on the task. However, the generated summaries are often inconsistent with the source content in semantics. In such cases, when generating summaries, the model selects semantically unrelated words with respect to the source content as the most probable output. The problem can be attributed to heuristically constructed training data, where summaries can be unrelated to the source content, thus containing semantically unrelated words and spurious word correspondence. In this paper, we propose a regularization approach for the sequence-to-sequence model and make use of what the model has learned to regularize the learning objective to alleviate the effect of the problem. In addition, we propose a practical human evaluation method to address the problem that the existing automatic evaluation method does not evaluate the semantic consistency with the source content properly. Experimental results demonstrate the effectiveness of the proposed approach, which outperforms almost all the existing models. Especially, the proposed approach improves the semantic consistency by 4\% in terms of human evaluation.

Conclusions	We propose a regularization approach for the sequence-to-sequence model on the Chinese social media summarization task. In the proposed approach, we use a cross-entropy based regularization term to make the model neglect the possible unrelated words. We propose two methods for obtaining the soft output word distribution used in the regularization, of which Dual-Train proves to be more effective. Experimental results show that the proposed method can improve the semantic consistency by 4% in terms of human evaluation. As shown by the analysis, the proposed method achieves the improvements by eliminating the less semantically-related word correspondence. The proposed human evaluation method is effective and efficient in judging the semantic consistency, which is absent in previous work but is crucial in the accurate evaluation of the text summarization systems. The proposed metric is simple to conduct and easy to interpret. It also provides an insight on how practicable the existing systems are in the real-world scenario.

In the extreme case, the generated text could be completely intelligible, but the characters could still match. In theory, calculating ROUGE metrics on the word level could alleviate the problem. However, word segmentation is also a non-trivial task for Chinese. There are many kinds of segmentation rules, which will produce different ROUGE scores. We argue that it is not acceptable to introduce additional systematic bias in automatic evaluations, and automatic evaluation for semantically related tasks can only serve as a reference. To avoid the deficiencies, we propose a simple human evaluation method to assess the semantic consistency. Each summary candidate is evaluated against the text rather than the reference. If the candidate is irrelevant or incorrect to the text, or the candidate is not understandable, the candidate is labeled bad. Otherwise, the candidate is labeled good. Then, we can get an accuracy of the good summaries. The proposed evaluation is very simple and straight-forward. It focuses on the relevance between the summary and the text. The semantic consistency should be the major consideration when putting the text summarization methods into practice, but the current automatic methods cannot judge properly. For detailed guidelines in human evaluation, please refer to Appendix SECREF6 .

Besides, as ROUGE is based on the reference, it requires high-quality reference summary for a reasonable evaluation, which is also lacking in the existing dataset for Chinese social media text summarization. We argue that for proper evaluation of text generation task, human evaluation cannot be avoided. We propose a simple and practical human evaluation for evaluating text summarization, where the summary is evaluated against the source content instead of the reference. It handles both of the problems of paraphrasing and lack of high-quality reference. The contributions of this work are summarized as follows:

After investigating its generated summaries, we find that the major reason is that the generated summaries are not grammatically complete and often stop too early, although the generated part is indeed more related to the source content. Because the definition of being good, the improved relevance does not make up the loss on intelligibility. Then, we compare the automatic evaluation results in Table TABREF25 . As we can see, only applying soft training without adaptation (Self-Train) hurts the performance. With the additional output layer (Dual-Train), the performance can be greatly improved over the baseline. Moreover, with the proposed method the simple baseline model is second to the best compared with the state-of-the-art models and even surpasses in ROUGE-2. It is promising that applying the proposed method to the state-of-the-art model could also improve its performance. The automatic evaluation is done on the original test set to facilitate comparison with existing work. However, a more reasonable setting would be to exclude the 52 test instances that are found bad in the human evaluation, because the quality of the automatic evaluation depends on the reference summary.

In the human evaluation, the text-summary pairs are dispatched to two human annotators who are native speakers of Chinese. As in our setting the summary is evaluated against the reference, the number of the pairs needs to be manually evaluated is four times the number of the pairs in the test set, because we need to compare four systems in total. To decrease the workload and get a hint about the annotation quality at the same time, we adopt the following procedure. We first randomly select 100 pairs in the validation set for the two human annotators to evaluate. Each pair is annotated twice, and the inter-annotator agreement is checked. We find that under the protocol, the inter-annotator agreement is quite high. In the evaluation of the test set, a pair is only annotated once to accelerate evaluation. To further maintain consistency, summaries of the same source content will not be distributed to different annotators.

Dual Output Layers	However, the aforementioned method tries to regularize the output word distribution based on what it has already learned. The relative order of the output words is kept. The self-dependency may not be desirable for regularization. It may be better if more correspondence that is spurious can be identified. In this paper, we further propose to obtain the soft target from a different view of the model, so that different knowledge of the dataset can be used to mitigate the overfitting problem. An additional output layer is introduced to generate the soft target. The two output layers share the same hidden representation but have independent parameters. They could learn different knowledge of the data. We refer to this approach as Dual-Train. For clarity, the original output layer is denoted by INLINEFORM0 and the new output layer INLINEFORM1 . Their outputs are denoted by INLINEFORM2 and INLINEFORM3 , respectively. The output layer INLINEFORM4 acts as the original output layer. We apply soft training using the output from INLINEFORM5 to this output layer to increase its ability of generalization. Suppose the correct label is INLINEFORM6 .

Based on the sequence-to-sequence model, we propose a regularization method to heuristically show down the learning of the spurious correspondence, so that the unrelated information in the dataset is less represented by the model. We incorporate a new soft training target to achieve this goal. For each output time in training, in addition to the gold reference word, the current output also targets at a softened output word distribution that regularizes the current output word distribution. In this way, a more robust correspondence of the source content and the output words can be learned, and potentially, the output summary will be more semantically consistent. To obtain the softened output word distribution, we propose two methods based on the sequence-to-sequence model. More detailed explanation is introduced in Section SECREF2 . Another problem for abstractive text summarization is that the system summary cannot be easily evaluated automatically. ROUGE  is widely used for summarization evaluation. However, as ROUGE is designed for extractive text summarization, it cannot deal with summary paraphrasing in abstractive text summarization.

Standard for Human Evaluation	For human evaluation, the annotators are asked to evaluate the summary against the source content based on the goodness of the summary. If the summary is not understandable, relevant or correct according to the source content, the summary is considered bad. More concretely, the annotators are asked to examine the following aspects to determine whether the summary is good: If a rule is not met, the summary is labeled bad, and the following rules do not need to be checked. In Table TABREF33 , we give examples for cases of each rule. In the first one, the summary is not fluent, because the patient of the predicate UTF8gbsn“找” (seek for) is missing. The second summary is fluent, but the content is not related to the source, in that we cannot determine if Lei Jun is actually fighting the scalpers based on the source content. In the third one, the summary is fluent and related to the source content, but the facts are wrong, as the summary is made up by facts of different people. The last one met all the three rules, and thus it is considered good.

Experimental Results	First, we show the results for human evaluation, which focuses on the semantic consistency of the summary with its source content. We evaluate the systems implemented by us as well as the reference. We cannot conduct human evaluations for the existing systems from other work, because the output summaries needed are not available for us. Besides, the baseline system we implemented is very competitive in terms of ROUGE and achieves better performance than almost all the existing systems. The results are listed in Table TABREF24 . It is surprising to see that the accuracy of the reference summaries does not reach 100%. It means that the test set still contains text-summary pairs of poor quality even after removing the pairs with relevance scores lower than 3 as suggested by the authors of the dataset. As we can see, Dual-Train improves the accuracy by 4%. Due to the rigorous definition of being good, the results mean that 4% more of the summaries are semantically consistent with their source content. However, Self-Train has a performance drop compared to the baseline.

The proposed method is different compared with the existing model distillation methods, in that the proposed method does not require additional models or additional space to record the past soft label distributions. The existing methods are not suitable for text summarization tasks, because the training of an additional model is costly, and the additional space is huge due to the massive number of data. The proposed method uses its current state as the soft target distribution and eliminates the need to train additional models or to store the history information.

Evaluation Protocol	For text summarization, a common automatic evaluation method is ROUGE  . The generated summary is evaluated against the reference summary, based on unigram recall (ROUGE-1), bigram recall (ROUGE-2), and recall of longest common subsequence (ROUGE-L). To facilitate comparison with the existing systems, we adopt ROUGE as the automatic evaluation method. The ROUGE is calculated on the character level, following the previous work  . However, for abstractive text summarization, the ROUGE is sub-optimal, and cannot assess the semantic consistency between the summary and the source content, especially when there is only one reference for a piece of text. The reason is that the same content may be expressed in different ways with different focuses. Simple word match cannot recognize the paraphrasing. It is the case for all of the existing large-scale datasets. Besides, as aforementioned, ROUGE is calculated on the character level in Chinese text summarization, making the metrics favor the models on the character level in practice. In Chinese, a word is the smallest semantic element that can be uttered in isolation, not a character.

Experiments	We evaluate the proposed approach on the Chinese social media text summarization task, based on the sequence-to-sequence model. We also analyze the output text and the output label distribution of the models, showing the power of the proposed approach. Finally, we show the cases where the correspondences learned by the proposed approach are still problematic, which can be explained based on the approach we adopt.

Experimental Analysis	To examine the effect of the proposed method and reveal how the proposed method improves the consistency, we compare the output of the baseline with Dual-Train, based on both the output text and the output label distribution. We also conduct error analysis to discover room for improvements. To gain a better understanding of the results, we analyze the summaries generated by the baseline model and our proposed model. Some of the summaries are listed in Table TABREF28 . As shown in the table, the summaries generated by the proposed method are much better than the baseline, and we believe they are more precise and informative than the references. In the first one, the baseline system generates a grammatical but unrelated summary, while the proposed method generates a more informative summary. In the second one, the baseline system generates a related but ungrammatical summary, while the proposed method generates a summary related to the source content but different from the reference. We believe the generated summary is actually better than the reference because the focus of the visit is not the event itself but its purpose. In the third one, the baseline system generates a related and grammatical summary, but the facts stated are completely incorrect.

Another representative example is the word “UTF8gbsn图像” (image), where the baseline also includes two particles in its most related words. The phenomenon shows that the baseline suffers from spurious correspondence in the data, and learns noisy and harmful relations, which rely too much on the co-occurrence. In contrast, the proposed method can capture more stable semantic relatedness of the words. For text summarization, grouping the words that are in the same topic together can help the model to generate sentences that are more coherent and can improve the quality of the summarization and the relevance to the source content. Although the proposed method resolves a large number of the noisy word relations, there are still cases that the less related words are not eliminated. For example, the top 4 most similar words of “UTF8gbsn期货业” (futures industry) from the proposed method include “UTF8gbsn改革” (reform). It is more related than “2013” from the baseline, but it can still be harmful to text summarization.

It is the same with the second example, where a human selects “UTF8gbsn专家” (expert) and Dual-Train selects “UTF8gbsn工作者” (worker), while the baseline selects “UTF8gbsn钻研” (research) and fails to generate a grammatical sentence later. In the third one, the reference and the baseline use the same word, while Dual-Train chooses a word of the same meaning. It can be concluded that Dual-Train indeed learns better word relations that could generalize to the test set, and good word relations can guide the decoder to generate semantically consistent summaries. To show why the generated text of the proposed method is more related to the source content, we further analyze the label distribution, i.e., the word distribution, generated by the (first) output layer, from which the output word is selected. To illustrate the relationship, we calculate a representation for each word based on the label distributions. Each representation is associated with a specific label (word), denoted by INLINEFORM0 , and each dimension INLINEFORM1 shows how likely the label indexed by INLINEFORM2 will be generated instead of the label INLINEFORM3 .

Methods for Obtaining Soft Training Target	Soft target aims to refine the supervisory signal in supervised learning. Related work includes soft target for traditional learning algorithms and model distillation for deep learning algorithms. The soft label methods are typically for binary classification  , where the human annotators not only assign a label for an example but also give information on how confident they are regarding the annotation. The main difference from our method is that the soft label methods require additional annotation information (e.g., the confidence information of the annotated labels) of the training data, which is costly in the text summarization task. There have also been prior studies on model distillation in deep learning that distills big models into a smaller one. Model distillation  combined different instances of the same model into a single one. It used the output distributions of the previously trained models as the soft target distribution to train a new model. A similar work to model distillation is the soft-target regularization method  for image classification. Instead of using the outputs of other instances, it used an exponential average of the past label distributions of the current instance as the soft target distribution.

Proposed Method	Base on the fact that the spurious correspondence is not stable and its realization in the model is prone to change, we propose to alleviate the issue heuristically by regularization. We use the cross-entropy with an annealed output distribution as the regularization term in the loss so that the little fluctuation in the distribution will be depressed and more robust and stable correspondence will be learned. By correspondence, we mean the relation between (a) the current output, and (b) the source content and the partially generated output. Furthermore, we propose to use an additional output layer to generate the annealed output distribution. Due to the same fact, the two output layers will differ more in the words that superficially co-occur, so that the output distribution can be better regularized.

The target of the output INLINEFORM7 includes both the one-hot distribution and the distribution generated from INLINEFORM8 : DISPLAYFORM0  The new output layer INLINEFORM0 is trained normally using the originally hard target. This output layer is not used in the prediction, and its only purpose is to generate the soft target to facilitate the soft training of INLINEFORM1 . Suppose the correct label is INLINEFORM2 . The target of the output INLINEFORM3 includes only the one-hot distribution: DISPLAYFORM0  Because of the random initialization of the parameters in the output layers, INLINEFORM0 and INLINEFORM1 could learn different things. The diversified knowledge is helpful when dealing with the spurious correspondence in the data. It can also be seen as an online kind of ensemble methods. Several different instances of the same model are softly aggregated into one to make classification. The right part of Figure FIGREF6 shows the architecture of the proposed Dual-Train method.

As the existing methods do not provide their test output, it is a non-trivial task to reproduce all their results of the same reported performance. Nonetheless, it does not change the fact that ROUGE cannot handle the issues in abstractive text summarization properly.

Introduction	Abstractive test summarization is an important text generation task. With the applying of the sequence-to-sequence model and the publication of large-scale datasets, the quality of the automatic generated summarization has been greatly improved  ,  ,  ,  ,  ,  ,  ,  ,  . However, the semantic consistency of the automatically generated summaries is still far from satisfactory. The commonly-used large-scale datasets for deep learning models are constructed based on naturally-annotated data with heuristic rules  ,  ,  . The summaries are not written for the source content specifically. It suggests that the provided summary may not be semantically consistent with the source content. For example, the dataset for Chinese social media text summarization, namely LCSTS, contains more than 20% text-summary pairs that are not related, according to the statistics of the manually checked data  . Table TABREF1 shows an example of semantic inconsistency. Typically, the reference summary contains extra information that cannot be understood from the source content. It is hard to conclude the summary even for a human.

Systems for Chinese Social Media Text Summarization	The Large-Scale Chinese Short Text Summarization dataset was proposed by  . Along with the datasets,  also proposed two systems to solve the task, namely RNN and RNN-context. They were two sequence-to-sequence based models with GRU as the encoder and the decoder. The difference between them was that RNN-context had attention mechanism while RNN did not. They conducted experiments both on the character level and on the word level. RNN-distract  was a distraction-based neural model, where the attention mechanism focused on different parts of the source content. CopyNet  incorporated a copy mechanism to allow part of the generated summary to be copied from the source content. The copy mechanism also explained that the results of their word-level model were better than the results of their character-level model. SRB  was a sequence-to-sequence based neural model to improve the semantic relevance between the input text and the output summary. DRGD  was a deep recurrent generative decoder model, combining the decoder with a variational autoencoder.

Dataset	Large-Scale Chinese Short Text Summarization Dataset (LCSTS) is constructed by  . The dataset consists of more than 2.4 million text-summary pairs in total, constructed from a famous Chinese social media microblogging service Weibo. The whole dataset is split into three parts, with 2,400,591 pairs in PART I for training, 10,666 pairs in PART II for validation, and 1,106 pairs in PART III for testing. The authors of the dataset have manually annotated the relevance scores, ranging from 1 to 5, of the text-summary pairs in PART II and PART III. They suggested that only pairs with scores no less than three should be used for evaluation, which leaves 8,685 pairs in PART II, and 725 pairs in PART III. From the statistics of the PART II and PART III, we can see that more than 20% of the pairs are dropped to maintain semantic quality. It indicates that the training set, which has not been manually annotated and checked, contains a huge quantity of unrelated text-summary pairs.

The problem could arise from the fact that words as “UTF8gbsn期货业” rarely occur in the training data, and their relatedness is not reflected in the data. Another issue is that there are some particles, e.g., “UTF8gbsn的” (DE) in the most related words. A possible explanation is that particles show up too often in the contexts of the word, and it is hard for the models to distinguish them from the real semantically-related words. As our proposed approach is based on regularization of the less common correspondence, it is reasonable that such kind of relation cannot be eliminated. The first case can be categorized into data sparsity, which usually needs the aid of knowledge bases to solve. The second case is due to the characteristics of natural language. However, as such words are often closed class words, the case can be resolved by manually restricting the relatedness of these words.

Regularizing the Neural Network with Annealed Distribution	Typically, in the training of the sequence-to-sequence model, only the one-hot hard target is used in the cross-entropy based loss function. For an example in the training set, the loss of an output vector is DISPLAYFORM0  where INLINEFORM0 is the output vector, INLINEFORM1 is the one-hot hard target vector, and INLINEFORM2 is the number of labels. However, as INLINEFORM3 is the one-hot vector, all the elements are zero except the one representing the correct label. Hence, the loss becomes DISPLAYFORM0  where INLINEFORM0 is the index of the correct label. The loss is then summed over the output sentences and across the minibatch and used as the source error signal in the backpropagation. The hard target could cause several problems in the training. Soft training methods try to use a soft target distribution to provide a generalized error signal to the training.

This work is supported in part by the GS501100001809National Natural Science Foundation of Chinahttp://dx.doi.org/10.13039/501100001809 under Grant No. GS50110000180961673028.

For the summarization task, a straight-forward way would be to use the current output vector as the soft target, which contains the knowledge learned by the current model, i.e., the correspondence of the source content and the current output word: DISPLAYFORM0  Then, the two losses are combined as the new loss function: DISPLAYFORM0  where INLINEFORM0 is the index of the true label and INLINEFORM1 is the strength of the soft training loss. We refer to this approach as Self-Train (The left part of Figure FIGREF6 ). The output of the model can be seen as a refined supervisory signal for the learning of the model. The added loss promotes the learning of more stable correspondence. The output not only learns from the one-hot distribution but also the distribution generated by the model itself. However, during the training, the output of the neural network can become too close to the one-hot distribution. To solve this, we make the soft target the soften output distribution. We apply the softmax with temperature INLINEFORM2 , which is computed by DISPLAYFORM0  This transformation keeps the relative order of the labels, and a higher temperature will make the output distributed more evenly.

The key motivation is that if the model is still not confident how to generate the current output word under the supervision of the reference summary, it means the correspondence can be spurious and the reference output is unlikely to be concluded from the source content. It makes no sense to force the model to learn such correspondence. The regularization follows that motivation, and in such case, the error signal will be less significant compared to the one-hot target. In the case where the model is extremely confident how to generate the current output, the annealed distribution will resemble the one-hot target. Thus, the regularization is not effective. In all, we make use of the model itself to identify the spurious correspondence and then regularize the output distribution accordingly.

Experimental Settings	We use the sequence-to-sequence model  with attention  ,  ,  ,  as the Baseline. Both the encoder and decoder are based on the single layer LSTM  . The word embedding size is 400, and the hidden state size of the LSTM unit is 500. We conduct experiments on the word level. To convert the character sequences into word sequences, we use Jieba to segment the words, the same with the existing work  ,  . Self-Train and Dual-Train are implemented based on the baseline model, with two more hyper-parameters, the temperature INLINEFORM0 and the soft training strength INLINEFORM1 . We use a very simple setting for all tasks, and set INLINEFORM2 , INLINEFORM3 . We pre-train the model without applying the soft training objective for 5 epochs out of total 10 epochs. We use the Adam optimizer  for all the tasks, using the default settings with INLINEFORM4 , INLINEFORM5 , and INLINEFORM6 . In testing, we use beam search to generate the summaries, and the beam size is set to 5. We report the test results at the epoch that achieves the best score on the development set.

To get such representation, we run the model on the training set and get the output vectors in the decoder, which are then averaged with respect to their corresponding labels to form a representation. We can obtain the most related words of a word by simply selecting the highest values from its representation. Table TABREF30 lists some of the labels and the top 4 labels that are most likely to replace each of the labels. It is a hint about the correspondence learned by the model. From the results, it can be observed that Dual-Train learns the better semantic relevance of a word compared to the baseline because the spurious word correspondence is alleviated by regularization. For example, the possible substitutes of the word “UTF8gbsn多长时间” (how long) considered by Dual-Train include “UTF8gbsn多少” (how many), “UTF8gbsn多久” (how long) and “UTF8gbsn时间” (time). However, the relatedness is learned poorly in the baseline, as there is “UTF8gbsn知道” (know), a number, and two particles in the possible substitutes considered by the baseline.

Related Work	Related work includes efforts on designing models for the Chinese social media text summarization task and the efforts on obtaining soft training target for supervised learning.

Due to the inconsistency, the system cannot extract enough information in the source text, and it would be hard for the model to learn to generate the summary accordingly. The model has to encode spurious correspondence of the summary and the source content by memorization. However, this kind of correspondence is superficial and is not actually needed for generating reasonable summaries. Moreover, the information is harmful to generating semantically consistent summaries, because unrelated information is modeled. For example, the word UTF8gbsn“利益” (benefits) in the summary is not related to the source content. Thus, it has to be remembered by the model, together with the source content. However, this correspondence is spurious, because the word UTF8gbsn“利益” is not related to any word in the source content. In the following, we refer to this problem as Spurious Correspondence caused by the semantically inconsistent data. In this work, we aim to alleviate the impact of the semantic inconsistency of the current dataset.

Question: What human evaluation method is proposed?
Answer: comparing the summary with the text instead of the reference and labeling the candidate bad if it is incorrect or irrelevant