Encoder-decoder Network as Loss Function for Summarization
Authors: Glen Jeh
Venue:  (2020)
Keywords: encoder-decoder
summarization
loss functions

Abstract: We present a new approach to defining a sequence loss function to train a summarizer by using a secondary encoder-decoder as a loss function, alleviating a shortcoming of word level training for sequence outputs. The technique is based on the intuition that if a summary is a good one, it should contain the most essential information from the original article, and therefore should itself be a good input sequence, in lieu of the original, from which a summary can be generated. We present experimental results where we apply this additional loss function to a general abstractive summarizer on a news summarization dataset. The result is an improvement in the ROUGE metric and an especially large improvement in human evaluations, suggesting enhanced performance that is competitive with specialized state-of-the-art models.

Under review as a conference paper at ICLR 2020
ENCODER-DECODER NETWORK AS SEQUENCE Loss FUNCTION FOR SUMMARIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
We present a new approach to defining a sequence loss function to train a sum- marizer by using secondary encoder-decoder shortcoming of word level training for sequence outputs. The technique based the intuition that if a summary is good one, it should contain the most es- sential information from the original article, and therefore should itself be a good input sequence, in lieu of the original, from which summary can be generated. We present experimental results where we apply this additional loss function to general abstractive summarizer on a news summarization dataset. The result is an improvement in the ROUGE metric and an especially large improvement in human evaluations, suggesting enhanced performance that is competitive with specialized state-of-the-ee models.
loss function, alleviating a
INTRODUCTION
Neural networks are popular solution to the problem of text summarization, the task of taking as input piece of natural language text, such as a paragraph or a news article, and generating a more succinct text that captures the most essential information from the original.
One popular type of neural network that has achieved state of the art results an attentional encoder- decoder neural network See et al. (201)); Paulus et al. (2018); Celikyilmaz al.) (2018). In an encoder-decoder network, the encoder scans over the input sequence by ingesting one word token at time create internal representation. The decoder trained compute a probability distribu- tion over next words conditioned on a sequence prefix. A beam search decoder is typically used to find a high likelihood output sequence based these conditional word probability distributions.
Since the next word depends heavily on previous words, the decoder has little hope of correct distribution over next words unless has the correct previous words. Thus the decoder is typically trained using teacher forcing WWilliass & Zipser (1989), where the reference sequence prefix is always given to the decoder each decoding step. In other words, regardless of what distributions are output by the decoder in training for timesteps (1, 1). at timestep f, it is given the reference sequence prefix (vī ..., v( and asked to output the distribution P(ye "i.,..,' v(-1).
Teacher foreing suffers from exposure bias Ranzato et al. (2016). At test time, the input to the decoder not the reference sequence prefix (yi,.... v2-1) but (y, constructed based on the decoder's s own outputs. Since the decoder is not perfect, as t increases too does the disparity between these two sequence prefixes, causing an increasing mismatch between training and test conditions.
Training at the sequence level can alleviate this discrepancy, but requires a differentiable loss func- tion. In the Related Work section review previous efforts.
We present a novel approach to address the problem by defining loss function at the sequence level using an encoder-decoder network decoded output sequence is fed as input into another network called the recoder. The recoder is an independent attentional encoder-decodes trained produce the reference summary.
loss function. In training, the summarizer's beam search
Our experiments show that adding the recoder as a loss function improves a general abstractive summarizer on the popular CNN/DailyMail dataset Hermann al. 2015): Nallapati et al. (0116).

---

Under review as a conference paper at ICLR 2020
achieving significant improvements in the ROUGE metric and an especially large improvement in human evaluations.
2 ATTENTIONAL ENCODER-DECODER MODEL FOR SUMMARIZATION
We first give a high level overview of attentional encoder-decoder model for sequence to sequence learning. We present the specific abstractive model See et al. (0017). which serves as the bascline model in our experiments for comparison. We chose this as the baseline model because is general and popular model whose concepts have often appeared in other abstractive summarization works, and its results have often been used as a comparison baseline & Bansal (2018): Li et al. (0018;; Celikyilmaz et al. (2018).
The ideas presented in this paper are largely independent of the specific summarizer and may be applicable to the training of other sequence to sequence models, many of which also trained at the word loss level using teacher forcing. Since our focus to demonstrate the effectiveness of our new loss function, the generality of the See al. (2017) model, which does account for specifics of the summarization problem such as article or sentence structure Li et al. (2018), suits our purpose well.
2.1 INPUTS AND OUTPUTS
The source article and target summary are each treated sequence of tokens. A token represents either a word or a punctuation mark. Let (zz) be the sequence of tokens in an article or input sequence, where ranges from to the length of the input sequence. Let (yt) be the sequence of word tokens in the summary or output sequence, where < f.
The goal is to train a neural network to compute. at each timestep t, the probability distribution P(y.|y,,...,, 11- over next tokens given previous tokens (y), Mt- 1)- In the teacher forcing training method, the network is trained to minimize the loss Jml, also called the maximum likeli- hood loss, defined as the average over timesteps t of (Jml); = log (vv li. where the sequence prefix vi..... y 1 is from the reference summary.
At test time, the network computes search decoder based on previously output distributions. The sequence ends when the special token STOP, appended to the end of all reference summaries in training, is output.
Bt-11) where ,...., It-1 are chosen by a beam
2.2 ENCODING
Each v; is embedded as a dense vector w; using a u x weight matrix Wemb, where u the size of the vocabulary and d is the number of hidden dimensions, hyperparameter. The weights Wemb are learned as part of training. Each w; is given in order as input into an LSTM to get hidden states (h²", ), and also in reverse order into another LSTM to get backward states (h?). The two state sequences are then stacked form hidden states (h:) for reference by the decoding phase's attention mechanism.
2.3 DECODING WITH ATTENTION
The decoding phase also uses an LSTM. At each output timestep a token y is embedded as dense vector using the same W emb and fed into the LSTM, and the LSTM outputs probability distribution over next tokens. For = 0, y is the special placeholder symbol START. For t > O, when training using teacher forcing, y V/-1 is the corresponding token from the reference sequence. When beam search decoding at test time, y = yt-1 is chosen based on the decoder's 's output probabilities at step / 1. This difference in information about previous output tokens causes discrepancy between training and test conditions.
The decoder makes use of an additional piece of information called attention Bahdanau al. 2015). which allows it to "read" the encoder's output states. At each timestep with decoder LSTM state St. the attention vector is computed as
at softmax(g" tanh + West + battm)
2

---

Under review as a conference paper at ICLR 2020
where 9, Wh. Wes and batta are parameters be trained. This vector gives distribution encoder states at timestep t. The attentional context vector h,, a "reading" of encoder states, is then computed as
h; Σ (at)shs
This context vector is concatenated with the LSTM state s to produce the dense vector outputs = [s, hi] + b, where I] denotes concatenation of vectors. Finally, another linear transformation using a weight matrix V' and weight vector D' is used map these outputs in the hidden space, through a softmax layer, into output probabilities over the vocabulary. That is,
Pvocab (y4 Mi...., ye-1) softmax 6')
where V, Vi, 6, are trainable parameters.
2.4 POINTER GENERATOR AND COVERAGE
The pointer generator mechanism allows the decoder to copy token directly from the input. is particularly helpful for out-of-vocabulary tokens. The decoder "copies" source tokens with distribution based on the attention vector ag and mixes in with weight Pgen. where Pgen is trainable function of the context vectors, states, and next word input. The final distribution P(yelyi.... Vt-1) the combined
Pgen Pvocab (yelV1, 1t-1) + (1 Pgen)
Σ o
The summarizer loss function Js Jml is computed based on this distribution.
In one model variant, a coverage loss Joov that encourages attention vectors a, to look at different encoder states i also added to the final loss function. That is, S = Jml + Joov.
Full details the pointer-generator and coverage mechanisms are found in See et al. (017). We have omitted them here for brevity. The recoder does not depend on these particulars the sum- marization model. We focus the quantities Pvocab. P, and the total summarizer loss Js. Specific details of how these are computed are not needed in the following sections.
RECODER
We are now ready to present our main contribution. The recoder neural network model that takes the decoded sequence from the summarizer input and is trained to output the reference summary. The purpose of the recoder is to serve as sophisticated loss function help train the summarizer, backpropagating errors the summarizer's decoded output sequence during training. The intuition that a good output sequence from the summarizer should allow the recoder to produce a good summary.
In principle, the recoder can be any sequence to sequence model that can backpropagate to its inputs. the same model structure the summarizer itself. For our experiments
One obvious choice we found was sufficient to use the same network structure with lower dimensions, with the same attentional pointer-generator encoder-decoder network as the summarizer, but with half the number of hidden dimensions (from 256 to 128) and GRU Cho et al. (2014) instead of an LSTM in the encoder. This helped reduce the amount of memory required in training.
Cho
3.1 RECODER INPUTS
Our first task is represent the summarizer's decoded outputs as inputs to the recoder in way that is differentiable. A beam search decoder will find sequence (y1. /2.... of high average log probability log (v (y.,..., Ut-1)) over timesteps We cannot this discrete token sequence directly, but we can look at the underlying signals that determine the choice of each token.
The output token y/ chosen based the computed probability distributions Po M--1). Let us denote this distribution by P. for short. For a beam search of width k the chosen token y will have probability in P. that among the k highest. The exact choice is determined by the beam
3

---

Under review as a conference paper at ICLR 2020
search mechanism, do not have continuous function that relates Pa directly yt. However, Pt does determine the range of choices, and feeding it as an input to the recoder can ensure that contains the right information. Since each P. is computed by the summarizer based the decoded prefix sequence (y1,-., 1t-1), propagating error to P. improves the summarizer via exposure to test time, even if the summarizer is not "aware" of the search mechanism and
what it would cannot optimize for it.
Since Pt has dimensions equal to the size of the vocabulary (v 50000 in the experiments), multiply Pt by a weight matrix to produce a dense representation in lower dimensional space (128 the experiments), just as we did for the one-hot token imputs of the summarizer. While we can use a new weight matrix here, reusing the same embedding matrix Wemb as the summarizer helps avoid increasing the number of parameters. For the same reason we also reuse the summarizer's 's mapping V back to vocabulary space. Together they account for 90% of the summarizer's s parameters.
The input to the recoder is the sequence (w() where
wi R
P.Wemb.
with Pt treated vector of length By training the recoder output the reference summary, errors are propagated to its inputs w and then to P.-
Note that although the summarizer's input w; is in effect Wemb multiplied with one-hot vectors corresponding to Fj, P; is the output of a softmax function and will be more evenly distributed than one-hot vector. Embedded representations w will thus be weighted sum of embedding vectors instead of individual ones like wi. The recoder can accommodate this difference since is an independent network with its own parameters.
3.2 RECODER TRAINING
The recoder can be trained jointly with the summarizer from scratch. We have found that it also suffices add the recoder to the pretrained summarizer and continue training jointly using their combined losses.
The recoder is trained using teacher forcing to produce the reference summary, analogously to Jul- To be specific, we minimize the loss function J equal to the average of - log PR (y: lor .... vr-1) across timesteps f, where p8 denotes the recoder's 's output distributions. This maximum likelihood loss is sufficient for the recoder because its output never used for decoding test time.
3.3 LENGTH Loss
Meaningful recoder output depends on relevant information from the original article being encoded in its input wf'. However, JH only places requirements the presence of information. We have not placed any requirements on brevity, so longer sequences wff would likely yield better results under this loss metric, barring any confounding effects from too much information. The effect is only constrained because recoder training is performed jointly with the summarizer, and summaries that are longer than reference summaries would tend to do worse with respect to Js Intuitively, JR, encourages recall of information. We add a loss function on length to counterbalance for precision.
The end of output sequence is determined when the special STOP token is output. Actual words are likely to convey more useful information to the recoder, so training using JR, lowers the STOP token's output probability. We can control length by applying a loss to every other token for timesteps beyond a desired length. We define the length loss as the average over of
(Jien): X Penalty(()(1 P((SO)))
The function Penalty(t) > defines penalty curve for timesteps beyond desired length, while hyperparameter defines the desired tradeoff between precision and recall.
Other ways to control output length possible, such by explicitly adding the desired length into decoder state Kikuchi et al. (2016); Fan et al. (2017), although these methods require changes to the model.
4

---

Under review as a conference paper at ICLR 2020
some Icenngers get driving lossons from their parents other leens are imught by licensed instructors. but malia obama a't average 16-year-old ber driving lessons provided the
Article
drive, lisst loly michelle obama celebrilyy chel and time talk--show rachadl my in an inserview that it was the armed agents who provide around the clock security for the fumily. [..]
asked taught
service.
first bay michelle ohaima told celebrity chef and daytime talk host rachnel familyy..
interview
PRenLCOv
agents who provide hound-Ihe-clock security
mrs, ohama has n'1
seven or eight years..
malia chama, seen with her mocher michelle ohama apeil 2009, reportedly was taught how to drive by secret service agents. but mulia oboma n's tought malla average year her driving lessons provided by the lody michelle ohuma said
service.
pgen.cov-recoder
interview
that it was the ammdd agents who provide around-tte tea-tthe-llleee clock security for the family...
michelle obuma told talk show ruchael that mus. she added that driving gives malia sense of nommalyy helping her feel like the rest her friends who are also driving.
service agents tought daughter malia how drive.
reference
driven
seven or eight years said.
Figure I: Sample output from the CNN//ailyMail dataset.
3.4
COMBINED Loss
The final recoder loss JR is comprised of the teacher forced recoder loss and the length loss
J
J Jien
Abstracting away individual components of the summarizer's and recoder's loss functions, the end end training is performed using combination of their losses J:
J + JR
3.5 EXAMPLE
In Figure 1 comparing output summaries from the baseline pgen_cov model trained with J (de- scribed in Experiments section below) and the pgen.cov+zecodes model additionally trained with loss f + JJ. the latter's 's output contains mention of "malia", relevant name in the article that also appears in the reference summary.
If we trained only with the recoder loss JR, we would encourage the summarizer to output the right information, but the output may not conform language model, except to the extent helps make the output more intelligible for the recoder. For the purpose of illustration, we continued training the pgen_cov model using only JR, produced the following summary for one of the articles in the CNN/DailyMail dataset:
boston 's mi serable winter is now also its snowiest season it had special 2.9 9 inches pushed the city into 108.6 inches in one season
This example output contains some relevant information but has grammatical errors.
4 RELATED WORK
Summarization is well studied problem, including both EestractiveZhhueee al. (2018) and abstractive approaches Zeng et al. (2016); Nallapati et al. (2016); Rush et al. (2015). Many of the existing abstractive models are based on encoder-decoders and trained using teacher foreing. In this work we focus on improving the training of models such as these, and we have picked one particular model as the baseline for improvement. Many aspects of the [See et al. (2017) baseline model also appear in other works. For example, [i ct al. (2018) applies similar attentional coverage mechanism at the sentence level, while Chen Bansal (2018); Celikyilmaz et al. (0018) employ pointer-generan mechanism at the word sentence levels.
While more recent work, such as Celikyilmaz et al. 20188) ) Chen & Bansal (008 Li et al. 2018
have reported better ROUGE results on the CNN/DailyMai dataset, chose the See et al. (2017) model for its generality. By using an unspecialized model with commonly occurring elements as baseline, hope the same concepts can apply to more advanced and specialized models.
5

---

Under review as a conference paper at ICLR 2020
4.1 REINFORCEMENT LEARNING
There are a number of related techniques to address exposure bias from teacher forcing in sequence learning problem. One class of such techniques motivated by the ideas of reinforcement learning (RL) for training actors to make discrete decisions Paulus et al. 2018 Ranzato al. (2016); Rennie et al. 2017); Bahdanau et al. (2016).
actor being trained to learn a policy for choosing output word at each time step. These decisions are evaluated in training not based on its output probabilities, but based on reward function defined on the full output sequence, selected via beam search other methods.
The encoder-decoded is analogous
Broadly speaking, there couple ways turn rewards into gradients for training probability dis- tributions at each timestep. One technique is the REINFORCE algorithm, [Williams| 1992);; Rennie al. (0177;; Paulus et al. (2018), which is based on computing expected future rewards, and the related technique of minimum risk training in Makino et al. (2019). Another approach is to train an
estimator called critic model to estimate expected rewards, as in Ranzato et al. (2016) Bahdanau
et al. (2016).
The common key to these approaches is the reward function. Typically the reward is defined based on n-gram overlap, such as the use of ROUGE Paulus et al. (2018); Rennie et al. (017). BLEU Ranzato et al. (2016), or METEOR Bahdanau et al. (2016) scores. The reward function serves as one component, while a technique such as REINFORCE or critic model serves as the second component. Together they turn full sequences into gradients for training the output of each timestep.
Cast in these terms, the recoder serves the role of both components. However, instead of training toward a heuristic such as ROUGE, the recoder uses an encoder-decoder network, allowing it to account for complexities in evaluation of quality that are sophisticated the model being trained. An algorithm can only be as good as the metric toward which it is trained, and the recoder helps case this upper bound quality. In short, the key difference from reinforcement learning approaches is that the recoder replaces a simple reward heuristic with an encoder-decoder and its loss function.
4.2 SEQUENCE OUTPUT LOss
Another approach to account for the beam search decoded output sequence is Wiseman Rush (201)). In this work, beam search is performed as part of training. and loss is defined that pe- nalizes the reference sequence prefix falling out of the beam. In Goyal et al. (2017) on two non- summarization word sequence tasks, a differentiable approximation to beam search is used, and loss defined on the sequence using Hamming distance. While these approaches can account for the decoder test time sequence output, they do not have the flexibility to credit alternative summaries that differ slightly in phrasing. In comparison, our approach can account for a range of correct sum- maries that differ from one another, regardless of subsequence overlap. Since the recoder is itself type of summarization network, it too can keep pace should further improvements in summarizers be developed that output more varied summaries of high quality. However, using differentiable approximation to beam search enhances backpropagation, and may be direction for future im- provement.
The idea of re-encoding an output summary also appears in Chu & Liu 2019). where Straight- Through Gumbel-Softmail reparaameterizationnaang et al. (2017) is used to derive a loss function that allows backpropagation. Their reconst ructi on cycle loss variant is closest in concept to our work, except that since there is no reference summary in their problem, they train their analogue of the recoder to produce the original source. We did not take this approach because in general the summary is a lossy representation of the original, so training the recoder to produce it would subject it large source of error that it cannot possibly reduce.
In the extension Xu et al. (2019) to scheduled sampling for machine translation, a decoded sequence prefix is used in training predict the next reference word after weighting over possible alignments, which allows for flexibility in word ordering. For the summarization problem, alternative summaries that differ by more than word alignment can still be of high quality.

---

Under review as a conference paper at ICLR 2020
4.3 COMPLEMENTARY MODELS
The idea of using a second model to help train the primary model has appeared in other contexts. In Sennrich et al. 2016); He et al. (2016) for machine translation, the decoded output of translation model is fed as ippt into a second backtranslation model. The two separate models translate back and forth between source and target languages. We can think of the backtranslation model from target to source language as serving an analogous role to the recoder, although the aim of these works has been to generate synthetic training data to augment limited human training data, rather than to compute gradients directly in an end-to-end model for the purpose of improving the original network. In contrast to the translation problem, we also cannot expect the original article to be recreated from the summary, which in general will be a lossy representation of the article, so the recoder is asked to recreate the reference summary and not the original article. An analogous idea also appears in the problem of sentence compression Miao & Blunsom (2016); Févry & Phang (2018), where decoding from shortened version of a sentence back to the original sentence is analogous backtrans lattnn from target source language.
In the related work Xia et al. (2018) on dual learning, the two models for the forward and backward problems share parameters to mutual benefit. In contrast, the recoder shares parameters with the summarizer except vocabulary mappings, relying instead on direct backpropagation.
In Kim & Rush (2016), the decoded output sequence of a translation model is used train student model to mimic the original model's output. The aim there is to better distill knowledge into a smaller student model, not improve the original model.
EXPERIMENTS
We ran experiments test the effectiveness of adding the loss function the attentional encoder- decoder summarization model of [See et al. (201)). The Tensorflow code, which is based on their published code, will also be made public along with trained models.
We started with the provided trained summarizer models and added on recoder losses and continued training. The additional training consumes about double the memory and computation time, as requires beam search decoding as part of training, using beam width of 4. The additional time was about 24 hours for each model, using an Nvidia Tesla P100 GPU on a cloud service provider. Test times are not affected since the recoder is only used during training.
We ran experiments using the two versions of the model from See et al. (2017). with and without the coverage mechanism, as baselines. The model with coverage is pgen_cov, while pgen omits the coverage mechanism. In terms of loss functions, pgen_cov is trained using a loss of J = Jeov presented above, while pgen is trained using J = Jml. In either case, the comparison models with recoder is additionally trained using JS + J
We used the summarizer's (Pvocab); directly in place of P; in computing recoder inputs, bypassing backpropagation to the pointer-generane mechanism, which is most helpful for of words that will be treated as UNK for w anyway. This yielded similar results while converging faster and requiring less computational resources.
We fixed the length penalty Penalty(t) to be a graduated curve 1.04max(t-0.8L.00 where L is the length of the reference summary for the example. We experimented with different settings of X to see its effects on output lengths and quality. The final setting of X = 0.1 used for the ROUGE comparison in Table 1, diseussed below, selected based its having the highest ROUGE-1 score on the validation set, where scores appeared in the same relative order as on the test set results shown in Table 2.
5.1 ROUGE COMPARISONS
In line with the baseline and for comparison with previous work, we first assess quality using the ROUGE metric Lin 2004).. The recoder can in principle capture more complex notions of quality than ROUGE, so this heuristic cannot fully capture all aspects of improvement that the human eval- uations described below can. Nonetheless, since we do not train toward this metric, should serve independent indicator of overall quality improvements.
7

---

Under review as a conference paper at ICLR 2020
Model
ROUGE-1
ROUGE-2
ROUGE-L
Pgen (baseline)
36.08
15.60
32.95
pgentrecoder (A 0.1)
37.07
16.02
33.83
pgen_cov (baseline)
39.47
17.37
36.26
pgen covtrecoder (A = 0.1)
40.44
18.15
36.90
lead-3
39.94
17.45
36.23
hierarchical Li et al. (2018)
40.30
18.02
37.36
Pgen (postproc)
39.28
17.40
36.13
pgen+recoder (postproc)
39.42
17.30
36.18
pgen.cov (postproc)
39.83
17.56
36.61
pgen_cov+recoder (postproc)
40.71
18.20
37.12
Table I: ROUGE scores for I-gram, 2-gram, and longest common subsequence on Call test set. The scores shown for pgen and pgen_cov differ slightly from that listed in [See et all (2017). c.g. 39.47 vs 39.53 for ROUGE-I. I. The pretrained models provided with their published code resulted in lower scores (differences may be due to changes in the underlying software libraries over time), but were able achieve these comparable scores by further training with lower learning rate.
The results shown in Table For pgen, we that adding the recoder loss results in an improvement in scores from (36.08, 15.60, 32.95) to (37.07, 16.02, 33.83). For the higher scoring pgen_cov baseline, adding the recoder loss improves scores from (39.47, 17.37, 36,26) to (40.44, 18.15, 36.90). In both cases we see close to a point improvement in ROUGE-1 scores, and smaller improvement in ROUGE-2 and ROUGE-L scores versus the baseline.
We see that adding the recoder to two variations of strong abstractive summarizer has improved their performance by significant amount, even though we have not changed the models, only the loss function used to train them. These scores also improvement over using the first sentences from the article as the summary (lead-3), which had outperformed the baseline models.
To get a sense of the magnitude of improvement, we look at one recently published model that uses the same non-anonymized version of the dataset without further processing, allowing for a relatively direct comparison. The "hierarchical" model Li et al. (2018) is an abstractive model that accounts for sentence level structure. Overall results are close to pgen.covi recoder, which has higher ROUGE-1 and ROUGE-2 scores but lower ROUGE-L. Applying the recoder to generic baseline has resulted in ROUGE gains that match that model with more advanced mechanisms.
There other sophisticated abstractive models with higher reported ROUGE scores, especially those that are extractive Hybrids/Chen & Bansal (2018) or employ RI. techniques Paulus et al. (2018); Chen & Bansal (2018): [Celikyilmaz al.] (2018), with high of 41. 1.69 ROUGE-1 score in the latter work after eliminating duplicate trigrams in post-processing. If we also eliminated duplicate trigrams in post-processing, the pgen_cov re reoder scores would be (40.71, 18.20 37.12), but full treatment should also eliminate duplicate trigrams during training, requiring far more complex implementation. These models are generally more tuned to the specific problem and metrie, such as accounting for sentence boundaries or including the ROUGE metric as reward, and it is reasonable that they achieve higher scores on this metric.
Since the recoder has the potential to capture notions of similarity beyond n-gram overlap as used in ROUGE, we performed further evaluation using human readers.
Parameler À
Length
R-1
R-2
R-I.
87.1
40.17
18.17
36.80
74.9
0.2
68.7
40.29
18.02
36.70
0.3
52.1
39.18
17.24
35.52
pgen_cov (baseline)
Table 2: Length and ROUGE scores on CNN//ailyMail test set for different settings of length loss weight À the pgen.cov+recodee model.
8

---

Under review as a conference paper at ICLR 2020
Model
Overall
Readability 164 (54.7%)
Relevance
pgen.cov+recoder (A 0.1)
181 (60.3%)
188 (62.7%
pgen_cov+recoder (A = 0.2)
165 (55.0%)
160 (53.3%)
166 (55.3%)
Table 3: Human rater preference versus pgen.cov on 300 random examples from the CNNi test set. 95% confidence intervals for pgen.cov+recoder (A 0.1) preference are (54.6%. 65.9%) overall, (48.8%, 60.4%) readability, and (56.9%, 68.2%) relevance, and (49.2%, 60.7%), (47.5%, 69. 69.1%) (49.5% 0.2).
61.1%) respectively for pgen.cov+recoder
5.2 HUMAN EVALUATIONS
We ran human evaluation experiments comparing both the pgen covere 0.1 model and the À 0.2 model against the pgen cov baseline. The À 0.2 model's average summary length of 68.7 is close to the baseline's 66.1, which minimizes differences due to length.
From the CNN/Daily Mail test set where the model gave beam search decoded outputs that differed from the baseline (95.2% for À = 0.1 model and 95.5% for À = 0.2 model, out 11490 examples), we randomly sampled 300 examples from each model. Workers from Amazon Mechanical Turk were shown up to the first 400 tokens from the article (same as model inputs), the reference summary, and the two generated summaries, randomly assigned to side A and side B. They were asked to select an overall preference between and B, and then preference in terms of readability and relevance. Since do not need confident score for any example for the purpose of evaluating the model, we limited each example to one worker and each worker to 5 examples to increase diversity across example-worker pairs.
Workers were asked the following questions with no other guidance on interpretation:
Which summary is better overall?
Which summary has better readability?
which summary contains information that is more relevant?
The preference results are shown in Table 3. We see that pgen_cov+recoder was preferred 0.2) of the time over the pgen_cov baseline. we account for the remaining 4.8% and 4.5% of cases where the models gave outputs identical to the baseline by assigning equal preference to them, the preference ratios would be adjusted slightly to 59.8%, 54.8% overall and 62.1 % 55.1% % relevance respectively. The overall improvement may be largely explained by the improvement in relevance, as the relevance preference was different from overall preference for only 19 and 25 examples respectively.
overall 60.3% (A 0.1) and 55.0%
The most direct comparison from previous work may the human evaluations of Chen Bansal (2018) that had also compared their model of 40.88, against the pgen_cov baseline. They allowed a choice of "Equally good/bad" which survey did not, but if assign those ratings equally to the two sides, their results suggest preference for relevance 52.8% of the time.
with a ROUGE-1 score
If we similarly split "equal" ratings in the head-to-head human evaluations in Celikyilmaz et al. (2088), their overall preference results would be 59.3%. However, their comparison pitted a model (m7) with ROUGE-I score of 41.69 against baseline model (m3) that had only achieved ROUGE-1 score of 38.01, which is lower than our pgen_cov comparison baseline.
Differences such post-processing and survey format make these comparisons imprecise, but they give us a sense of the magnitude of improvement reflected by a 54.8%-59.88% overall preference over the pgen_cov baseline.
CONCLUSION
We have presented the use of an encoder-decoder as a sophisticated loss function for sequence outputs in the problem of summarization. The recoder allows define differentiable loss function on the decoded output sequence during training. Experimental results using both ROUGE and human evaluations show that adding the recoder in training general abstractive summarizer

---

Under review as a conference paper at ICLR 2020
significantly boosts its performance, without requiring any changes to the model itself. In future work we may explore whether the general concept of using a model as loss function has wider applicability to other problems.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. Proceedings of the International Conference on Learning Repre- sentations, 2015.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. Proceedings of International Conference on Machine Learning. 2016.
Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. Deep communicating agents for abstractive summarization. Proceedings of NAACI-HLT, 2018.
Yen-Chun Chen and Mohit Bansal. Fast abstractive summarization with reinforce-selected sentence rewriting. Proceedings of the Association for Computatioonal Linguistics (ACL), 2018.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
Eric Chu and Peter Liu. Meansum: A neural model for unsupervised multi-document abstractive summarization. Proceedings of International Conference on Machine Learning, 2019.
Angela Fan, David Grangier, and Michael Auli. Controllable abstractive summarization. Proceed-
ings of the 2nd Workshop on Neural Machine Translation and Generation, 2017.
Thibault Févry and Jason Phang. Unsupervised sentence compression using denoising auto- encoders. Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL.,, 2018
Kartik Goyal, Graham Neubig, Chris Dyer, and Taylor Berg-Kirkpatrick. continuous relaxation of beam search for end end-to-end training of neural sequence models. Proceedings of the Thirty-Secont AAAI Conference on Artificial Intelligence (AAAI-18), 2017.
Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tieyan Liu, and Wei-Ying Ma. Dual learning for machine translation. In Advances in Neural Information Processing Systems, pp. 820-828, 2016.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines read and comprehend. In Advances in Neural Information Processing Systems 28, pp. 1693-1701. me, 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzation with gumbel-softmax, Pro- ceedings of the International Conference on Learning Representations, 2017.
Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. Control- ling output length in neural encoder-decoders. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016.
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016.
Wei Li, Xinyan Xiao, Yajuan Lyu, and Yuanzhuo Wang. Improving neural abstractive document summarization with structural regularization. Proceedings of the Conference Empirical Meth- ods in Natural Language Processing (EMNLP), 2018.
Chin- Yew Lin. Rouge: package for automatic evaluation of summaries. Text Summarization Branches Out, 2004.
10

---

Under review as a conference paper at ICLR 2020
Takuya Makino, Tomoya Iwakura, Hiroya Takamura, and Manabu Okumura. Global optimization under length constraint for neural text summarization. In Proceedings the of the Association for Computational Linguistics, 2019.
the 57th Anmual Meeting
Yishu Miao and Phil Blunsom. Language as a latent variable: Discrete generative models for sen- tence compression. Proceedings of the 2016 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), 2016.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. Abstractive text summarization using sequence- to-sequence mns and beyond. In Proceedings of the 20th SIGNLL Conference Computational Natural Language Learning (CoNLL), Berlin, Germany, 2016.
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. Proceedings of the International Conference on Learning Representations, 2018.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level train- ing with recurrent neural networks. Proceedings of International Conference on Learning Repre- sentations, 2016.
Steven Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2015.
Abigail See, Peter Liu, and Christopher Manning. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics, 2017. Code: httpt:: //gt thub. abi p
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. Proceedings of the 54th Annual Meeting of the Association for Computa- fional Linguistics, 2016.
Ronald J Williams.
Simple statistical grllient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-255 1992.
Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1:270-280, 1989.
Sam Wiseman and Alexander M Rush. Sequence-to-seque seequencce learning as beam-search optimiza- tion. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP, 2016.
Yingce Xia, Xu Tan, Fei Tian, Tao Qin, Nenghai Yu, and Tie- Yan Liu. Model-level dual learning. In Proceedings of the International Conference on Machine Learning, pp. 5379-5388, 2018.
Weijia Xu, Xing Niu, and Marine Carpuat. Differentiable sampling with flexible reference word or- der for neural machine translation. Proceedings of the North American Chupter of the Association for Computational Linguistics (NAACL). 2019.
Wenyuan Zeng, Wenjie Luo, Sanja Fidler, and Raquel Urtasun. Efficient summarization with read- again and copy mechanism. arXiv preprint urXiv:161 161110338. 2016.
Qingyu Zhou, Nan Yang. Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. Neural document summarization by jointly learning to score and select sentences. Proceedings of the Association for Computational Linguistics (ACL), 2018
11


Reviews:
[{"review_id": "rklzwRhQKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use an additional component to the commonly used encoder-decoder approach for summarization, which is referred to as the recoder, which is an RNN-syle component that takes the output of the decoder. The intuition offered in the paper is that a good summary should produce itself via the recoder network, and in training it together with the original encoder-decoder it should improve its performance as it would be able to capture more than what the word-level loss does.\n\nI have the following objections to this paper:\n- I can't see why this extra component should improve the quality of the summary produced. If say our encoder-decoder architecture models the training data perfectly, then a recoder that does not do anything would be the right choice. Taking this further, a recoder could actually be fixing problems in the output of the decoder, and thus not providing a good training signal. It doesn't make sense to my that a summary should produce itself via a neural network, unless we are training an auto-encoder. The experiments validate this; the difference in ROUGE score are less than a point, which is the kind of fluctuation one expects due to random seed differences, etc.\n\n- I find it odd that ROUGE score is dismissed as a loss to train against (referred to as a heuristic in the end of section 4.1), but then 1 ROUGE point difference is considered a \"significant\" improvement (making such claims without statistical significance testing is misleading). Sure it has flaws, if there is something better why not use it for evaluation? Furthermore, claiming that the recoder does a better job requires evidence. Why not train this extra function against human judgements? Assuming that what is wanted is to train a model that estimates the quality of the summary, it would make sense to look at the approaches used for the similar task of machine translation quality estimation: https://www.mitpressjournals.org/doi/pdf/10.1162/tacl_a_00056\n\n- The approach is in my view a kind of actor-critic approach; the recoder could play that role, and in fact the Bahdanau et al. 2016 paper cited does this. However no comparison is offered, be it theoretical or experimental. Furthermore, the criticism that sequence-level training requires differentiable losses is incorrect; MIXER for example does train against scores such as ROUGE that are not differentiable. Furthermore, the BSO approach by Wiseman and Rush (2016) cited does give a continuous output to optimize for seq2seq that is asked for in the beginning of page 4.\n\n- In the experiments, how was the length penalty determined to be the graduated curve mentioned in section 5? I would expect to have comparison against other approaches that try to train encoder-decoder to improve summarization, such as those mentioned in section 4.1.\n\n- On the human evaluation experiments: the difference between the two models is quite small, especially given that the workers were not allowed to say that the models were equally good/bad. Furthermore, there is no inter-annotator agreement, as each comparison was done by a single crowd worker. Showing only the first 400 tokens of the original document would incorrectly disadvantage models selecting content from later in the articles. Finally, showing the reference summary creates another bias, since equally good summaries can disagree on what content to include. It might be helpful to look at some recent work on manual evaluation of summarization that tried to address these issues: https://arxiv.org/abs/1906.01361\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}, {"review_id": "rklzdrx-cS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an encoder-decoder as a differentiable loss function for sequential autoregressive generation tasks and more specifically for summarization. \nThis is done by adding a recorder network that that takes the decoded sequence from the summarizer as input and is trained to output the reference summary.\n\nI see a fundamental issue with this work:\n\n* During inference, authors decode from the probability distribution of the seq2seq model using beam search. \n* But for training (original seq2seq + recorder) authors backpropagate the NLL loss (which is fully differentiable) of the recorder on reference summaries through the softmax probabilities of outputs from the seq2seq model. \n\n>> This whole architecture can be seen as a traditional end-to-end seq2seq model with non-linearity and normalization (softmax) in the middle. \n\nAdditionally: \n>> \"backpropagating through the softmax weights during training and using the argmax during inference\" falls into a long line of work for propagating non-differential objective functions through continuous relaxations \n of categorical latent variables, more specifically the \"straight through\"  and \"gumbel-softmax\" (see refs.)\nThese methods have proven to be a strong alternative to reinforcement learning to train non-differential objectives and have been implemented quite a lot for sequence generation mainly for SeqGANs and even for text summarization connections to this line of work must be established in this paper. \n\nreferences \n- Estimating or propagating gradients through stochastic ´neurons for conditional computation. \nBengio et al. 2013 arXiv preprint arXiv:1308.3432, 2013.\n- CATEGORICAL REPARAMETERIZATION WITH GUMBEL-SOFTMAX\nJang et al. 2018 https://arxiv.org/pdf/1611.01144.pdf\n- GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution\nhttps://arxiv.org/pdf/1810.05739.pdf\n- Learning to Ask Questions in Open-domain Conversational Systems with Typed Decoders\nhttps://arxiv.org/pdf/1805.04843.pdf\n- MeanSum : A Neural Model for Unsupervised Multi-Document Abstractive Summarization\nhttps://arxiv.org/pdf/1810.05739.pdf\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "rkeIUEz6YB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an encoder-decoder-based summarization network as a loss function within a similar encoder-decoder-based summarization framework to demonstrate that the proposed model obtains better automatic and human evaluation scores compared to the baseline model of See et al. (2017) with just traditional loss functions. Overall, the paper is well-written and the presented results, analyses, and comparisons appear to be reasonable. One notable advantage of the proposed model would be to circumvent the approaches that rely on the evaluation metric, ROUGE as a reward component e.g. in a reinforcement learning setting, although with the expense of additional memory and time complexity.   \n\nFew comments:\n\n- \"A presents either a word ....\" --> this sentence is not clear.\n\n- \"Embedded representations ... differ somewhat from w_i\"--> Please clarify this aspect with more details.\n\n- In Figure 1, the proposed model with recoder seems to be suffering from issues related to redundancy and referential clarity, as it repeats the name \"malia\" several times. Would you comment on why this is the case?\n\n- It would be great if you could provide more details on the selection criteria/qualifications of the mechanical turk workers. Also, it is not clear why each example was given to only one worker and not to multiple workers. Wouldn't it be ideal to evaluate each example by multiple workers to get a sense of the inter-rater agreement? Please clarify. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an encoder-decoder based architecture to generate summaries. The real contribution of the paper is to use  a recoder matrix which takes the output from an existing encoder-decoder network and tries to generate the reference summary again. The output here is basically the softmax layer produced by the first encoder-decoder network which then goes through a feed-forward layer before being fed as embeddings into the recoder. So, since there is no discretization, the whole model can be trained jointly. (the original loss of the first encoder-decoder model is used as well anyway).

I agree with the reviewers here, that this whole model can in fact be viewed as a large encoder-decoder model, its not really clear where the improvements come from. Can you just increase the number of parameters of the original encoder-decoder model and see if it performs as good as the encoder-decoder + recoder? The paper also does not achieve SOTA on the task as there are other RL based papers which have been shown to perform better, so the choice of the recorder model is also not empirically justified. I recommend rejection of the paper in its current form.
Decision: Reject