Controlling Output Length in Neural Encoder-Decoders
Yuta Kikuchi1

Graham Neubig2∗

Ryohei Sasano1

kikuchi@lr.pi.titech.ac.jp

gneubig@cs.cmu.edu

sasano@pi.titech.ac.jp

Hiroya Takamura1

Manabu Okumura1

takamura@pi.titech.ac.jp

oku@pi.titecjh.ac.jp

1

Tokyo Institute of Technology, Japan
2
Carnegie Mellon University, USA

Abstract
Neural encoder-decoder models have shown
great success in many sequence generation
tasks. However, previous work has not investigated situations in which we would liketo control the length of encoder-decoder outputs. This capability is crucial for applications such as text summarization, in which
we have to generate concise summaries witha desired length. In this paper, we propose methods for controlling the output sequence length for neural encoder-decoder
models: two decoding-based methods and twolearning-based methods.1 Results show that
our learning-based methods have the capability to control length without degrading summary quality in a summarization task.

1

Introduction

Since its first use for machine translation (Kalchbrenner and Blunsom, 2013; Cho et al., 2014;
Sutskever et al., 2014), the encoder-decoder approach has demonstrated great success in manyother sequence generation tasks including image
caption generation (Vinyals et al., 2015b; Xu etal., 2015), parsing (Vinyals et al., 2015a), dialogue
response generation (Li et al., 2016a; Serban etal., 2016) and sentence summarization (Rush et al.,2015; Chopra et al., 2016). In particular, in this paper we focus on sentence summarization, which as
its name suggests, consists of generating shorter versions of sentences for applications such as document∗

This work was done when the author was at the Nara Institute of Science and Technology.
1
Available at https://github.com/kiyukuta/lencon.

summarization (Nenkova and McKeown, 2011) orheadline generation (Dorr et al., 2003). Recently,
Rush et al. (2015) automatically constructed large
training data for sentence summarization, and thishas led to the rapid development of neural sentencesummarization (NSS) or neural headline generation(NHG) models. There are already many studies that
address this task (Nallapati et al., 2016; Ayana et al.,2016; Ranzato et al., 2015; Lopyrev, 2015; Gulcehre
et al., 2016; Gu et al., 2016; Chopra et al., 2016).
One of the essential properties that text summarization systems should have is the ability to generate a summary with the desired length. Desired
lengths of summaries strongly depends on the sceneof use, such as the granularity of information theuser wants to understand, or the monitor size of thedevice the user has. The length also depends on theamount of information contained in the given source
document. Hence, in the traditional setting of text
summarization, both the source document and thedesired length of the summary will be given as input
to a summarization system. However, methods forcontrolling the output sequence length of encoderdecoder models have not been investigated yet, despite their importance in these settings.
In this paper, we propose and investigate four
methods for controlling the output sequence length
for neural encoder-decoder models. The former two
methods are decoding-based; they receive the desired length during the decoding process, and thetraining process is the same as standard encoderdecoder models.
The latter two methods arelearning-based; we modify the network architecture
to receive the desired length as input.

In experiments, we show that the learning-based
methods outperform the decoding-based methods
for long (such as 50 or 75 byte) summaries. We
also find that despite this additional length-control
capability, the proposed methods remain competitive to existing methods on standard settings of the
DUC2004 shared task-1.

2

Background

2.1 Related Work
Text summarization is one of the oldest fields ofstudy in natural language processing, and manysummarization methods have focused specificallyon sentence compression or headline generation.
Traditional approaches to this task focus on word
deletion using rule-based (Dorr et al., 2003; Zajic
et al., 2004) or statistical (Woodsend et al., 2010;
Galanis and Androutsopoulos, 2010; Filippova andStrube, 2008; Filippova and Altun, 2013; Filippova et al., 2015) methods. There are also several
studies of abstractive sentence summarization using syntactic transduction (Cohn and Lapata, 2008;
Napoles et al., 2011) or taking a phrase-based statistical machine translation approach (Banko et al.,
2000; Wubben et al., 2012; Cohn and Lapata, 2013).
Recent work has adopted techniques such as
encoder-decoder (Kalchbrenner and Blunsom, 2013;
Sutskever et al., 2014; Cho et al., 2014) and attentional (Bahdanau et al., 2015; Luong et al., 2015)
neural network models from the field of machinetranslation, and tailored them to the sentence summarization task. Rush et al. (2015) were the firstto pose sentence summarization as a new target task
for neural sequence-to-sequence learning. Several
studies have used this task as one of the benchmarks of their neural sequence transduction methods (Ranzato et al., 2015; Lopyrev, 2015; Ayana
et al., 2016). Some studies address the other important phenomena frequently occurred in humanwritten summaries, such as copying from the source
document (Gu et al., 2016; Gulcehre et al., 2016).
Nallapati et al. (2016) investigate a way to solve
many important problems capturing keywords, or
inputting multiple sentences.
Neural encoder-decoders can also be viewed asstatistical language models conditioned on the target sentence context. Rosenfeld et al. (2001) haveproposed whole-sentence language models that canconsider features such as sentence length. However,as described in the introduction, to our knowledge,
explicitly controlling length of output sequences inneural language models or encoder-decoders has notbeen investigated.
Finally, there are some studies to modify the output sequence according some meta information suchas the dialogue act (Wen et al., 2015), user personality (Li et al., 2016b), or politeness (Sennrich et al.,2016). However, these studies have not focused on
length, the topic of this paper.

2.2 Importance of Controlling Output Length
As we already mentioned in Section 1, the moststandard setting in text summarization is to input
both the source document and the desired length of
the summary to a summarization system. Summarization systems thus must be able to generate summaries of various lengths. Obviously, this property
is also essential for summarization methods basedon neural encoder-decoder models.
Since an encoder-decoder model is a completely
data-driven approach, the output sequence length
depends on the training data that the model is trainedon. For example, we use sentence-summary pairs
extracted from the Annotated English Gigaword corpus as training data (Rush et al., 2015), and theaverage length of human-written summary is 51.38
bytes. Figure 1 shows the statistics of the corpus.
When we train a standard encoder-decoder modeland perform the standard beam search decoding on
the corpus, the average length of its output sequenceis 38.02 byte.
However, there are other situations where wewant summaries with other lengths. For example, DUC2004 is a shared task where the maximum
length of summaries is set to 75 bytes, and summarization systems would benefit from generating sentences up to this length limit.
While recent NSS models themselves cannot control their output length, Rush et al. (2015) and others
following use an ad-hoc method, in which the system is inhibited from generating the end-of-sentence(EOS) tag by assigning a score of −∞ to the tag and4000014000035000

120000180000160000140000

30000

100000120000

25000
80000

100000

60000

80000

20000
15000

60000

40000

10000

40000

5000

20000

0

0

0

100

200

300

400

500

600

700

20000

0

50

100

150

200

250

0

300

0

0.5

1

1.5

2

2.5

3

3.5

(a) first sentence (181.87)(b) article headline (51.38)(c) ratio (0.30)
Figure 1: Histograms of first sentence length, headline length, and their ratio in Annotated Gigaword English Gigaword corpus. Bracketed values in each subcaption are averages.
yt

s̃t
mt
st



yt 1

at
Attenderctht

yt
a given source sentence,the summarizer generates
a shortened version
oftheinput (i.e. N > M ),s̃t
as summary sentence y = m(yt1 , y2 , y3 , ..., yM ). The
bc
model estimates conditionals probability p(y|x) ust
ing
parameters trainedon large training data consistatyt 1
ing of sentence-summary
pairs. Typically, this conlength
ditional probability is factorized as the product ofconditional probabilities of the next word in the sec t quence:ct
ht

Figure 2: The encoder-decoder architecture we used as abase model in this paper.

generating a fixed number of words2 , and finally theoutput summaries are truncated to 75 bytes. Ideally,
the models should be able to change the output sequence depending on the given output length, and to
output the EOS tag at the appropriate time point in anatural manner.

3

Network Architecture: Encoder-Decoderwith Attention

In this section, we describe the model architecture used for our experiments: an encoder-decoder
consisting of bi-directional RNNs and an attention
mechanism. Figure 2 shows the architecture of themodel.
Suppose that the source sentence is represented asa sequence of words x = (x1 , x2 , x3 , ..., xN ). For
2
Accordingtothepublishedcode(https://github.com/facebook/NAMAS), the default number of words is set to 15, which is too long for the DUC2004
setting. The average number of words of human summaries inthe evaluation set is 10.43.

Attender

ht

ct
ht

p(y|x) =M∏p(yt |y<t , x),t=1

where y<t = (y1 , y2 , y3 , ..., yt−1 ). In the following,
we describe how to compute p(yt |y<t , x).
3.1 Encoder
We use the bi-directional RNN (BiRNN) as encoder which has been shown effective in neural machine translation (Bahdanau et al., 2015) and speech
recognition (Schuster and Paliwal, 1997; Graves etal., 2013).
A BiRNN processes the source sentence forboth forward and backward directions with two
separate RNNs. During the encoding process,the BiRNN computes both forward hidden states
−
→ −
→
−
→
( h 1 , h 2 , ..., h N ) and backward hidden states
←
− ←
−
←
−
( h 1 , h 2 , ..., h N ) as follows:−→−→
h t = g( h t−1 , xt ),←−←−h t = g( h t+1 , xt ).
While g can be any kind of recurrent unit, we use
long short-term memory (LSTM) (Hochreiter andSchmidhuber, 1997) networks that have memory→−
cells for both directions (−c t and ←c t ).

After encoding, we set the initial hidden states s0
and memory-cell m0 of the decoder as follows:←−s0 = h 1 ,
−
m = ←
c .
0

1

3.2 Decoder and Attender
Our decoder is based on an RNN with LSTM g:
st = g(st−1 , xt ).
We also use the attention mechanism developedby Luong et al. (2015), which uses st to computecontextual information dt of time step t. We firstsummarize the forward and backward encoder states−→←−
by taking their sum h̄i = h i + h i , and then calculate the context vector dt as the weighted sum ofthese summarized vectors:∑
dt =ati h̄i ,iwhere at is the weight at the t-th step for h̄i computed by a softmax operation:ati =exp(st · h̄i )∑.
′h̄′ exp(st · h̄ )After context vector dt is calculated, the model
updates the distribution over the next word as follows:s̃t = tanh(Whs [st ; dt ] + bhs ),p(yt |y<t , x) = softmax(Wso s˜t + bso ).
Note that s̃t is also provided as input to the LSTM
with yt for the next step, which is called the input
feeding architecture (Luong et al., 2015).
3.3 Training and Decoding
The training objective of our models is to maximize
log likelihood of the sentence-summary pairs in agiven training set D:∑
Lt (θ) =log p(y|x; θ),(x,y)∈Dp(y|x; θ) =∏p(yt |y<t , x).

t

Once models are trained, we use beam search to findthe output that maximizes the conditional probability.

4 Controlling Length in Encoder-decoders
In this section, we propose our four methods thatcan control the length of the output in the encoderdecoder framework. In the first two methods, thedecoding process is used to control the output length
without changing the model itself. In the other two
methods, the model itself has been changed and istrained to obtain the capability of controlling thelength. Following the evaluation dataset used in our
experiments, we use bytes as the unit of length, although our models can use either words or bytes asnecessary.
4.1

f ixLen: Beam Search without EOS Tags

The first method we examine is a decoding approachsimilar to the one taken in many recent NSS methods that is slightly less ad-hoc. In this method, we
inhibit the decoder from generating the EOS tag byassigning it a score of −∞. Since the model cannot stop the decoding process by itself, we simply
stop the decoding process when the length of output
sequence reaches the desired length. More specifically, during beam search, when the length of the sequence generated so far exceeds the desired length,
the last word is replaced with the EOS tag and also
the score of the last word is replaced with the scoreof the EOS tag (EOS replacement).
4.2

f ixRng: Discarding Out-of-range
Sequences

Our second decoding method is based on discarding
out-of-range sequences, and is not inhibited fromgenerating the EOS tag, allowing it to decide whento stop generation. Instead, we define the legitimate
range of the sequence by setting minimum and maximum lengths. Specifically, in addition to the normal
beam search procedure, we set two rules:• If the model generates the EOS tag when the
output sequence is shorter than the minimum
length, we discard the sequence from the beam.
• If the generated sequence exceeds the maximum length, we also discard the sequence fromthe beam. We then replace its last word with
the EOS tag and add this sequence to the beam(EOS replacement in Section 4.1).3
mt

mt 1

In other words, we keep only the sequences that
contain the EOS tag and are in the defined length
range. This method is a compromise that allows
the model some flexibility to plan the generated sequences, but only within a certain acceptable length
range.
It should be noted that this method needs a larger
beam size if the desired length is very different fromthe average summary length in the training data, as
it will need to preserve hypotheses that haveyt the desired length.
s̃

st

st 1

e1 (xt )e2 (lt )xt

lt

Figure 3: LenEmb: remaining length is used as additional input for the LSTM of the decoder.
yt

s̃t

t

4.3

mt

LenEmb: Length Embedding as
Additional Input for the LSTM



mt

bc

st

st



at

yt 1
Our third method is a learning-based
method specifically trained to control the length of the output sequence. Inspired by previous work that has demonstrated that additionalc tinputs to decoder models can
h t characteristics of the output
effectively control thect(Wen et al., 2015; Lihetal., 2016b), this model protvides information about the length in the form of anadditional input to the net. Specifically, the model
uses an embedding e2 (lt ) ∈ RD for each potential
desired length, which is parameterized by a length
embedding matrix Wle ∈ RD×L where L is thenumber of length types. In the decoding process, we
input the embedding of the remaining length lt asadditional input to the LSTM (Figure 3). lt is initialized after the encoding process and updated during
the decoding process as follows:
Attenderl1 = length,{0(lt − byte(yt ) ≤ 0)
lt+1 =
lt − byte(yt ) (otherwise),
where byte(yt ) is the length of output word yt andlength is the desired length. We learn the valuesof the length embedding matrix Wle during training. This method provides additional information
about the amount of length remaining in the output
sequence, allowing the decoder to “plan” its output
based on the remaining number of words it can generate.
3

i t jt o tftThis is a workaround to prevent the situation in which allsequences are discarded from a beam.

yt 1

at
Attenderlength

ct
ht
ct
ht

Figure 4: LenInit: initial state of the decoder’s memory
cell m0 manages output length.

LenInit: Length-based Memory Cell
Initialization
While LenEmb inputs the remaining length lt to thedecoder at each step of the decoding process, theLenInit method inputs the desired length once at
the initial state of the decoder. Figure 4 shows the architecture of LenInit. Specifically, the model uses
the memory cell mt to control the output length byinitializing the states of decoder (hidden state s0 andmemory cell m0 ) as follows:
4.4

s0
m0

=
=

←
−
h 1,
bc ∗ length,(1)

where bc ∈ RH is a trainable parameter and length
is the desired length.
While the model of LenEmb is guided towards
the appropriate output length by inputting the remaining length at each step, this LenInit attempts
to provide the model with the ability to manage theoutput length on its own using its inner state. Specifically, the memory cell of LSTM networks is suitable for this endeavour, as it is possible for LSTMs

10

300

140

9

120

250
8
7

100

200

80

6
150
5

60

4

100

40

3
50

20

2
1

50

100

150

200

250

300

350

400

450

500

(a) first sentence (206.91)

550

0

35

40

45

50

55

60

65

70

75

80

0

0

(b) summary (70.00)

0.2

0.4

0.6

0.8

1

1.2

(c) ratio (0.35)

Figure 5: Histograms of first sentence length, summary length, and their ratio in DUC2004.

to learn functions that, for example, subtract a fixedamount from a particular memory cell every timethey output a word. Although other ways for managing the length are also possible,4 we found thisapproach to be both simple and effective.

5

Experiment

5.1 Dataset
We trained our models on a part of the AnnotatedEnglish Gigaword corpus (Napoles et al., 2012),
which Rush et al. (2015) constructed for sentencesummarization. We perform preprocessing using thestandard script for the dataset5 . The dataset consists of approximately 3.6 million pairs of the first
sentence from each source document and its headline. Figure 1 shows the length histograms of thesummaries in the training set. The vocabulary sizeis 116,875 for the source documents and 67,564
for the target summaries including the beginning-ofsentence, end-of-sentence, and unknown word tags.
For LenEmb and LenInit, we input the length ofeach headline during training. Note that we do nottrain multiple summarization models for each headline length, but a single model that is capable of controlling the length of its output.
We evaluate the methods on the evaluation set
of DUC2004 task-1 (generating very short singledocument summaries). In this task, summarization
systems are required to create a very short summary for each given document. Summaries overthe length limit (75 bytes) will be truncated andthere is no bonus for creating a shorter summary.
The evaluation set consists of 500 source documents
and 4 human-written (reference) summaries for each4

For example, we can also add another memory cell for
managing the length.
5
https://github.com/facebook/NAMAS

source document. Figure 5 shows the length histograms of the summaries in the evaluation set. Notethat the human-written summaries are not always aslong as 75 bytes. We used three variants of ROUGE(Lin, 2004) as evaluation metrics: ROUGE-1 (unigram), ROUGE-2 (bigram), and ROUGE-L (longest
common subsequence). The two-sided permutation
test (Chinchor, 1992) was used for statistical significance testing (p ≤ 0.05).
5.2 Implementation
We use Adam (Kingma and Ba, 2015) (α=0.001,
β1 =0.9, β2 =0.999, eps=10−8 ) to optimize parameters with a mini-batch of size 80. Before every
10,000 updates, we first sampled 800,000 training
examples and made groups of 80 examples withthe same source sentence length, and shuffled the
10,000 groups.
We set the dimension of word embeddings to 100
and that of the hidden state to 200. For LSTMs,
we initialize the bias of the forget gate to 1.0 anduse 0.0 for the other gate biases (Józefowicz et al.,2015). We use Chainer (Tokui et al., 2015) to implement our models. For LenEmb, we set L to 300,
which is larger than the longest summary lengths inour dataset (see Figure 1-(b) and Figure 5-(b)).
For all methods except f ixRng, we found a beam
size of 10 to be sufficient, but for f ixRng we useda beam size of 30 because it more aggressively discards candidate sequences from its beams during decoding.

6 Result
6.1 ROUGE Evaluation
Table 1 shows the ROUGE scores of each methodwith various length limits (30, 50 and 75 byte). Regardless of the length limit set for the summariza-

modelf ixLenf ixRng
LenEmb(0,L)
LenInit(0,L)
LenEmb(0,∞)
LenInit(0,∞)

R-1
14.34
13.83∗
14.23
14.31
13.75
13.92

30 byte
R-2
3.10∗
3.08∗
3.21
3.27
3.30
3.49

R-L
13.23
12.88
13.02
13.19
12.68
12.90

R-1
20.00∗
20.08∗
20.78
20.87
20.62
20.87

50 byte
R-2
R-L
5.98 18.26∗
5.74 18.19∗
5.97
18.57
6.16
19.00
6.22
18.64
6.19
19.09

R-1
25.87∗
26.01
26.73
25.87
26.42
25.29∗

75 byte
R-2
R-L
7.93
23.07∗∗
7.69
22.77∗
8.39
23.88
8.27
23.24
8.26
23.59
8.00
22.71∗

Table 1: ROUGE scores with various length limits. The scores with ∗ are significantly worse than the best score in
the column (bolded).
sourcereferencef ixLen (30)
(50)
(75)
f ixRng (30)
(50)
(75)
LenEmb (30)
(50)
(75)
LenInit (30)
(50)
(75)

five-time world champion michelle kwan withdrew from the #### us figure skating championships
on wednesday , but will petition us skating officials for the chance to compete at the #### turin
olympics .
injury leaves kwan ’s olympic hopes in limbo
kwan withdraws from us gp
kwan withdraws from us skating championships
kwan pulls out of us figure skating championships for turin olympics
kwan withdraws from us gp
kwan withdraws from figure skating championships
kwan pulls out of us figure skating championships for turin olympics bid
kwan withdraws from us skating
kwan withdraws from us figure skating championships
world champion kwan withdraws from #### olympic figure skating championships
kwan quits us figure skating
kwan withdraws from #### us figure skating worlds
kwan withdraws from #### us figure skating championships for #### olympics

Table 2: Examples of the output of each method with various specified lengths.

tion methods, we use the same reference summaries.
Note that, f ixLen and f ixRng generate the summaries with a hard constraint due to their decoding process, which allows them to follow the hardconstraint on length. Hence, when we calculate thescores of LenEmb and LenInit, we impose a hardconstraint on length to make the comparison fair(i.e. LenEmb(0,L) and LenInit(0,L) in the table).
Specifically, we use the same beam search as thatfor f ixRng with minimum length of 0.
For the purpose of showing the length control
capability of LenEmb and LenInit, we show atthe bottom two lines the results of the standard
beam search without the hard constraints on thelength6 . We will use the results of LenEmb(0,∞)
and LenInit(0,∞) in the discussions in Sections 6.2
and 6.3.
The results show that the learning-based meth6

f ixRng is equivalence to the standard beam search when
we set the range as (0, ∞).

ods (LenEmb and LenInit) tend to outperform
decoding-based methods (f ixLen and f ixRng) forthe longer summaries of 50 and 75 bytes. However, in the 30-byte setting, there is no significant
difference between these two types of methods. We
hypothesize that this is because average compression rate in the training data is 30% (Figure 1-(c))
while the 30-byte setting forces the model to generate summaries with 15.38% in average compression rate, and thus the learning-based models did nothave enough training data to learn compression atsuch a steep rate.
6.2 Examples of Generated Summaries
Tables 2 and 3 show examples from the validation
set of the Annotated Gigaword Corpus. The tables show that all models, including both learningbased methods and decoding-based methods, can often generate well-formed sentences.
We can see various paraphrases of “#### us figuresourcereferencef ixLen (30)
(50)
(75)
f ixRng (30)
(50)
(75)
LenEmb (30)
(50)
(75)
LenInit (30)
(50)
(75)

at least two people have tested positive for the bird flu virus in eastern turkey , health minister
recep akdag told a news conference wednesday .
two test positive for bird flu virus in turkey
two infected with bird flu
two infected with bird flu in eastern turkey
two people tested positive for bird flu in eastern turkey says minister
two infected with bird flu
two more infected with bird flu in eastern turkey
two people tested positive for bird flu in eastern turkey says minister
two bird flu cases in turkey
two confirmed positive for bird flu in eastern turkey
at least two bird flu patients test positive for bird flu in eastern turkey
two cases of bird flu in turkey
two people tested positive for bird flu in turkey
two people tested positive for bird flu in eastern turkey health conference

Table 3: More examples of the output of each method.

championships”7 and “withdrew”. Some examples
are generated as a single noun phrase (LenEmb(30)and LenInit(30)) which may be suitable for the
short length setting.
6.3 Length Control Capability of
Learning-based Models
Figure 6 shows histograms of output length from thestandard encoder-decoder, LenEmb, and LenInit.
While the output lengths from the standard model
disperse widely, the lengths from our learning-basedmodels are concentrated to the desired length. These
histograms clearly show the length controlling capability of our learning-based models.
Table 4-(a) shows the final state of the beam when
LenInit generates the sentence with a length of 30
bytes for the example with standard beam search in
Table 3. We can see all the sentences in the beam
are generated with length close to the desired length.
This shows that our method has obtained the ability
to control the output length as expected. For comparison, Table 4-(b) shows the final state of the beam
if we perform standard beam search in the standard encoder-decoder model (used in f ixLen andf ixRng). Although each sentence is well-formed,
the lengths of them are much more varied.

task-1. Although the objective of this paper is not to
obtain state-of-the-art scores on this evaluation set, it
is of interest whether our length-controllable models
are competitive on this task. Table 5 shows that thescores of our methods, which are copied from Table
1, in addition to the scores of some existing methods.
ABS (Rush et al., 2015) is the most standard modelof neural sentence summarization and is the mostsimilar method to our baseline setting (f ixLen).
This table shows that the score of f ixLen is comparable to those of the existing methods. The tablealso shows the LenEmb and the LenInit have thecapability of controlling the length without decreasing the ROUGE score.

7 Conclusion

6.4 Comparison with Existing Methods

In this paper, we presented the first examination of
the problem of controlling length in neural encoderdecoder models, from the point of view of summarization. We examined methods for controlling
length of output sequences: two decoding-based
methods (f ixLen and f ixRng) and two learningbased methods (LenEmb and LenInit). The results showed that learning-based methods generally
outperform the decoding-based methods, and thelearning-based methods obtained the capability ofcontrolling the output length without losing ROUGE
score compared to existing summarization methods.

Finally, we compare our methods to existing methods on standard settings of the DUC2004 shared

Acknowledgments

7

Note that “#” is a normalized number and “us” is “US”
(United States).

This work was supported by JSPS KAKENHI Grant
Number JP26280080. We are grateful to have thelogp(y|x)
-4.27
-4.41
-4.65
-5.25
-5.27
-5.51
-5.55
-5.72
-6.04

byte
31
28
30
30
31
29
32
30
30

candidate summary
two cases of bird flu in turkey
two bird flu cases in turkey
two people tested for bird flu
two people tested in e. turkey
two bird flu cases in e. turkey
two bird flu cases in eastern
two people tested in east turkey
two bird flu cases in turkey :two people fail bird flu virus

logp(y|x)
-5.05
-5.13
-5.30
-5.49
-5.52
-5.55
-6.00
-6.04
-6.50

(a) the beam of LenInit

byte
57
50
39
51
32
44
49
54
49

candidate summary
two people tested positive for bird flu in eastern turkey
two tested positive for bird flu in eastern turkey
two people tested positive for bird flu
two people infected with bird flu in eastern turkey
two tested positive for bird flu
two infected with bird flu in eastern turkey
two more infected with bird flu in eastern turkey
two more confirmed cases of bird flu in eastern turkey
two people tested positive for bird flu in turkey(b) the beam of the standard encoder-decoder

Table 4: Final state of the beam when the learning-based model is instructed to output a 30 byte summary for thesource document in Table 3.




(QF'HF



(QF'HF/HQ(PE
/HQ(PE
/HQ(PE





































OHQJWK





(QF'HF/HQ,QLW
/HQ,QLW
/HQ,QLW



IUHTXHQF\

IUHTXHQF\

IUHTXHQF\







OHQJWK















OHQJWK







(a) encoder-decoder(b) LenEmb(c) LenInit
Figure 6: Histograms of output lengths generated by (a) the standard encoder-decoder , (b) LenEmb, and (c)LenInit. For LenEmb and LenInit, the bracketed numbers in each region are the desired lengths we
set.
model
f ixLenf ixRng
LenEmbLenInit
ABS(Rush et al., 2015)
ABS+(Rush et al., 2015)
RAS-Elman(Chopra et al., 2016)
RAS-LSTM(Chopra et al., 2016)

R-1
25.88
26.02
26.73
25.87
26.55
28.18
28.97
27.41

R-2
7.93
7.69
8.40
8.28
7.06
8.49
8.26
7.69

R-L
23.07
22.78
23.88
23.25
22.05
23.81
24.06
23.06

Table 5: Comparison with existing studies for
DUC2004.
Note that top four rows are
reproduced from Table 1.

opportunity to use the Kurisu server of Dwango Co.,
Ltd. for our experiments.

References
Ayana, S. Shen, Z. Liu, and M. Sun. 2016. Neural Headline Generation with Minimum Risk Training. CoRR,
abs/1604.01904.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-

gio. 2015. Neural machine translation by jointly
learning to align and translate. In Proceedings of
ICLR15.
Michele Banko, Vibhu O. Mittal, and Michael J. Witbrock. 2000. Headline generation based on statisticaltranslation. In Proceedings of ACL00, pages 318–325.
Nancy Chinchor. 1992. The statistical significance ofthe muc-4 results. In Proceedings MUC4 ’92, pages
30–50.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre,
Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk,
and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the EMNLP14,
pages 1724–1734.
Sumit Chopra, Michael Auli, and Alexander M. Rush.
2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of
NAACL-HLT16, pages 93–98.
Trevor Cohn and Mirella Lapata. 2008. Sentence compression beyond word deletion. In Proceedings of
COLING08, pages 137–144.
Trevor Cohn and Mirella Lapata. 2013. An abstrac-

tive approach to sentence compression. ACM TIST13,
4(3):41:1–41:35, July.
Bonnie Dorr, David Zajic, and Richard Schwartz. 2003.
Hedge trimmer: A parse-and-trim approach to headline generation. In Proceedings of the HLT-NAACL 03
Text Summarization Workshop, pages 1–8.
Katja Filippova and Yasemin Altun. 2013. Overcoming
the lack of parallel data in sentence compression. In
Proceedings of EMNLP13, pages 1481–1491.
Katja Filippova and Michael Strube. 2008. Dependency
tree based sentence compression. In Proceedings of
INLG08, pages 25–32.
Katja Filippova, Enrique Alfonseca, Carlos A. Colmenares, Lukasz Kaiser, and Oriol Vinyals. 2015.
Sentence compression by deletion with lstms. In Proceedings of EMNLP15, pages 360–368.
Dimitrios Galanis and Ion Androutsopoulos. 2010.
An extractive supervised two-stage method for sentence compression. In Proceedings of NAACL-HLT10,
pages 885–893.
A. Graves, N. Jaitly, and A. r. Mohamed. 2013. Hybrid speech recognition with deep bidirectional lstm.
In Proceedings of IEEE Workshop on ASRU13, pages
273–278.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li.
2016. Incorporating copying mechanism in sequenceto-sequence learning. In Proceedings of ACL16, pages
1631–1640.
Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati,
Bowen Zhou, and Yoshua Bengio. 2016. Pointing
the unknown words. In Proceedings of ACL16, pages
140–149.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long
short-term memory. Neural Computation, 9(8):1735–
1780.
Rafal Józefowicz, Wojciech Zaremba, and Ilya Sutskever.
2015. An empirical exploration of recurrent network architectures. In Proceedings of ICML15, pages
2342–2350.
Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedingsof EMNLP13, pages 1700–1709, Seattle, Washington,
USA, October. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A
method for stochastic optimization. In Proceedings of
ICLR15.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao,and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In Proceedings of NAACL-HLT16, pages 110–119.
Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016b. A

persona-based neural conversation model. In Proceedings of ACL16, pages 994–1003.
Chin-Yew Lin. 2004. Rouge: A package for automatic
evaluation of summaries. In Proceedings of the ACL04
Workshop, pages 74–81.
Konstantin Lopyrev. 2015. Generating news headlines with recurrent neural networks.
CoRR,
abs/1512.01712.
Thang Luong, Hieu Pham, and Christopher D. Manning.
2015. Effective approaches to attention-based neuralmachine translation. In Proceedings of EMNLP15,
pages 1412–1421.
Ramesh Nallapati, Bing Xiang, and Bowen Zhou. 2016.
Sequence-to-sequence rnns for text summarization.
CoRR, abs/1602.06023.
Courtney Napoles, Chris Callison-Burch, Juri Ganitkevitch, and Benjamin Van Durme. 2011. Paraphrastic sentence compression with a character-based metric: Tightening without deletion. In Proceedings ofthe Workshop on Monolingual Text-To-Text Generation, pages 84–90.
Courtney Napoles, Matthew Gormley, and Benjamin
Van Durme. 2012. Annotated gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge
Base Construction and Web-scale Knowledge Extraction, pages 95–100.
Ani Nenkova and Kathleen McKeown. 2011. Automatic
R in Inforsummarization. In Foundations and Trends⃝
mation Retrieval, volume 2-3, pages 103–233.
Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli,
and Wojciech Zaremba. 2015. Sequence leveltraining with recurrent neural networks.
CoRR,
abs/1511.06732.
Ronald Rosenfeld, Stanley F. Chen, and Xiaojin Zhu.
2001. Whole-sentence exponential language models:a vehicle for linguistic-statistical integration. Computer Speech & Language, 15(1):55–73.
Alexander M. Rush, Sumit Chopra, and Jason Weston.
2015. A neural attention model for abstractive sentence summarization. In Proceedings of EMNLP15,
pages 379–389.
M. Schuster and K.K. Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal
Processing, 45(11):2673–2681.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016. Controlling politeness in neural machine translation via side constraints. In Proceedings of NAACLHLT16, pages 35–40.
Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio,
Aaron C. Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of
AAAI16, pages 3776–3784.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In
Proceedings of NIPS14, pages 3104–3112.
Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. 2015. Chainer: a next-generation open source
framework for deep learning. In Proceedings of
NIPS15 Workshop on LearningSys.
Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov,
Ilya Sutskever, and Geoffrey E. Hinton. 2015a. Grammar as a foreign language. In Proceedings of NIPS15,
pages 2773–2781.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015b. Show and tell: A neural image
caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
pages 3156–3164.
Tsung-Hsien Wen, Milica Gasic, Nikola Mrkšić, PeiHao Su, David Vandyke, and Steve Young. 2015.
Semantically conditioned lstm-based natural language
generation for spoken dialogue systems. In Proceedings of EMNLP15, pages 1711–1721, Lisbon, Portugal, September. Association for Computational Linguistics.
Kristian Woodsend, Yansong Feng, and Mirella Lapata.
2010. Title generation with quasi-synchronous grammar. In Proceedings of the EMNLP10, pages 513–523.
Sander Wubben, Antal van den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingualmachine translation. In Proceedings of ACL12, pages
1015–1024.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho,
Aaron Courville, Ruslan Salakhudinov, Rich Zemel,
and Yoshua Bengio. 2015. Show, attend and tell:
Neural image caption generation with visual attention.
In David Blei and Francis Bach, editors, Proceedingsof ICML15, pages 2048–2057. JMLR Workshop andConference Proceedings.
David Zajic, Bonnie J Dorr, and R. Schwartz. 2004.
Bbn/umd at duc-2004: Topiary. In Proceedings of
NAACL-HLT04 Document Understanding Workshop,
pages 112 – 119.