DEIM Forum 2019 B4-2

Two-Encoder Pointer-Generator Network for Summarizing Segments of Long
ArticlesJunhao LI† and Mizuho IWAIHARA‡
Graduate School of Information, Production and Systems
Waseda University, Kitakyushu, Japan
E-mail: †lijunhao@toki.waseda.jp, ‡iwaihara@waseda.jp
Abstract Usually long documents contain many sections and segments. In Wikipedia, one article can usually be divided
into sections and one section can be divide into segments. But although one article is already divided into smaller segments,
one segment is often still too long to read. So, we consider that segments should have a short summary for readers to grasp aquick view of the segment. This paper discusses applying neural summarization models including seq2seq model and pointergenerator network model to segment summarization. These models for summarization can take target articles as the only input
to the model. However, in our case, it is very likely that other segments in the same article contain descriptions related to thetarget segment. Therefore, we propose several ways to add additional information from the whole article to the short target
segment, as the input for summarization. We compare the results against the original models without additional information.
Furthermore, we propose a new model that uses two encoders to process target segments and additional sequences separately.
Our results show the two-encoder model outperforms the original models in terms of ROGUE and METEOR scores.
Keyword text summarization, deep learning, seq2seq, pointer generator network, multi-encoder
article may lose too much information from target

1. INTRODUCTION
Wikipedia, started from 2001, has been emerging as thesegments or sections. So we think it is necessary to focus

world’s largest encyclopedia. The English version of

on segment summarization.

Wikipedia contains 5.7 million articles and over 600

There are two broad approaches to summarization:

articles are newly created in one day as of August 2018.

Extractive and abstractive.

Wikipedia

Extractive methods involve the selection of the phrases

Foundation, an international non -profit organization, andand sentences from the source document to make up new

a great number of collaborators in the world under somesummary. Techniques involve ranking the relevance ofinternational projects, its articles are continuously editedsentences or phrases in order to choose only those mostand developed. Therefore, its content is quite reliable

relevant to the meaning of the source. One widely usedregardless of its openness [5].

example is TextRank [17] that uses graph-based ranking

Since

the

encyclopedia

ismanagedbyWith the number of Wikipedia articles growing very
quickly, the information that Wikipedia contains isinvaluable. But generally, the usage of Wikipedia isalgorithm to extract important sentences from a single
document.
Abstractive

methods

attempttodevelopanlimited to mainly for human readers. The most importantunderstanding of the main concepts in a document and

work

using

then express those concepts in clear natural language. It

Wikipedia resource is searching and showing interested

uses linguistic methods to examine and interpret the text

articles to readers [13]. With the continuously growing

and then to find the new concepts and expressions to best

number of articles in Wikipedia, the search results might

describe it by generating a new shorter text that conveys

be very long. In additional, segments in articles sometimes

the most important information from the original text

become longer, which is making it difficult for human

document [19]. Recently deep learning methods havereaders to grasp the whole idea from one specific segment.

shown promising results for text summarization. The most

Summarizing the whole article may be one solution for

popular model is sequence to sequence attention model

this problem. But the summary of article contains

[9].

that

computersystems

c ansupportin

information from the whole article, which woul d make it

In this paper, we mainly focus on summarization

not suitable for the reader who only looks for information

methods

in several segments or sections. Also, the summary of one

contains both approaches. Since we need to add additional

that

involves

deep

learning

models whichinformation from the whole article and sometimes themulti-task Seq2Seq model to solve different NLP problems

whole article is too long containing too much noisy

using the same model in the same time. Their model

information. So we are using extractive methods tosettings include one encoder to many decoders, many

shorten the article into several sentences as additional

encoders to one decoder and many encoders to many

information to the model. The rest of this paper isdecoders. Their results show that it is a promising modelorganized as follows. Section 2 covers related work.

for solving multi-task problems. Several results that use

Section 3 is models we used. Section 4 shows threemultiple encoders are even better than only one encoder.

additional sequence methods. Section 5 is the results of

Vinyals et al. [15] proposed a Seq2Seq model called

our experiments and section 6 make discussions about it.

pointer network which produces an output sequence

Section 6 concludes this paper.

consisting of elements from the input sequence applying
soft attention distribution of Bahdanau et al. [4]. The
pointer network has been utilized to create hybrid

2. RELATED WORKLong short-term memory (LSTM) was first proposed by
Seep

HochreiterandJürgenSchmidhuber

[18]

and

approaches

for

NMT,

languagemodeling

andsummarization.

improved in 2002 by Felix Gers’ team [6]. LSTM networks

Abigail et al. [2] proposed a pointer-generator network

are well-suited to classifying, processing and making

model that combines pointer network with original

predictions based on time series data. LSTMs were

Seq2Seq + attention model. Their model allows bothdeveloped to deal with the exploding and vanishingcopying words through pointing and generating words

gradient problems that can be encountered when training

from a fixed vocabulary. In their model, they are using

traditional RNNs. Later Graves et al. [7] proposed a neural

Bahdanau attention mechanism [4] in equations (1) (2) andnetwork model using bidirectional LSTM (Bi -LSTM) for(3) to calculate attention distribution 𝑎𝑡 and context

phoneme classification, showing great improvement overvector ℎ𝑡∗ . Their generation probability 𝑝𝑔𝑒𝑛 ∈ [0,1] fororiginal LSTM.

timestep t is calculated from context vector, decoder state

Neural network-based Seq2Seq learning has achieveds𝑡 and the decoder input 𝑥𝑡 .
𝑝𝑔𝑒𝑛 = σ(𝑤ℎ𝑇∗ ℎ𝑡∗ + 𝑤𝑠𝑇 𝑠𝑡 + 𝑤𝑥𝑇 𝑥𝑡 + 𝑏𝑝𝑡𝑟 )remarkable success in various NLP tasks, including but(4)

not limited to machine translation and text summarization.

Where 𝑤ℎ𝑇∗ , 𝑤𝑠𝑇 , 𝑤𝑥𝑇 and 𝑏𝑝𝑡𝑟 are learnable parameters

The Seq2Seq model is firstly proposed by Ilya et al. [2].

and σ is the sigmoid function. Then the 𝑝𝑔𝑒𝑛 is used as a

The Seq2Seq model is an encoder-decoder model such thatsoft switch to choose between generating a word from thethe encoder converts the input sequence into one context

vocabulary by sampling from 𝑝𝑣𝑜𝑐𝑎𝑏 or copying a word

vector that contains all information in the input sequence.

from the input sequence by sampling from the attention

Then the decoder generates the output sequence based on

distribution 𝑎𝑡 .

Bahdanau et al. [4] were the first to introduce attention

𝑃𝑣𝑜𝑐𝑎𝑏 = softmax(𝑉 ′ (𝑉[𝑠𝑡 , ℎ𝑡∗ ] + 𝑏) + 𝑏 ′ )(5)𝑃(𝑤) = 𝑝𝑔𝑒𝑛 𝑃𝑣𝑜𝑐𝑎𝑏 (𝑤) + (1 − 𝑝𝑔𝑒𝑛 ) ∑𝑖:𝑤𝑖 =𝑤 𝑎𝑖𝑡 (6)mechanism to the seq2seq model to release the burden of

Note that if w is an out-of-vocabulary (OOV) word, thencompressing of entire source into a fixed-length vector as
context. Instead, they proposed to use a dynamically

𝑝𝑣𝑜𝑐𝑎𝑏 (𝑤) is zero; similarly, if w does not appear in thesource document, then ∑𝑖:𝑤𝑖 =𝑤 𝑎𝑖𝑡 is zero. Their model haschanging context vector ℎ𝑡∗ in decoding process. 𝑒𝑖𝑡 is thethe ability to produce OOV words which is one of theattention energy of hidden states of encoder at timestep iprimary advantages compared to the original Seq2Seq

the context vector.

in decoder step t.

𝑎𝑡

is the attention weights distribution.

model.

v T , W h , W s and b att en are learnable parameters.
𝑒𝑖𝑡 = 𝑣 𝑇 tanh(𝑊ℎ ℎ𝑖 + 𝑊𝑠 𝑠𝑡 + 𝑏𝑎𝑡𝑡𝑛 )𝑎𝑡 = softmax(𝑒 𝑡 )ℎ𝑡∗ = ∑𝑖 𝑎𝑖𝑡 ℎ𝑖(1)

3. NEURAL SUMMARIZATION MODELS

(2)
(3)

Rush, Chopra, et al. [1] were the first to apply Seq2Seq

In

this

section,we

describe

thebaselinesequence-to-sequence model, pointer-generator networkand our two-encoder pointer-generator network models.

and attention model on text summarization, achieving
state-of-the-art performance on DUC-2004 and Gigaword,
two sentence-level summarization datasets.
Based on Rush [1], Luong et al. [14] proposedA.

Sequence-to-sequence attentional model
We utilize the sequence-to-sequence attentional model

whose input is only target segment text as our baselinemodel. This Seq2Seq model is similar to that of Abigail etal. [2]. Figure 1 shows the structure of this model. The

B.

Pointer-generator network

words of target segment w are fed into the encoder (onesingle-layer Bi-LSTM) one by one in encoding timestep i,
producing a sequence of encoder hidden states h. Then the
decoder (one single-layer LSTM) receives the wordembedding of the word from the previous step (whentraining, the previous word is from the reference segment
summary) on each step t and has decoder state 𝑠𝑡 .

Fig. 2.

Pointer-generator network

We utilized pointer-generator network with coverageFig. 1.

sequence-to-sequence attentional model

Then the attention distribution𝑎𝑇

is calculated by

Bahdanau attention [4] in (1) and (2). The attention

mechanism proposed by Abigail et al. [2], which allows
both copying words by pointing from the source text andgenerating words from a fixed vocabulary. Figure 3 shows
the structure of this model.

distribution can be considered as a probability distribution

For the purpose of solving the common repetition

over the source words, which helps the decoder d ecide

problem for sequence-to-sequence models, they proposed

where to focus to produce next word.

coverage mechanism. They defined the sum of attention

Next the attention distribution is used to produce aweighted sum of the encoder hidden states, context vectorℎ𝑡∗distributions over all previous decoder timesteps ascoverage vector 𝑐 𝑡 :𝑡′
𝑐 𝑡 = ∑𝑡−1
𝑡 ′ =0 𝑎in (3). The context vector can be viewed as afixed-length representation of what has been read from thetarget segment for this step.

The coverage vector
distribution

over

the

𝑐𝑡

(9)

could be considered as asourcedocument

words

that

Then the context vector is concatenated with therepresents the degree of coverage that those words havedecoder state 𝑠𝑡 and fed through two linear layers toreceived from the attention mechanism so far. Note that 𝑐 0

produce the vocabulary distribution 𝑝𝑣𝑜𝑐𝑎𝑏 in (5). 𝑝𝑣𝑜𝑐𝑎𝑏

is a zero vector because on the first decoding timestep,is a probability distribution over all words in thenone has been covered from the source document. Then

vocabulary. In this model the final distribution P (w) to

the coverage vector is used as extra input to the attention

predict words w is:mechanism.
𝑃(𝑤) = 𝑃𝑣𝑜𝑐𝑎𝑏 (w)(6)

The loss for training of decode step t is the negative log
likehood of the target word𝑤𝑡∗:

𝑙𝑜𝑠𝑠𝑡 = − log 𝑃(𝑤𝑡∗ )
1

𝑇

the

formulatocalculatethe𝑒𝑖𝑡 = 𝑣 𝑇 𝑡𝑎𝑛ℎ(𝑊ℎ ℎ𝑖 + 𝑊𝑠 𝑠𝑡 + 𝑊𝑐 𝑐𝑖𝑡 + 𝑏𝑎𝑡𝑡𝑛 )(10)

Where 𝑊𝑐 is a learnable parameter vector of same(7)

The overall loss for the whole sequence is:𝑙𝑜𝑠𝑠 = ∑𝑇𝑡=0 𝑙𝑜𝑠𝑠𝑡

Changing

attention energy from (1) to:(8)length as v. Adding the coverage vector to the attention
mechanism making the attention mechanism’s current
decision is considering its previous decisions whichsummarized in the coverage vector 𝑐 𝑡 .

The experiment of this model has two input settings.

Then the attention distribution 𝑎𝑡 and context vector

One is the baseline that uses only target segment and theℎ𝑡∗ are calculated as in (2) and (3). The generation

other concatenates the target segment and compressedprobability 𝑝𝑔𝑒𝑛 ∈ [0,1] for timestep t is calculated as inwhole article as one input sequence to the model.

(4). Then the final probability distribution is calculated in(6).

fit our problem. We calculate the attention energy for two

They also defined a coverage loss to penalize repeatedly

hidden state sequences separately:

attending to the same locations:𝑐𝑜𝑣𝑙𝑜𝑠𝑠𝑡 = ∑𝑖 min(𝑎𝑖𝑡 , 𝑐𝑖𝑡 )The coverage loss is bounded(11)

as: 𝑐𝑜𝑣𝑙𝑜𝑠𝑠𝑡 ≤ ∑𝑖 𝑎𝑖𝑡 = 1.

Then they reweighted the coverage loss by hyperparameter𝑒1𝑡𝑖 = 𝑣1𝑇 tanh(𝑊1ℎ ℎ1𝑖 + 𝑊1𝑠 𝑠𝑡 + 𝑊1𝑐 𝑐𝑖𝑡 + 𝑏1𝑎𝑡𝑡𝑛 )(13)

𝑒2𝑡𝑖 = 𝑣2𝑇 tanh(𝑊2ℎ ℎ1𝑖 + 𝑊2𝑠 𝑠𝑡 + 𝑊2𝑐 𝑐𝑖𝑡 + 𝑏2𝑎𝑡𝑡𝑛 )

(14)

where

𝑒1𝑡𝑖

is the target segment’s attention energy and

λ and added it to the primary loos function. The final loss

𝑒2𝑡𝑖 is the additional sequence ‘s attention energy. 𝑣1𝑇 ,function is:

𝑊1ℎ , 𝑊1𝑠 , 𝑏1𝑎𝑡𝑡𝑛 , 𝑣2𝑇 , 𝑊2ℎ , 𝑊2𝑠 and 𝑏2𝑎𝑡𝑡𝑛 are
𝑙𝑜𝑠𝑠𝑡 = − log 𝑃(𝑤𝑡∗ ) + 𝜆 𝑐𝑜𝑣𝑙𝑜𝑠𝑠𝑡(12)

learnable parameters. Then we concatenate the two

For experiments, we use two input settings. On e is using

attention energy sequences into one by a learnabletarget segment text only and the other is using theparameter γ (1.0 as initial value) to calculate the whole

concatenated sequence with target segment and additional

attention weight distribution 𝑎𝑡 :𝑎𝑡 = softmax([𝑒1𝑡 , 𝛾𝑒2𝑡 ])sequence.
Considering just concatenating the target segment andadditional information sequence might make it hard for(15)

Then we change the formula to calculate the context
vector using attention weight distribution s from (3) to:𝑡 2ℎ𝑡∗ = ∑𝑘𝑖=0 𝑎𝑖𝑡 ℎ1𝑖 + ∑𝑚+𝑘
𝑖=𝑘+1 𝑎𝑖 ℎ𝑖

decoder to recognize which part is from the segment. In(16)

addition, the encoder is using Bi -LSTM which means thewhere k is the length of the target segment encoder’s

additional sequence would affect the target segment ’s

hidden states sequence and m is the length of theresult. Considering the noise information in additional

additional sequence encoder’s hidden state sequence.

sequence might have bad influence, we also con sider

Then the remaining step is similar to the previous

another model that uses different encoders to process thepointer-Generator network model. We use the sametarget segment and additional sequence separately.

calculation to obtain the final distribution, the loss is stillusing the formula (12).

C.

Two-encoder Pointer-generator network

4. TECHNIQUESFOR

ADDITIONALSEQUENCE
In this paper, we consider that the contents outside thetarget segment in the article contains useful information
for segment summarization. But normally the wholearticle would be too long.

Such long sequences would

make it hard to train the RNN-based model. In addition,
we cannot ignore that there would also be a large volume
of noise information outside the target segment. So in thissection, we discussed three single -document extractive
method to extract the important sentence s from the article
to form the additional input sequence.
Fig. 3.

Two-encoder Pointer-generator network
A.

Basic method– leading sentences

This model is similar to the previous model. There are

The basic method works by picking leading two

two encoders in the model, one is to process the target

sentences in each segment of the article and concatenate

segment and the other is to process the additional

these sentences into one sequence as the additionalsequence. Figure 3 shows the structure of this model.

sequence for our model.

There are two hidden state sequences produced by two

According to Abigail et al. [2], their experiment using

different encoders. The target segment encoder produces

only leading three sentences performs better than using

hidden states ℎ1 , and the additional sequence encoder

longer inputs. We consider that the important information

produces hidden states ℎ2 . Having two hidden stateis usually in the leading sentences. Thus, we utilize thissequences makes it not suitable for original Bahdanau

method as our basic method to extract important sentences

attention mechanism. So we are changing the attention tofrom the target article.

summaries is not realistic. So, we utilized Word2Vec to
calculate the average vector of segment title and segment
B.

TextRank + word2vec

sentences. Then use cosine similarity to compare segment

TextRank is a graph-based ranking algorithm inspired

title and segment sentences and choose the most similar

by PageRank [16] algorithm. It was first proposed bysentence. We concatenate it after the segment title as our

Mihalcea and Tarau [12]. TextRank is mainly used forgolden standard.

keyphrase extraction and important sentence extraction. In
our work, we are using it for sentence extraction.

Settings

B.

To apply TextRank and word2vec on sentence extraction,

For all experiments, we follow the settings in Abigail etthey first build a graph associated with the text, where theal. [2]. Our models have 256-dimensional hidden statesgraph vertices are representative for the sentences in theanddocument to be ranked. The main steps are:pointer-generator network models, we use a vocabulary of

Split the target document into sentences 𝑉 =(1)128-dimensional

word

embeddings.

For

the

50k words for both source and target.

[𝑆1 , 𝑆2 , 𝑆3 , … , 𝑆𝑚 ] , and construct graph G = (V, E). Then split

The word embeddings we are using are not pre -trained.

sentences into words and remove stop words to obtain 𝑆𝑖 =They are learned from scratch during training. We tra in[𝑡𝑖,1 , 𝑡𝑖,2 , … , 𝑡𝑖,𝑛 ], where 𝑡𝑖,𝑗 ∈ 𝑆𝑖 is the candidate keyword.

using Adagrad [10] with learning rate 0.15 and an initial

Using word2vec to convert words in a sentence

accumulator value of 0.1. We use gradient clipping with ato vectors and calculate the average vector of the

maximum gradient norm of 2 without using any form ofsentence:

regularization.

(2)

1

𝑣𝑒𝑐(𝑆𝑖 ) = ∑𝑛𝑗=1 𝑤𝑜𝑟𝑑2𝑣𝑒𝑐(𝑡𝑖,𝑗 )(3)CalculatethecosineFor training and testing, we truncated the target

(17)

𝑛

segment and additional sequence into leading 400 words

similarity

betweensentences as the edge weight s in graph G:

According to Abigail et al. [2], truncating the input

𝑆𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦(𝑆𝑖 , 𝑆𝑗 ) = 𝑐𝑜𝑠(𝑣𝑒𝑐(𝑆𝑖 ), 𝑣𝑒𝑐(𝑆𝑗 ))
(4)

(18)

the

We trained our models on a single 1080Ti GPU with abatch size of 32. For testing, we utilized beam search

𝑊𝑆(𝑆𝑖 ) = (1 − 𝑑) + 𝑑 ∑𝑆𝑗 ∈𝐼𝑛(𝑆𝑖) ∑bysequence could raise the performance of the model.

Calculate the score of each sentence, where thescore of 𝑆𝑖 is:(5)

and limit the length of the summary to 50 words.

𝑤𝑗𝑖
𝑆𝑘 ∈𝑂𝑢𝑡(𝑆𝑗 )𝑤𝑗𝑘

𝑊𝑆(𝑆𝑗 ) (19)algorithm to produce segment summaries with beam size
8.

Pick the top-10 highly scored sentences and sortpositionsofsentencesinthedocument.

Concatenate the sentences as our additional sequence.

To compare the performance between models, wetrainedourmodelsseparately

using

the

additi onal

sequence which is generated by the additional technique s
including leading sentences, TextRank and TextTeaser. We

C.

TextTeaser
TextTeaser is a feature-based extractive summarization

algorithm. It selects sentences that possess best scoreamong others. Its’ features include title feature, sentence
length, sentence position and keyword frequency.

also did the experiment with different length of target
segments to see how the length affect the result .
C.

Results
Our results are shown in Table I. We evaluate ourmodels with the standard ROUGE metric [ 3], reporting theF 1 scores for ROUGE-1, ROUGE-2 and ROUGE-L, which

5. EXPERIMENTS

respectively measure the word -overlap, bigram-overlap,
and longest common sequence between the referenceA.

Datasetssummary and the summary to be evaluated. According to

We collected 26,318 pages from English version of

Abigail et al. [2], ROUGE trends to reward safe s trategies

Wikipedia as our dataset for the experiment s. There aresuch as selecting the first-appearing content, or preserving

81,124 segments and 50,056 sections altogether in theoriginal phrasing, which may lead to extractive systems

dataset. We convert all the words to lower case and theobtains higher ROUGE scores on average . Therefore, we

numbers to “TAG_OF_NUM”.

also evaluate our systems with the METEOR metric, which

Since there is no human labeled segment summary inrewards not only exact word matches, but also matching

Wikipedia articles, and writing this number of segment

stems, synonyms and paraphrases.

TABLE I. RESU LT S OF DIFFERENT MODE LS

Additional
Sequence
Method
None

1

2

L

Exact

+stem/syn/para

44.99

33.20

38.84

26.72

27.46

None

36.70

19.33

29.90

17.08

17.65

None

42.17

28.01

35.82

23.26

23.91

Basic (leading)

42.70

29.02

36.00

25.20

25.73

ModelsTextRank
Seq2Seq
attentional model
Pointer-Generator
Network
Two-Encoder
Pointer-Generator
Network

ROUGE

METEOR

TextRank

43.96

30.23

37.31

26.33

26.86

TextTeaser

43.13

29.43

36.38

25.46

25.98

TextRank

44.69

31.04

38.17

27.17

27.69

target segment.
For the baseline models, the Seq2Seq attentional model
andPointer-Generator

network,

applying

Additionally, we performed experiments on how theadditional

length of a target segment affects the result. We divided

sequence method means concatenating the additional

the testing dataset into three parts by the length of thesequence after the target segment. For no -additionaltarget segments: less than 200 words; between 200 andsequence method, only truncated target segment was input

400 words; over 400 words. Then we use our trained

to the model.

Two-Encoder Pointer-Generator network model from theIn the results of the no-additional sequence method, we
can see that pointer-generator network outperforms thelast experiment to produce the results. Table II shows theresults.

traditional Seq2Seq attentional model. The model trains
faster and takes less iterations. Thus, in succeeding
experiments on additional sequence methods we only
focus on Pointer-Generator network.
We first perform experiments that simply concatenate
the additional sequence after the target segment. The

TABLE II. EXPERIMENT ON LENGTH OF TAR GET SEGMENT S
LengthOverall(0,200](200,400](400,∞ )

1
44.69
53.86
45.39
29.86

ROUGE
2
31.04
40.65
33.19
15.65

L
38.17
47.30
40.70
23.68

Exact
27.17
33.83
28.73
16.53

METEOR+stem/syn/para
27.69
34.26
28.93
17.20

results show that our idea of additional sequences from the

The results show that our model performs much better

entire article is improving the performance. Among them,when the target segment is less than 200 words andsequences that extracted by TextRank perform the best.

slightly better between 200 and 400 words. But it becomes

Therefore, for our models that process two sequences

worse when the length exceeds 400 words. It implies that

separately, we use additional sequences extracted byour model is more capable to handle s horter articles,

TextRank only.

which is one of the existing common problems for RNNs

The last row of Table 1 is the result of our proposingthat longer articles are harder to process , including LSTM.

model Two-Encoder Pointer-Generator network. As we

Also, shortercan see from the scores, our model out -performed all thesentences, given a higher possibility to locate theotherreference sentence in the segment.

modelsincludingoriginal

Pointer-Generator

segments mean a smaller number of

network with additional sequences by TextRank. Thus, ourConsidering that there has no perfect evaluation methodmodel that uses two encoders to process the target segment

for text summarization, we picked several results from theand the additional sequence separately appears to befour different methods we evaluated and observed byworking.

human eyes. Table III is two of those examples. We can

We also evaluate TextRank as an extractive method tosee that the Seq2Seq model are clearly the worst model. It

extract summary from the target segment. The first row

failed to generate a large number of Out -Of-Vocabularyshows TextRank is slightly better than our model. We(OOV) words, which the Pointer-Generator Network is

consider the possible reason can be that the major part ofcapable of.

golden standard in our dataset is one sentence from theTABLE I.

S UMMARY E X AMP LE S

Reference summary:
TAG_OF_NUM
state
representativeelectionin
TAG_OF_NUM. newhall was elected as a democrat to servein the massachusetts house of representatives of
TAG_OF_NUM.
Seq2Seq Attention model:
state representative in the TAG_OF_NUM election [UNK]
received TAG_OF_NUM [UNK] TAG_OF_NUM votes
coming in second behind republican candidate john [UNK]
gonzalez
who
received
TAG_OF_NUM[UNK]
TAG_OF_NUM [UNK] and just ahead of fell ow democrat
michael [UNK] jay who received TAG_OF_NUM [UNK]TAG_OF_NUM [UNK]Pointer-Generator
Network
Without
Additional
Sequence:
TAG_OF_NUM election in the TAG_OF_NUM election
newhall received TAG_OF_NUM , TAG_OF_NUM votes
coming in second behind republican candidate john w.
blaney who received TAG_OF_NUM , TAG_OF_NUM
votes, and just ahead of fellow democrat michael f. phelan
who received TAG_OF_NUM , TAG_OF_NUM [UNK]TAG_OF_NUM.
onmarch
TAG_OF_NUM,
TAG_OF_NUM, johnston released his first comic bookpublished by boom!.

and jonathan [UNK] on march TAG_OF_NUM [UNK]TAG_OF_NUM [UNK] johnston released his first comic[UNK] at [UNK] published by [UNK] [UNK] [UNK]
comic book [UNK] with the album and an warren's[UNK] has collaborated with skateboarding and clothingcompany supreme on numerous collections [UNK] ofclothing and various [UNK].
on march TAG_OF_NUM , TAG_OF_NUM , johnston
released his comic book, published by boom! [UNK] thecomic book ties-in with the album and an ios app.
johnston has collaborated with skateboarding andclothing company supreme on numerous collections

Pointer-Generator Network With TextRank additional
Sequence:
congress in TAG_OF_NUM newhall was elected as ademocrat to serve in the massachusetts house ofrepresentatives of TAG_OF_NUM.

and on march TAG_OF_NUM , TAG_OF_NUM , johnston
released his first comic book, at sxsw, published byboom! studios.

Two-Encoder Pointer-Generator Network (proposed):
TAG_OF_NUM election in TAG_OF_NUM newhall waselected as a democrat to serve in the massachusetts house ofrepresentatives of TAG_OF_NUM.

TAG_OF_NUM
onmarch
TAG_OF_NUM,
TAG_OF_NUM , johnston released his first comic book,at sxsw, published by boom! studios.

TextRank:
newhall was elected as a democrat to serve in themassachusetts house of representatives of TAG_OF_NUM.

on march TAG_OF_NUM, TAG_OF_NUM, johnston
released his first comic book published by boom!

• Green denotes segment title in the reference summary
Furthermore, Both the Seq2Seq attentional model andFirstly, for Wikipedia articles that have multiple

Pointer-Generator Network are generating a large number

segments, segments are usually not isolated from eachof extra words we need. Also, our methods with additional

other. They tend to contain similar information. It issequence are not generating so much words. This maybe

possible that these information could be extracted into the

because that the additional sequence we are using is

additional sequence that would be useful for segment

extracted from the entire article including the target

summarization.

segment. By adding the additional sequence as an extra

Secondly, the additional sequence can be regarded as

input to the model, the most important information of thecompressed information from the entire article. It is likely

target segment is emphasized twicethat it contains the target segment’s most important
information. By supplying the target segment and the

6. DISSCUSSIONA.

How the additional sequence works

additional sequence, the important information in thetarget segment could be emphasized.

It is clear from Table I that applying additional

Furthermore, we consider that additional sequence

sequence tends to achieve higher scores than none, no

could be useful for generating lon ger summaries like news

matter simple concatenation or using two encoder s toarticle. Existing similar news article’s summary could beprocess it separately. We offer two possible explanations

used as e additional sequences to the model, since thefor these observations.

summary of similar articles usually have similar writing

patterns. By introducing similar news articles’ highly
compressed summary, words that widely used in summary
but not appearing in the article could be introduced, which
could help the system more easily to generate.
B.

Why processing separately performs better
Rows 5 and 7 in Table I show that using the same

additional sequence and target segment, processing themseparately scores higher than concatenation. We haveseveral explanations for that.
As we can see in Table II, the results are getting worse
when the target segment is getting longer. We consider thisas the common problem for RNNs that they have problems
handling longer articles. Therefore, in our problem ,concatenating the target segment and additional sequenceis making the input sequence longer. Which wo uld make itharder for our LSTM-structured encoder to handle. Thus,in our two-encoder model, using two LSTM s to process
the target segment and additional sequence can avoid thisproblem in some degree.
Using several LSTMs to process parts of the input text
has been researched by researchers. Tan et al. [11] areusing a hierarchical encoder-decoder framework for text
summarization. Tan utilized word encoders and sentence
encoders in their model and it shows effectiveness. Thus,
we can assume that in future study, we could use a similar
structured hierarchical encoder-decoder model to process
long articles, by dividing the encoders into word encoders,sentence encoders and maybe paragraph encoders.

7. CONCLUSIONSIn this paper, we performed experiments on threedifferent

modelssummarizationforWikipedia

articlework including Seq2Seq

Pointer-Generator

network

andoursegment+ attention,
Two-Encoder

Pointer-Generator network. We also considered threedifferent ways to extract important information from thewhole article. The results show that our idea of additionalsequence and processing it separately is effective. Ourmodels out-performed the other models. Although there
still exist problems, one is that our model suffers fromlonger segments. But our Two -Encoder model’s results
suggest that the LSTM has the potential for dealing withlonger articles, if an appropriate hierarchical structure isemployed.

References[1] Alexander M Rush, Sumit Chopra, and Jason Weston.
2015. “A neural attention model for abstractive
sentence summarization”. In Empirical Methods in
Natural Language Processing.
[2] A See, PJ Liu, CD Manning. 2017. “Get To the Point:
Summarization with Pointer-Generator Networks”.

AnnualMeetingofthe
Associationfor
Computational Linguistics[3] Chin-Yew Lin. 2004a. Looking for a few good
metrics: Automatic summarization evaluation -howmany samples are enough? In NACSIS/NII Test
Collection for Information Retrieval (NTCIR)
Workshop[4] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshu a
Bengio. 2014. “Neural machine translation by jointly
learning to align and translate”. In Computer
Science.
[5] Dat P.T Nguyen, Yutaka Matsuo, and Mitsuru
Ishizuka. “Exploiting Syntactic and Semantic
Information for Relation Extraction from Wikipedia ”
Text-Mining and Link-Analysis (TextLink2007),
2007.
[6] Gers, Felix A., Nicol N. Schraudolph, and Jürgen
Schmidhuber. "Learning precise timing with LSTM
recurrent networks." Journal of machine learning
research 3, no. Aug (2002): 115 -143.
[7] Graves and Schmidhuber, J. “Framewise phonemeclassification with bidirectional LSTM networks ”. In
IEEE International Joint Conference on Neural
Networks. 2005, 4:2047-2052[8] Hu M, Sun A and Lim E-p 2007 Comments-oriented
Blog Summarization by Sentence Extraction Proc. of
the Sixteenth ACM Conf. on Conf. on Information
and Knowledge Management 901 -4 (Lisbon, ACM)
[9] Ilya Sutskever, Oriol Vinyals, Quoc V. Le. Sequence
to Sequence Learning with Neural Networks. In NIPS,
2014. 2, 3, 7
[10] John Duchi, Elad Hazan, and Yoram Singer. 2011.
Adaptive subgradient methods for online learningand stochastic optimization. Journal of Machine
Learning Research 12:2121–2159.
[11] Jiwei Tan, Xiaojun Wan and Jianguo Xiao. 2017.
Proceedings of the 55th Annual Meeting of the
Association for Computational Linguistics, pages
1171-1181[12] Mihalcea R, Tarau P. TextRank: Bringing order into
text. In Proceedings of the 2004 conference on
empirical methods in natural language processing
2004[13] M. Volkel, M. Krotzsch, D. Vrandecic, H. Haller, and
R. Studer. Semantic Wikipedia. In Proceedings of the
WWW2006, pages 585-594, 2006.
[14] Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol
Vinyals, and Lukasz Kaiser. 2016. “Multi-task
sequence to sequence learning”. In ICLR.
[15] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly.
2015. “Pointer networks”. In Neural Information
Processing Systems.
[16] Page, Lawrence, et al. The PageRank citation
ranking: Bringing order to the web. Stanford Info
Lab, 1999.
[17] Rada Mihalcea and Paul Tarau. “TextRank: Bringing
Order into Texts” 2004[18] S Hochreiter, J Schmidhuber. 1997. “Long short-term
memory”.
In
NeuralComputation,
1997,
9(8):1735-1780[19] Vishal Gupta, Gurpreet Singh Lehal. “A Survey of
Text Summarization Extractive Techniques ”. Journalof emerging technologies in web intelligence (Vol. 2,
No. 3, August 2010).