DEIM Forum 2019 B2-3

Query-Focused Extractive Summarization
based on Deep Learning:
Comparison of Similarity Measures for
Pseudo Ground Truth Generation
Yuliska † and Tetsuya SAKAI †† Department of Computer Science and Engineering, Waseda University
3–4–1 Okubo, Shinjuku-ku, Tokyo 169–8555, Japan
E-mail: †yuliska0791@fuji.waseda.jp, †tetsuyasakai@acm.org
Abstract Query-focused summarization aims to produce a single, short document that summarizes a set of documents that are relevant to a given query. While some deep learning approaches have recently been applied tosolve this task, how to automatically generate reliable ground truth labels for training remains an open problem.
In this study, we employ eight existing textual similarity measures to generate ground truth labels at the sentencelevel given a reference summary. We then feed these diﬀerent labelled data to deep learning approaches to generate
extractive summaries. We use the DUC 2005-2007 benchmark datasets in our experiment. Our study shows that
ROUGE-WE2 and ROUGE-SU measures achieved the best ROUGE scores in all deep neural models we employed.
Keywords Query-Focused Summarization, Extractive Summarization, Ground Truth Generation, Similarity Measures, Deep Neural Networkof each sentence in the final summary. However, Document
Understanding Conference (DUC)（注 1） dataset as benchmark

1 Introduction
Text summarization has recently gained much attention
in natural language processing due to its promise in various applications. Examples include search engine snippets
generation, news article headlines generation, question answering, and personalized recommendation engines. Amongthem, query-focused summarization (QFS), i.e, the task of
producing a short and concise summary of a document ora set of documents based on user’s query, remains a highly
challenging task.
Recently, deep learning has been applied to text summarization and the majority of such approaches are extractive.
Cao et al. [19] addressed the problem of query-focused extractive summarization using query-attention-weighted CNNs
(Convolutioal Neural Network).

Ren et al. [14] proposed

Query Sentence Relation (QSR) which also used CNN withattention mechanism while Cheng and Lapatta [3] employed
CNN and RNN (Recurrent Neural Network) as sentence extractor.
Neural-based Extractive summarization is often regarded
as classification problem [19] [15] [16] [3]. Thus, ground truthlabels are needed for training so that the model can generate correct prediction in the form of membership probability

dataset in text summarization only contains manual abstractive summaries as ground truth. To solve this problem, recent extractive summarization studies applied an unsupervised approach to convert the abstractive summaries to extractive labels. Nallapati et al. [15] and Nallapati and Ma [16]employed the following variants of ROUGE (Recall Oriented
Study for Gisting Evaluation) [10]: ROUGE-1, ROUGE-2,
and ROUGE-L similarity measures to generate ground truthin the form of sentence-level binary labels while Cao et al. [19]and Ren et al. [14] only utilized ROUGE-2. Cheng and Lapatta [3] used a rule-based system which is similar to ROUGE1 and ROUGE-2 to form ground truth labels. These studies applied diﬀerent similarity measures to generate groundtruth, but which measure is the most reliable remains an
open problem.
In this study, we focus on query-focused extractive summarization and reimplement eight textual similarity measures togenerate ground truth at sentence level. We then feed these
diﬀerent labelled data to deep learning models to extract
summaries and analyze their result. Our study shows that
ROUGE-WE2 and ROUGE-SU measures achieved the best
（注 1）：https://duc.nist.gov/data/

ROUGE scores in all deep neural models we employed.

level binary labels for query-focused extractive summarization system based on deep learning. In our experiment, we

2 Related Workstudy the eﬀectiveness of each textual similarity in generat-

2. 1 Query-Focused Extractive Summarization
Studies on query-focused extractive summarization spans

ing ground truth to identify the most reliable measure.

3 Approacha large range of approaches. Early studies on this task mostly
used unsupervised graph-based approach to extract bothThis section describes the DUC 2005-2007 datasets, tex-salient and query-dependent sentences [11] [2], where nodes

tual similarity measures we employed, and the general

are sentences and the edge scores reflect the similarity be-scheme of ground truth generation. Following that, we ex-

tween sentences, each node is given a relevance weight based

plain about some deep learning models we utilized as extrac-

on its relevance to the query. Following that, supervised

tive summarizer.

machine learning approaches are applied to solve a queryfocused summarization task. Ouyang et al. [18], Daume III
and Marcu [7], Conroy et al. [8] used Support Vector Regression (SVR), Bayesian Statistical Model (BAYESUM), andHidden Markov Model (HMM) respectively to extract querydependent and query-independent features and thereby estimate the importance of sentences.
2. 2 Extractive

Summarization

3. 1 Dataset
Our experiments are conducted on the DUC 2005-2007
datasets. All the documents are from news articles and clustered into various thematic clusters. In DUC 2005, there
are 4-9 reference summaries in each cluster. While in DUC
2006-2007, there are four reference summaries in each cluster.
These reference summaries are created by NIST (National In-

based

on

Deep

Learning

stitute Standards and Technology) assessors which consistsof approximately 250 words. Table 1 shows the statistics ofFollowing the popularization of deep learning, many sum-

the three datasets. We use DUC 2005 as the training set,marization systems have also employed deep learning tech-

DUC 2006 as the evaluation set, and DUC 2007 as the testniques to address both general and query-focused summa-

set.

rization.

Cheng and Lapatta [3] treated single document

summarization as a sequence labelling task by utilizing CNN

Table 1 Statistics of DUC 2005-2007 Datasets
Year Clusters Sentences Data Source

as sentence encoder and LSTM (Long Short-Term Memory)

2005

50

45931

TREC

as sentence extractor. Cao et al. [19] addressed the prob-

2006

50

34560

AQUAINT

2007

45

24282

AQUAINTlem of query-focused extractive summarization using queryattention-weighted CNNs. Ren et al. [14] proposed Query
Sentence Relation (QSR) which also used CNN with atten-

3. 2 Similarity Measurestion mechanism. Kobayashi et al. [5] simply used the sum of
trained word embeddings as sentence or document represen-

The following are the similarity measures that we apply
for generating the ground truth:

tation.

3. 2. 1 ROUGE
2. 3 Ground Truth in Extractive Summarization
Many extractive summarization systems take a sentence

We use the following ROUGE measures:
ROUGE-N: ROUGE-N is an N-grams recall between a sys-

Hence, how to generate reliable

tem summary (sentence in pseudo ground truth generation)

ground truth in the form of sentence label is one of the pri-and a set of reference summaries. It is computed as follows:classification approach.

mary issues [12]. Ouyang et al. [18] generated ground truthlabels by using several N-gram based methods to compute

RN -gram =Countmatch (Ref erenceN -gram ,SummaryN -gram )Count(Ref erenceN -gram )similarity between sentence and reference or gold summary.
Nallapati et al. [15] and Nallapati and Ma [16] employedCountmatch is the overlapping number of N-grams of refer-

ROUGE-1, ROUGE-2, and ROUGE-L similarity measuresence summary and system summary and Count is the total

to generate ground truth in the form of sentence-level binary

number of N-grams. Our goal is to give ROUGE score tolabels while Cao et al. [19] and Ren et al. [14] only utilizedeach sentence, but the gap of the length between sentences

ROUGE-2. Cheng and Lapatta [3] used combination of uni-and reference summaries are quite big, so it is not wise togram and bigram to form ground truth labels.

score sentences based on ROUGE-N based recall only, be-

In the present study, we apply eight textual similarity

cause we want to also take the sentence length into consid-

measures to generate ground truth in the form of sentence-eration. To alleviate this problem, we use the ROUGE-N

based F-Measure, which is computed as follows:

ity between words in a system summary and reference summaries. Thus, it gives a better evaluation for summarization

PN -gram =Countmatch (Ref erenceN -gram ,SummaryN -gram )Count(SummaryN -gram )task. The following explains how word embeddings can beincorporated into ROUGE.

(1+β 2 )RN -gram PN -gram
FN -gram = R
2
N -gram +β PN -gram

The ROUGE-N based F-Measure (FN -gram ) combines therecall and precision computation, so it will give a fair score
to the long and short sentences. We set the value of β to
1 and we let N = 1,2 for ROUGE-1 and ROUGE-2 respectively.
ROUGE-L: ROUGE-L computes the Longest Common
Subsequence (LCS) of a system summary and reference sum-

Formally, ROUGE defines the following scenario to compute similarity between two words:1, if w1 = w2
fR (w1 , w2 ) =0, otherwise
In ROUGE-WE, it defines similarity by the following:0,
if v1 or v2 are OOV
fW E (w1 , w2 ) =v .v , otherwise
1

2

maries. Let X and Y be a sentence and a reference summary
respectively, given those two sequences, X and Y , the longest

OOV here means a situation where one of the words com-

common sequence of X and Y is a common subsequence withpared is not in the word embeddings vocabulary.

maximum length. We use LCS based F-Measure, as follows:

are 3 variants of ROUGE-WE in original paper: ROUGE-

There

WE1, ROUGE-WE2 and ROUGE-WE-SU. We only employ
ROUGE-WE2 and ROUGE-WE-SU in our experiment since)R(lcs) = LCS(X,Ymthese measures have good correlations with human judgements. For word embeddings, we use the same pretrained)P(lcs) = LCS(X,Ynword embeddings that we use for training.
2)Rlcs Plcs
F(lcs) = (1+βR
+β 2 Plcs

lcs

3. 2. 3 Keyword Overlap
We combine some variants of ROUGE (ROUGE-Comb)

where m is the length of the reference summary and n is thewith the query keywords overlapping number. To extract

length of the sentence.

keywords from query, we utilize the NLTK（注 2） parser to gen-

ROUGE-SU: ROUGE-SU computes SKIP-Bigram be-erate parsing tree from each sentence. The words in thetween a system summary and a set of reference summaries,

query are considered to be keywords if they are the tags:with the addition of unigram as counting unit. SKIP-Bigram

NN, NNS, NNP, NNPS, VB, VBD, VBG, VBN, VBP, andis any pair of words in their sentence order, allowing for ar-

VBZ.

bitrary gaps. We use SKIP-Bigram based F-Measure. It is

3. 2. 4 Embedding Similarity

computed as follows:

Following the study of Kobayashi et al. [5], we use pretrained word2vec (same as that we use for training) to trans-)+U nigram
R(skip) = SKIP -Bigram(X,Ymform the words in the sentences into high dimensional vectors)+U nigram
P(skip) = SKIP -Bigram(X,Yn

For reference summary representation, we apply a similar ap-(1+β 2 )Rskip Pskip
F(skip) = R
2
skip +β Pskip

reference summary Y , we use cosine similarity as follows:

ROUGE-SU comes with skip-distance parameter that defines
the distance between the word during SKIP-Bigram creation.
In this study, we set skip-distance parameter to 4.
ROUGE-Comb: A combination of ROUGE-1, ROUGE-2,
ROUGE-L, and ROUGE-SU.
3. 2. 2 ROUGE-WE (Word Embeddings)
ROUGE is biased toward surface lexical similarities whichmakes it unsuitable for evaluating summaries with substantial paraphrasing. ROUGE-WE [13] comes with a solution to

and get the average of those word vectors as sentence vectors.
proach. To compute the similarity between sentence X and(X)∗V (Y )Cos(Y, X) = ||VV(X)|∗|V(Y )||

3. 3 Pseudo Ground Truth Generation
Figure 1 explains how we generate the ground truth byutilizing textual similarity measures described in previous
section. Given sequence of sentences S = [s1 , s2 , …, sn ] andsequence of abstractive summaries M = [m1 , m2 , …, mn ],similarity measures will give a score to each sentence basedon its similarity to each abstractive summary. We then sortthe sentences in descending order based on their similarity

overcome this shortcoming. Instead of using N-grams overlap, it utilizes word embeddings to determine the similar-（注 2）：https://www.nltk.org/

score, and finally label the top K sentences as “1” and “0”These vectors are then fed into MLP to binary classify theotherwise. Regarding the value for K, we experimented withsentence. As for encoders, we explain them as the following:

5%-50% of the training set and decided to label the top 10%

Bi-LSTM-Max: A concatenation of forward and backward

scored sentences as “1” as it derived the best result. This

LSTMs. Hence, the representation produced using this net-ground truth generation approach is based on the idea thatwork combines information from forward and backward di-

the sentences with label “1” should be the one that maximize

rection of sentence and question. We perform Max-pooling

the ROUGE score with respect to gold summaries and highly

operation by selecting the maximum value over each dimen-

relevant to the related query as gold summary also contains

sion of the hidden units [4].

information which is questioned in query. Thus, the mod-

HieConv: We adopt this network architecture from Con-els should also give a high membership score to the sentenceneau et al. [1]. It consists of four layers of CNN where atwith label “1”.

every layer, a Max-pooling operation is applied over the feature maps. Then, the concatenation of each Max-pooling
output is the final representation.
Stacked LSTM: Herman and Schrauwen [6] shows thatstacking multiple Recurrent Neural Network (RNN) can potentially improve the sequence prediction problem. In ourstudy, we attempt to stack two LSTM layers on top of eachother, making the model capable of learning higher-level ofrepresentation both sentence and query.
3. 5 Summary Sentences Selection

Figure 1 Ground Truth Generation Scheme

To select summary sentences, we employ a greedy approach as in many previous studies. Specifically, at test time,
we sort the sentences in descending order according to thederived membership score from the model. We add a new
sentence to the current summary if it contributes new bigrams to a certain degree. More specifically, if at least 50%of the bigrams from candidate sentences are new, we add it
to the summary, until the summary contains approximately
250 words.

4 Experiment
4. 1 Experimental Setup
We use NLTK to perform preprocessing as well as keyword
extraction. All neural models are implemented on Keras（注 3）.
Figure 2 General Training Scheme

The 300 dimensional pretrained Word2Vec（注 4） vectors areused as word embeddings and the word embeddings are fine-

3. 4 Deep Learning Models

tuned during training. We set the same hidden size for allmodels which is set to 50. For CNN, we apply ReLU ac-

This subsection introduces the models we considered forachieving query-focused extractive summarization. We prefer to treat extractive query-focused summarization as binary
sentence classification task. We adopt the work of Conneau
et al. [1] to train our model as depicted in Figure 2. As can beseen, we treat these models as an encoder which outputs sentence and query representations. After those representations
are obtained, three operations are applied: (i) concatenation(S, Q); (ii) element-wise product (S ∗ Q); and (iii) absolute

tivation function and set the filter to 2. For LSTM and
Bi-LSTM, we apply Tanh activation. Pooling size is set to
2 at Max-pooling layer, for all models that is applied Maxpooling. Dropout operation is performed before feeding theword embeddings to the neural models and at every layerof network, we set the ratio of dropout to 0.5. In all neural networks, we also apply weight regularization (L2 norm)
with the weight is set to 10-3 . The hidden size of MLP lay-element-wise diﬀerence (|S − Q|). The result vectors will

（注 3）：https://keras.io/

contain the information of sentence and query relationship.

（注 4）：https://code.google.com/archive/p/word2vec/

Table 2 *Mean* ROUGE Scores (%) on Three Neural Models. Emb-Sim: Embedding
Similarity, Comb+Query: Combination of ROUGE and keyword overlapping.
Model

Sim Measures

Evaluation Measures
ROUGE-1 ROUGE-2 ROUGE-L ROUGE-SU

Bi-LSTM-Max ROUGE-2

41.00

10.51

18.10

15.99

ROUGE-L

42.46

10.86

18.30

16.93

ROUGE-SU

43.56

11.60

18.56

17.73

Comb

43.33

11.43

18.63

17.63

ROUGE-WE2

42.66

11.93

19.02

17.33

ROUGE-WE-SU 41.84

11.26

17.97

16.61

Emb-Sim

42.76

10.75

18.32

17.34

Comb+Query

42.56

11.30

18.66

17.11

ROUGE-2

41.16

10.37

17.99

16.03

ROUGE-L

43.05

10.91

18.63

17.55

ROUGE-SU

43.26

11.30

18.43

17.57

Comb

42.89

11.07

18.46

17.35

ROUGE-WE2

42.48

11.75

18.79

17.24

ROUGE-WE-SU 42.27

11.24

18.37

16.99

Emb-Sim

41.62

10.27

18.00

16.41

Comb+Query

42.22

10.81

18.40

16.86

StackedROUGE-2

40.63

10.23

17.98

15.93

LSTM

ROUGE-L

42.45

10.57

17.91

16.81

ROUGE-SU

42.94

11.14

18.32

17.43

Comb

42.19

10.84

18.02

16.63

ROUGE-WE2

41.29

11.40

18.31

16.24

ROUGE-WE-SU 41.96

11.04

18.35

16.85

Emb-Sim

41.97

10.65

18.10

16.59

Comb+Query

41.99

10.46

18.09

16.56

HieConv

Table 3 Tukey HSD Test Results on Three Diﬀerent Models. p: p-value, ES: Eﬀect Size.
Bi-LSTM-Max

ROUGE-L

ROUGE-SU Comb

ROUGE-WE2 ROUGE-WE-SU Comb+Query Emb-Sim

p=0.9958

p=0.2377

p=0.3347

p=0.0006

p=0.3761

p=0.6205

p=0.9999

ES= 1.2010 ES=0.9431

ES=1.1888

ES= 0.9330

ES=1.2179

ES=1.0983

ES=1.2955-

p=0.7011

p=0.8087

p=0.0640

p=0.8435

p=0.9637

p=0.9999-

ES=1.2777

ES=1.1936

ES=1.3986

ES=1.4592

ES=1.2543

ES=1.1855--

p=0.9999

p=0.9040

p=0.9999

p=0.9986

p=0.5768--

ES=0.9887

ES=0.8452

ES=1.2290

ES=1.2543

ES=1.1112---

p=0.8247

p=1.000

p=0.9998

p=0.6988---

ES=0.9433

ES=1.2749

ES=1.3451

ES=12863

-

-

-

-

p=0.7879

p=0.5521

p=0.0377

-

-

-

-

ES=1.0238

ES=1.2095

ES=1.4246

ROUGE-WE-SU -

-

-

-

-

p=0.9999

p=0.7418

-

-

-

-

-

ES=1.2516

ES=1.3208

-

-

-

-

-

-

p=0.9163

-

-

-

-

-

-

ES=1.4372

p=0.9875

p=0.4356

p=0.9386

p=0.0986

p=0.6079

p=0.9949

p=0.9999

ES=1.2246

ES=1.0872

ES=1.2134

ES=1.0662

ES=1.0013

ES= 1.1311

ES=1.1264-

p=0.9343

p=0.9999

p=0.5388

p=0.9821

p=1.000

p=0.9667-

ES=0.8587

ES= 1.1255 ES=1.2870

ES=1.0748

ES=0.8379

ES=1.1700--

p=0.9861

p=0.9960

p=0.9999

p=0.8941

p=0.3327--

ES=0.8645

ES=0.9654

ES=1.0241

ES=0.8379

ES=1.1256---

p=0.7341

p=0.9982

p=0.9999

p=0.8830---

ES=0.9849

ES=0.9946

ES=1.0312

ES=1.2835

-

-

-

-

p=0.9767

p=0.4565

p=0.0636

-

-

-

-

ES=1.0913

ES=1.0514

ES=1.1863

ROUGE-WE-SU -

-

-

-

-

p=0.9647

p=0.4946

-

-

-

-

-

ES=1.0302

ES=1.3883

-

-

-

-

-

-

p=0.9834

-

-

-

-

-

-

ES=1.1994

p=0.9729

p=0.5565

p= 0.9080

p=0.1121

p=0.5183

p=0.9984

p=0.9383

ES=1.3743

ES=1.2481

ES=1.1688

ES=0.9556

ES=1.1556

ES=1.0046

ES=0.9696-

p=0.9872

p=0.9999

p=0.6605

p=0.9818

p=0.9999

p= 0.9999-

ES=1.3993

ES=1.0583

ES=1.2222

ES=1.4417

ES=1.2374

ES=1.1360--

p=0.9985

p=0.9895

p=1.000

p=0.9114

p=0.9964--

ES=1.1512

ES=1.3092

ES=1.2331

ES=1.2374

ES=1.0421---

p=0.8198

p=0.9975

p=0.9982

p=1.000---

ES=1.2670

ES=1.2327

ES=0.9343

ES=0.9321

-

-

-

-

p=0.9930

p=0.4048

p=0.7658

-

-

-

-

ES=1.2384

ES=1.0534

ES=1.0672

ROUGE-WE-SU -

-

-

-

-

p=0.8911

p=0.9943

-

-

-

-

-

ES=1.7193

ES=1.2806

-

-

-

-

-

-

p=0.9993

-

-

-

-

-

-

ES=0.9510

ROUGE-2
ROUGE-L
ROUGE-SUROUGE-Comb
ROUGE-WE2

Comb+Query

HieConv

ROUGE-2
ROUGE-L
ROUGE-SUROUGE-Comb
ROUGE-WE2

Comb+Query

Stacked LSTM ROUGE-2
ROUGE-L
ROUGE-SUROUGE-Comb
ROUGE-WE2

Comb+Queryers are 100, 50, and 1. We implement minibatch gradientdescent using Adam [9] with learning rate is set to 10-2 and11.14, 18.32, and 17.43 for ROUGE-1, ROUGE-2, ROUGE-

batch size = 100. To evaluate the quality of summary, we use

statistically outperform the other measures. ROUGE-WE2,

variants of recall-based ROUGE metric for a deeper analy-

ROUGE-Comb, and ROUGE-WE-SU have indistinguishable

sis: ROUGE-1, ROUGE-2, ROUGE-L and ROUGE-SU. To

performances. Among them, ROUGE-WE-SU achieves thecompare the performance of each textual measure on eachbest ROUGE scores. This model also has a similar trend formodel statistically, we perform Tukey HSD (Honestly Sig-

ROUGE-2 measure which obtains the worst ROUGE scoresnificant Diﬀerent) test based on two-way ANOVA (withoutamong the measures compared.

L and ROUGE-SU respectively. Nevertheless, it does not

Furthermore, we also notice a specific trend for ROUGE-

replication) [17].
4. 2 Result and Discussions

SU and ROUGE-WE2 in all models.

ROUGE-SU tends

to generate summaries which have the best ROUGE-1 andTable 2 reports on ROUGE scores of the summaries ob-

ROUGE-SU scores among the measures compared. As fortained based on diﬀerent kinds of textual similarity mea-

ROUGE-WE2, it tends to output summaries which havesures for pseudo ground truth generation on three deep neu-

good ROUGE-2 and ROUGE-L scores among the measures

ral models.

compared.

Table 3 shows the Tukey HSD p-values and eﬀect sizes(ES) (i.e, standardized mean diﬀerences) based on two-way

5 Conclusion

ANOVA (without replication) of diﬀerent kinds of textual
similarity measures for pseudo ground truth generation on

Among the textual similarities that we explored in

We follow the study of Cao

three deep neural networks, ROUGE-WE2 and ROUGE-SU

et al. [19] which considered ROUGE-2 as the main evalua-achieved the best ROUGE scores. However, from Table 3, ittion score, hence, we perform the Tukey HSD test based onseems that ROUGE-WE2 only statistically significantly out-

ROUGE-2 scores of output summaries.

performed ROUGE2 and Emb-Sim measures for Bi-LSTM

three deep neural models.

In Bi-LSTM with Max-pooling model, ROUGE-SU out-

with Max-pooling model, but otherwise the diﬀerences be-

performs the other similarity measures. It achieves the best

tween diﬀerent measures were not statistically significant.

ROUGE scores in general, with 43.56, 11.6, 18.56, and 17.73

Overall, we suggest that ROUGE-WE2 and ROUGE-SU are

for ROUGE-1, ROUGE-2, ROUGE-L and ROUGE-SU re-the best measures among the textual similarity measuresspectively. However, the diﬀerences between ROUGE-SU

compared and ROUGE-2 might be the worst measure forand the other measures are not statistically significant (p-pseudo ground generation.

value > 0.05).

ROUGE-WE2 comes as the second-bestmeasure with 42.66, 11.93, 19.02, and 17.33 for ROUGE-

References

1, ROUGE-2, ROUGE-L and ROUGE-SU respectively.

[1] Alexis Conneau, Douwe Kiela, H. S. L. B. and Bordes, A.:
Supervised Learning of Universal Sentence Representations
from Natural Language Inference Data, Proceeding of the2017 Conference on Empirical Methods on Natural Language Processing, pp. 670–680 (2017).
[2] Bhaskar, P. and Bandyopadhyay, S.: A Query FocusedMulti-Document Automatic Summarization, Proceeding of
Twenty-Four Pacific Asia Conference on Language, Information and Computation, pp. 545–554 (2010).
[3] Cheng, J. and Lapatta, M.: Neural Summarization by Extracting Sentences and Words, Proceeding of 54th Annual
Meeting of the Association for Computational Linguistics,
pp. 484–494 (2016).
[4] Collobert, R. and Weston, J.: A Unified Architecture for
Natural Language Processing: Deep Neural Networks with
Multitask Learning, Proceeding of the 25th International
Conference on Machine Learning, pp. 160–167 (2008).
[5] Hayato Kobayashi, M. N. and Yatsuka, T.: Summarization Based on Embedding Distributions, Proceeding of the2015 Conference on Empirical Methods in Natural Language Processing, pp. 1984–1989 (2015).
[6] Hermans, M. and Schrauwen, B.: Training and Analyzing
Deep Recurrent Neural Network, In Advances in Neural Information Processing System 26, pp. 190–198 (2013).
[7] III, H. D. and Marcu, D.: Bayesian Query-Focused Summarization, Proceeding of the 21st International Conference on

Nonetheless, ROUGE-WE2 only statistically significantly
outperforms ROUGE-2 (p-value = 0.0006) and Emb-Sim(p-value = 0.0377). Meanwhile, we can easily notice that
ROUGE-2 clearly underperforms the other similarity measures.
In HieConv model, ROUGE-WE2 measures achieves thebest ROUGE scores with 42.48, 11.75, 18.79, and 17.24 for
ROUGE-1, ROUGE-2, ROUGE-L and ROUGE-SU respectively. But, its diﬀerences to the other similarity measures
are not statistically significant. It is followed by ROUGE-SU
measure that is outperformed by ROUGE-WE2 moderately
with 43.26, 11.30, 18.43, and 17.57 for ROUGE-1, ROUGE-2,
ROUGE-L and ROUGE-SU respectively although it is notstatistically better than the rest measures. In this model,
ROUGE-2 measure also noticeably underperforms the othersimilarity measures.
In Stacked LSTM model, it is clear that ROUGE-SU measure outperforms the other similarity measures with 42.94,

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

[18]

[19]

Computational Linguistics and 44th Annual Meeting of the
ACL, pp. 305–312 (2006).
Jhon M. Conroy, J. D. S. and Stewart, J. G.: Classy QueryBased Multi-Document Summarization, Proceeding of theDocument Understanding Conf. Wksp. 2005 at the Human
Language Technology Conf./Conf. on Emprical Methods in
Natural Language Processing (HLT/EMNLP) (2006).
Kingma, D. P. and Ba, J. L.: Adam: A Method for Stochastic Optimization, Proceeding of the 3rd International Conference on Learning Representations (2015).
Lin, C.-Y.: ROUGE: A Package for Automatic Evaluationof Summaries, Proceedings of the ACL-04 Workshop, pp.
74–81 (2004).
M. Sravanti, C. R. C. and Kumar, P. S.: QueSTS: A
Query Specific Summarization System, Proceeding of theTwenty-First International FLAIRS Conference, pp. 219–
224 (2008).
Mehdi Allahyahri, Seyedamin Pouriyeh, M. A. S. S. E. D. T.
J. B. G. and Kochut, K.: Text Summarization: A Brief Survey, International Journal of Advanced Computer Scienceand Applications, Vol. 8, No. 10, pp. 397–405 (2017).
Ng, J.-P. and Abrecht, V.: Better Summarization Evaluation with Word Embeddings for ROUGE, Proceeding of the2015 Conference on Empirical Methods on Natural Language Processing, pp. 1925–1930 (2015).
Pengjie Ren, Zhumin Chen, Z. R. F. W. L. N. J. M. andRidjke, M. D.: Sentence Relation for Extractive Summarization with Deep Neural Network, ACM Transaction on
Information System (TOIS), Volume 36 Issue 4, Article
No. 39 (2018).
Ramesh Nallapati, B. Z. and Ma, M.: Classify or Select:
Neural Architectures for Extractive Document Summarization, Under Review as Conference Paper at ICLR 2017(2017).
Ramesh Nallapati, F. Z. and Zhou, B.: SummaRuNNer:
A Recurrent Neural Network Based Sequence Model for
Extractive Summarization of Documents, Proceeding ofThirty-First AAAI Conference on Artificial Intelligence,
pp. 3075–3081 (2017).
Sakai, T.: Laboratory Experiments in Information Retrieval: Sample Sizes, Eﬀect Sizes, and Statistical Power,
Springer, pp. 51–80 (2018).
You Ouyang, Wenjie Li, S. L. and Lu, Q.: Applying Regression Model to Query-Focused Multi-Document Summarization, International Journal of Information Processing andManagement, pp. 227–237 (2011).
Ziqiang Cao, Wenjie Li, S. L. F. W. and Li, Y.: Attsum:
Join Learning of Focusing and Summarization with Neural
Attention, Proceeding of COLING 2016, the 26th International Conference on Computational Linguistic: TechnicalPaper, pp. 547–556 (2017).