言語処理学会 第25回年次大会 発表論文集 (2019年3月)

Informative Sections and Relevant Words
for the Generation of NLP Article Abstracts
Tianjiao Li and Yves Lepage
IPS, Waseda University
ltj1994@fuji.waseda.jp, yves.lepage@waseda.jp

Abstract
In this paper, we show that, in the field of NLP, abstractsof articles make only use of the text found inside the article. We introduce a multi-channel summarization method
which makes use of only some sections to generate asummary. We show that the introduction and the conclusion are the most useful sections to generate accurateabstracts. We also show that reducing the content to informative N-grams suffices to retain the necessary information to generate accurate abstracts.

1Introduction

There are two approaches to automatic summarization:
extractive and abstractive [2]. Extractive summarization
only copies words and sentences from the source text. It
cannot create new words. Summarization techniques onscientific articles are mostly extractive approaches. The
most advanced techniques use extra information such as
citation summaries or citation networks. In contrast, abstractive summarization reads the whole source text andgenerates a summary which may contain words not foundin the source text [6]. If machine learning techniques areused, these new words are picked from the vocabulary
learned during the training phase.
Original extractive methods usually calculate a score
to reflect how important a word or a sentence is. They
use words and sentences which have highest score in
their output [8]. However, sentences selected in this way
may be not relevant in content. They may also not be in
the right order in structure. Work has been done to try
to solve this problem, including work on rhetorical status and structure [7, 13], and citation networks [9, 10].
Such previous work concluded that, in order to improve
the performance of extractive summarization, extra information such as rhetorical status or citation networks isneeded.
On the contrary to such previous studies, we show
that, to generate a summary for a scientific article, all
we need is the content of the article itself and that noextra information is needed. Therefore, we choose abstractive method to generate summaries. Abstractive
methods are usually based on recurrent neural networks(RNN) [1, 12]. RNNs are a kind of neural networks

equipped with a memory about the history of the input.
They can generate words based on this history. SuchNNs are suitable for the generation of abstractive summaries. OpenNMT-py [4] is an open-source tool originally designed for neural machine translation. It has been
extended to text summarization and other tasks. It uses
RNNs as its basic framework. However, being a mockup tool, it has a heavy practical restriction on the lengthof the input. We confirmed this during our experimental manipulations. This practical constraint is a challenge
which goes into the direction of our claim that only thecontent of a scientific article is sufficient for the generation of its abstract. It goes even further in constraining us
to propose some light pre-processing to make the article
content even shorter, without having an impact on the result of summarization. In other words, the pre-processing
to shorten the content of the article should keep the informative content that may appear in the summary.
To train the neural network, a reasonable amount ofdata is necessary. But there is no such data set available
on scientific articles. So we built one by ourselves.
In this paper, we introduce a multi-channel abstractive
approach to generate summaries of scientific articles only
using their sections. We show that to generate a summary,all we need is the article itself. Moreover, we hope we cangenerate abstract-like summaries, so that once an article
is written, its abstract can be generated automatically.

2Background

2.1

Recurrent Neural Network

Recurrent neural networks (RNN) use hidden units tostore the information from the current input and previous
hidden units, so that they have a memory of inputs of all
the time. It is suitable to be used on summary generation.
One of their drawbacks is that their performance drop on
longer inputs [3].
In the encoder-decoder model, the encoder part reads
the source sentence token by token and uses hidden states
to store the information. The final output of the encoder
part is passed to the decoder part which generates the target sentence token by token. All tokens previously generated influence the generation of the next token. As aresult of an output token being conditioned on the source― 1281 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

Figure 2: Multi-channel method

Figure 1: Number of times a token in an article appears
in the abstract
and the previously generated target part, sentences generated in this way are more grammatical and more fluent.

2.2

ACL-ARC data set

Collections of articles built for extractive summarization
are too small in size for training a neural network. Therefore, we built a data set using the ACL-ARC corpus1 .
ACL-ARC is a corpus of scholarly publications about
computational linguistics. It consists of 22,878 articles.
We extracted the text from each article and broke it down
into four parts: abstract, introduction, conclusion andbody (i.e., the rest). So far, we have already extracted
texts from 15,000 articles.

2.3

ROUGE

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) [5] is a metric to evaluate the performance of summarization. The scores range from 0 to 100. The higher
the score, the better the performance. Rouge computesa recall, a precision and an F-score against a gold standard. The unit to compute the scores may be unigrams(ROUGE-1) or bigrams (ROUGE-2). The ROUGE-L scoreuses the length of the longest common subsequence as
the numerator to calculate the recall and precision scores.

3

Which sections are informative
for an abstract?

We use the open-source tool OpenNMT-py for our experiments. The basic mechanism of OpenNMT-py is attention based recurrent neural network. As mentioned
above, the performance of RNNs drops when the input
1 http://acl-arc.comp.nus.edu.sg/

length increases. In practice, OpenNMT-py does not allow to use the whole body of an article as an input. Itlimits the input to 400 tokens. To remedy this constraint,
we propose to use only some sections of an article. It is
like a filter through which only some piece of information
is passed. We call each piece of information a channel.
In this way a whole article will be divided into several
channels.
The next question is which parts of an article, i.e.,
which channel, is most effective for our goal of summarizing an article and getting an abstract-like summary.
Usually, an abstract is structured in the following way:
it introduces the background of a topic, states the existing problems, proposes a new approach, mentions themain results of experiments and stresses the improvements made.
A natural hypothesis is that all these points are described in details mainly in the introduction and the conclusion sections. To confirm this hypothesis, we performthe following experiment on our ACL-ARC data set: fora given article, we replace each token in the paper, fromthe first token of the introduction to the last token of theconclusion, with the number of times it appears in theabstract. We average all articles by scaling them to thesame length and plot the picture obtained in Figure 1. In
this figure, the pixels are the averaged positions of tokens.
The darker they are, the higher number of times they appear in the abstract. It is obvious that the top and bottom
areas of Figure 1, which correspond to the introduction
and conclusion sections, are much darker.
However, the body of the article is longer than the introduction and the conclusion. It may be the case that thewords in the abstract also come from the body of the article. We calculate the ROUGE scores of different partsof articles using their abstracts as gold standards. Thesescores are shown in Table 1a. Introduction, body and conclusion all exhibit a high ROUGE-1 recall score, with thebody part getting a higher score. It means that most of
the words in an abstract are actually found in the body ofthe article. In other words, the abstract is already almost
entirely in the body of an article.

4Which words are relevant for anabstract?

Compared to recall scores, ROUGE-1 precision scores aremuch lower. This means that all parts contain more words
which are not in the abstract, than which are. Thesewords are irrelevant to generate the abstract. Neural networks will learn which words are more important and― 1282 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

Table 1: ROUGE scores of different sections of scientific papers and summaries generated from them. (a) and (b) arefor the input sections, raw or after pre-processing. (c) is for generated summaries. I, C and B stand for introduction,conclusion and body, respectively. Boldfaced numbers show best ROUGE scores for each unit (unigram (ROUGE-1),
bigram (ROUGE-2) or longest common subsequence (ROUGE-L)) and each measure (recall (R), precision (P) and F-score(F)).
(a) Raw sections.

I
B
C# of tokens
avg. ± std.dev.

RROUGE-1
PFRROUGE-2
PFR

471 ±
2656 ±
232 ±

74.38
82.43
56.96

19.77
6.22
34.04

29.67
10.43
38.71

31.52
34.36
19.46

8.04
2.17
12.14

12.22
3.81
13.59

56.59
67.18
43.55

2331363
164

ROUGE-L
PF
15.03
4.96
26.52

22.54
8.32
29.90

(b) Sections after pre-processing. Stop words and punctuation marks have been removed from introduction and conclusion. (I0, C0) Body is reduced to the more informative N-grams only. (B00)

I0
B00
C0

# of tokens
avg. ± std.dev.

R

ROUGE-1
PFR

ROUGE-2
PFR

295 ± 00146
31 ±24139 ±99

65.26
10.20
46.52

17.12
38.88
28.97

25.83
14.95
32.45

26.24
1.43
16.07

6.79
5.40
10.71

10.29
2.08
11.72

40.74
7.86
28.12

ROUGE-LPF
10.66
31.37
18.15

16.07
11.62
20.03

(c) Summaries generated from different combinations of sections after pre-processing.

Summary of I0
Summary of B00
Summary of C0
Summary of I0+B00
Summary of B00+C0
Summary of I0+C0
Summary of I0+B00+C0

R

ROUGE-1
PF

R

ROUGE-2
PF

R

40.38
28.59
39.22
41.56
38.76
43.44
43.51

44.77
31.03
40.10
43.98
41.37
45.83
45.33

40.50
28.56
38.22
41.10
38.43
42.83
42.71

10.82
5.43
9.80
11.43
9.92
12.73
12.72

12.17
5.87
10.03
12.06
10.53
13.37
13.29

10.84
5.42
9.55
11.27
9.78
12.51
12.48

31.22
22.72
31.00
32.25
30.55
33.54
33.58

which are irrelevant given enough training data. However, experimentally, RNNs perform poorly on longer inputs and practically OpenNMT-py imposes a limit of 400
tokens in length. Table 1a shows that the average lengthof each part is well over this limit. We propose a preprocessing to reduce length.
For the introduction and conclusion channels, we remove all stop words and punctuation marks. These tokens are not significant as they can be reproduced by theRNN model because they are remembered from the abstracts themselves. Table 1b shows that, after removing
stop words and punctuation marks, the ROUGE scores ofthe introduction and the conclusion channels decrease.
This means that information is now missing.
The body channel scores in Table 1a are taken for all
the text which extends from end of the introduction to thebeginning of the conclusion. The ROUGE-1 recall score ishigher in average than for the introduction of the conclusion. This means that these sections contain more words
which appear in the abstract. To solve the length problemROUGE-LPF
35.10
24.87
32.01
34.53
32.92
35.83
35.48

31.46
22.79
30.34
32.05
30.39
33.26
33.16

(average of 2,656 tokens), simply removing stop words
and punctuation marks is not enough. Instead, we perform keyword extraction using the Rake tool2 [11]. The
length of the entire body is reduced to a very small number of tokens (average 31). ROUGE recall scores decrease
but precision scores increase, which was expected.
After the pre-processing presented above, an article isreduced to three channels: introduction, body and conclusion. Their lengths are acceptable for OpenNMT-py,
even when concatenated. If too long, OpenNMT-py uses
only the first 400 tokens.
We randomly divide the ACL-ARC articles into a training set (14,000 articles), a validation set (500) and a testset (500). Different combinations of the three channels
are tested. The results are shown in Table 1c. The lengthof the summaries generated by different combinations ofchannels range from 110 tokens to 120 tokens, whichmatches well the average length of the golden standard
average length (113 tokens).

― 1283 ―2 https://github.com/zelandiya/RAKE-tutorial

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

5Analysis

Among summaries generated using only one channel, theintroduction channel has the highest ROUGE score. It iseasy to understand from Table 1b that this channel has
the highest recall score. This means that it has the largest
number of important words, and that it is the most informative one. For summaries generated using a combination of two channels, introduction and conclusion (I0+C0)
is the most informative one. Table 1b also shows thatthe length of I0+B00+C0is slightly longer than 400 in average. OpenNMT-py will drop the last tokens after 400.
Compared to I0+C0, in I0+B00+C0, some tokens in conclusion will be replaced by tokens from the body, but body
is not as informative as conclusion. This explains whysummaries generated using I0+B00+C0end up with a lower
ROUGE than summaries generated only from I0+C0.

6Conclusion

In this paper, we showed that, at least on the collection ofpapers from our data set, ACL-ARC, most of the words inthe abstract appear in the article. The conclusion is that,to generate the abstract of a scientific paper automatically,
the task consists of removing irrelevant words.
In order to solve the input length problem for a tool
like OpenNMT-py, we proposed a multi-channel method,where the channels are the introduction, the body and theconclusion of an article, on which a light pre-processing
has been performed.
We performed experiments on a set of articles fromthe natural language processing domain: ACL-ARC. We
showed that generating an abstract from the introductionand the conclusion only is better than relying on the entire
article. The introduction and the conclusion are the mostinformative parts of the article.

7

Acknowledgement

This work was supported by JSPS KAKENHI Grant
Number JP18K11446.

References[1] Sumit Chopra, Michael Auli, and Alexander M
Rush. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings ofthe 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 93–98,
2016.
[2] Dipanjan Das and André F. T. Martins. A survey
on automatic text summarization. Literature Survey
for the Language and Statistics II course at CMU,
4:192–195, 2007.

[3] Trieu H. Trinh, Andrew M. Dai, Lng Thng, andQuoc V. Le. Learning longer-term dependencies inRNNs with auxiliary losses. In International Conference in Machine Learning, February 2018.
[4] Guillaume Klein, Yoon Kim, Yuntian Deng, Jean
Senellart, and Alexander M. Rush. OpenNMT:Open-source toolkit for neural machine translation.
In Proc. ACL, 2017. doi: 10.18653/v1/P17-4012.
[5] Chin-Yew Lin. Rouge: A package for automatic
evaluation of summaries. In Stan Szpakowicz
Marie-Francine Moens, editor, Text Summarization
Branches Out: Proceedings of the ACL-04 Workshop, pages 74–81, Barcelona, Spain, July 2004.
Association for Computational Linguistics.
[6] Elena Lloret and Manuel Palomar. Text summarization in progress: a literature overview. Artificial intelligence review, 37(1):1–41, January 2012.
[7] Kenji Ono, Kazuo Sumita, and Seiji Miike. Abstract
generation based on rhetorical structure extraction.
In Proceedings of the 15th conference on Computational linguistics-Volume 1, pages 344–348. Association for Computational Linguistics, 1994.
[8] H P. Luhn. The automatic creation of literature abstracts. IBM Journal of Research and Development,2(2):157–165, 05 1958.
[9] Vahed Qazvinian and Dragomir R Radev. Scientificpaper summarization using citation summary networks. In Proceedings of the 22nd International
Conference on Computational Linguistics-Volume
1, pages 689–696. Association for Computational
Linguistics, 2008.
[10] Vahed Qazvinian, Dragomir R. Radev, Saif M.
Mohammad, Bonnie Dorr, David Zajic, Michael
Whidby, and Taesun Moon. Generating extractivesummaries of scientific paradigms. Journal of Artificial Intelligence Research, 46(1):165–201, 2013.
[11] S. Rose, D. Engel, N. Cramer, and W. Cowley. Text
Mining: Theory and Applications, chapter Automatic Keyword Extraction from Individual Documents, pages 1–20. John Wiley & Sons., 2010.
[12] Ilya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural networks.
In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 1017–1024, 2011.
[13] Simone Teufel and Marc Moens. Summarizing
scientific articles - experiments with relevance andrhetorical status. Computational Liguistics, 28(4):409–446, 2002.

― 1284 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.