SUGISAKAMASANORI

Journal of Advances in Artificial Life Robotics
Vol. 1(2); September (2020), pp. 96–100
ON LINE ISSN 2435-8061; ISSN-L 2435-8061
https://alife-robotics.org/jallr.html

Research Article

Effectiveness of Data Augmentation in Pointer-Generator Model
Tomohito Ouchi, Masayoshi Tabuse

Graduate School of Life and Environmental Sciences, Kyoto Prefectural University, 1-5 Shimogamohangi-cho, Sakyo-ku, Kyoto 606-8522, Japan

ARTICLE INFO
Article History
Received 13 November 2019
Accepted 21 July 2020
Keywords
Automatic summarization
Data augmentation
Pointer-Generator Model

ABSTRACT

We propose a new data augmentation method in automatic summarization system, especially the
Pointer-Generator model. A large corpus is required to create an automatic summarization systemusing deep learning. However, in the field of natural language processing, especially in the fieldof automatic summarization, there are not many data sets that are sufficient to train automatic
summarization system. Therefore, we propose a new method of data augmentation. We use thePointer-Generator model. First, we determine the importance of each sentence in an article using
topic model. In order to augment the data, we remove the least important sentence from an input
article and use it as a new article. We examine the effectiveness of our proposed data augmentation
method in automatic summarization system.

© 2022 The Author. Published by Sugisaka Masanori at ALife Robotics Corporation Ltd.
This is an open access article distributed under the CC BY-NC 4.0 license(http://creativecommons.org/licenses/by-nc/4.0/).

nature of automatic summarization is that there is little
influence on the generated summary whether there is
redundant part or not. Further, in image processing, data
may be expanded by changing the background. This
assumes that the background has no direct impact on thetarget task. Therefore, we focused on change ofbackground which is one of data expansion of image
processing, and expanded data by deleting redundantsentences of original article in automatic summary.
Automatic summarization is classified into two
approaches. One is an extractive summarization in which
words, phrases or sentences of input articles are directly
extracted and combined to create a summary. The other isan abstractive summarization in which an input article istemporarily converted to an intermediate vector and then asummary is generated based on the intermediate vector. In
extractive summarization, there is a problem of how toconnect extracted words, phrases or sentences. And there
is also the possibility that in extractive summarization it isunknown what the directive word is pointing at. On theother hand, in abstractive summarization, since it isgenerated from an internal semantic representation, there

1. Introduction
In recent years, the amount of information on the net of
the world has increased exponentially, and it is expected to
become 44 ZB in 2020 [1]. Under the circumstance, thetechnique of automatic summarization becomes
indispensable for efficient selection of information.
However, in the field of automatic summarization, there
are not many data sets that are sufficient to train automatic
summarization system. Therefore, we consider whether
the technique of data augmentation commonly used inimage processing can be applied to natural language
processing.
Several studies have applied data augmentation tonatural language processing. For example, in documentclassification the data augmentation methods of the studies
are replacement with synonyms, replacement by
calculating similarity, replacement with antonyms,
replacement of words with words in sentences and random
deletion, etc. Also, in machine translation, there is the
Back Translation method [2]. However, none have yet
been adapted to an automatic summarization system. The
Corresponding author’s E-mail: t_ouchi@mei.kpu.ac.jp, tabuse@kpu.ac.jp

96

is a possibility that it becomes one integrated summary. It
also learns grammatical information, so it is easy toconnect words and phrases in contrast to an extractive
summarization. However, the abstract summarizationneeds to be learned with a large number of pairs of input
articles and generation summaries, which takes time andeffort to prepare the corpus.
In the previous research [3], we showed data
augmentation is effective in abstractive summary. But inthat research, using model is only a union of EncoderDecoder model and attention mechanism. Therefore, inthis study, we confirm the effect of data augmentation
using the latest model, the Pointer-Generator model [4].
The automatic summarization system used in this study is
described in Section 2. The method of data augmentation
used in this research is described in Section 3. Experimentand evaluation are described in Section 4. Discussion andsummary are given in Section 5.

× (1 − p gen )× p genp gen

Germany

emergeArgentina<start> Germany beatFig. 1. Pointer-Generator model [4].

3. Data augmentation Method
We measure the importance for each sentence of the
input article and use the article generated excluding thelowest important sentence as new data.
Refer to [5] for how to determine importance for eachsentence in the input article. First, the topic model iscreated from the training data set, and the sentence weightis determined for each topic of the input article. Then, thesum of sentence weights for all topics is taken as the final
weight of the sentence, and it was taken as the importanceof the sentence.

2. Pointer-Generator model
The pointer-Generator model [4] shown in Fig. 1
processes according to the following procedure.
1. Encode input text (Source Text) with bidirectional
LSTM. Convert variable length data to fixed length
vector. (Red part)
2. The word generation probability (Green part) is
obtained by using the attention mechanism (Bluepart) with the decoder LSTM. By performing thisprocess, the summary should be generated using theinformation of the entire input sentence as much as
possible.
3. Add the attention value in the encoding part to theword generation probability distribution in the
decoding part shown as the final distribution in Fig.
1 (copy mechanism). With the copy mechanism, it ispossible to generate words that are not included in the
output vocabulary of the model for inefficient
calculations, delete low-frequency words in advance.
4. Select the final generated word from final
distribution.
5. Repeat steps 2-4 to generate words sequentially. If
the token that represents the end of the sentence isselected, the decoding ends there.
Furthermore, Attention is attenuated where the value of
attention was larger than before (coverage mechanism).

Wt ,i

bt ,i =

Nibi = ∑ bt ,i

(1)
(2)

t

Firstly the weight bt ,i of the sentence i with respect tothe topic t is examined. In Equation (1), bt ,i multiplies the
sum Wt ,i of an importance of the words constituting thesentence i by the inverse of the square root of the total
number N i of words as a coefficient. This coefficient isfor weighting sentences that are not influenced by thesentence length. In Equation (2), we let the sum of bt ,i be
the final weight of sentence i.
4. Experiment and Evaluation

In this section, we evaluate the effectiveness of the data
augmentation proposed in Section 3.

97

4.1. Dataset

Table 1. the values of ROUGE when the maximum
number of words is 400 and 2,380

We used CNN/DailyMail dataset [6]. This dataset has
287,226
training pairs, 13,368 validation pairs and11,490 test pairs. In the experiment, 10,000 articles,
45,000 articles, 90,000 articles, 180,000 articles, and287,226 articles are prepared as training data.

ROUGE [7] is used as a general evaluation method of
automatic summarization, which is evaluated bymatching of reference summary and generated summary
word. We used ROUGE-1, ROUGE-2, ROUGE-L.
ROUGE-1 and ROUGE-2 evaluate unigram and bigram
coincidence, respectively. ROUGE-L evaluates matching
the longest common subsequence. Each ROUGE
evaluation method has F value, recall, and precision.
“Recall” means how many words of the generated
summary are included in the reference summary, and“precision” means how many words of the reference
summary are included in the generated summary. The F
value is obtained by Equation (3).

2 × Precision × Recall
Precision + Recall

ROUGE-1-f ROUGE-1-r ROUGE-1-p ROUGE-2-f ROUGE-2-r ROUGE-2-p
0.39350.43720.38000.17090.18910.16620.39580.41810.39940.17410.18320.1770400
2380

ROUGE-L-f ROUGE-L-r ROUGE-L-p
0.36160.40140.34930.36440.3846
0.3679

Table 1 shows when the upper limit of the number ofwords is increased from 400 to 2380, the value of ROUGE
increases slightly. In the following, the experiment isperformed with the upper limit of the number of words set
to 2,380.
In the research of [4], after learning with the copy
mechanism, it learns with the coverage mechanism.
Therefore, we use copy mechanism and coveragemechanism in this research. Table 2 shows how manytimes batch-sized articles are learned.

4.2. Evaluation method

F=

400
2380

Table 2. the number of iteration times each articles waslearned
10,000 articles
45,000 articles
90,000 articles
180,000 articles
287,226 articles(3)

iteration in copy mechanism iteration in coverage mechanism
20,000 times
300 times
80,000 times
1,000 times
160,000 times
2,000 times
320,000 times
4,000 times
460,000 times
6,000 times

The Pointer-Generator program used in this researchtakes about 0.27 seconds to learn 1 batch using GeForceGTX 1080 Ti GPU. Therefore, it takes about an hour anda half to learn 10,000 articles. Similarly, it takes about 6hours, 12 hours, 24 hours, 35 hours to learn 45,000 articles,
90,000 articles, 180,000 articles, 287,226 articles. Whileadding extended articles, the total epoch count is halved,
therefore the learning time is almost the same.

4.3. Parameter settings
In this research we use the program written by
PyTorch[8]. It has been verified that this program canachieve the same result as [4]. The hidden layer vector size
was set to 256 and the embedded vector size was set to 128.
The batch size was set to 8. In the original paper, the batch
size is 16, so double learning is required to learn the same
number of articles. The beam size was set to 4. The beam
search will be described later. The number of vocabulary
was set to 50,000. The learning rate was set to 0.15.
In this program, the number of words used to encode an
input article is limited to 400. However, in the case there
is no sentence with the lowest importance within 400
words from the beginning when we use the data
augmentation method, since the sentence is omitted there
is no effect on learning. Therefore, I found the article withthe most words among the articles used in the training data.
The number of words with the most words was 2,380. And
the upper limit of the number of words used in encoding
the input article was set to 2,380. Table 1 show the valuesof ROUGE when the maximum number of words is 400
and 2,380. In the Table 1, f, r, and p represent the F value,
recall, and precision, respectively.

4.4. Beam Search
In our research we use the beam search. The beam
search contrasts with the greedy method. This is because,
when generating a word, one word with the highest
generation probability is selected in the greedy method,while in the beam search, processing is performed while
holding the top K words. We make the final
summarizations by multiplying the probabilities of eachword generation, and make the highest one the final
summarization. In this experiment, this K value is set to 4.
4.5. Result
The ROUGE evaluation is shown below.

98not many articles, the data augmentation is effective due tolack of training data. Therefore, it can be said that thismethod is more effective when the number of articles issmall.

Table 3. Results on 10,000 articles. ‘original’ is original
data and ‘augmented’ is augmented data.
original
argmented

ROUGE-1-f ROUGE-1-r ROUGE-1-p ROUGE-2-f ROUGE-2-r ROUGE-2-p
0.3226
0.30630.36650.12090.11420.13890.33350.31810.37610.12750.12100.1452

original
argmented

ROUGE-L-f ROUGE-L-r ROUGE-L-p
0.29940.28410.34030.30850.29400.3482

Table 8. the value obtained by subtracting the F value of
Rouge of ‘original’ from that of ‘augmented’

Table 4. Results on 45,000 articles
original
argmented

ROUGE-1-f ROUGE-1-r ROUGE-1-p ROUGE-2-f ROUGE-2-r ROUGE-2-p
0.33980.33220.37310.13040.12690.14460.3506
0.3556
0.36980.13780.13930.1464

original
argmented

ROUGE-L-f ROUGE-L-r ROUGE-L-p
0.31550.3082
0.34680.32490.32930.3428ROUGE-1-f
ROUGE-2-f
ROUGE-L-f

5. Conclusion
In this experiment, it is found that the value of ROUGE
is improved by learning on adding an article from which
the sentence with the least importance is extracted, rather
than learning normally. Moreover, since this method has
the same number of learning times as the case where it is
not augmented, it succeeded in improving the value of
ROUGE without changing the learning time. In addition,compared to previous research [3], the overall value of
ROUGE was much better, and it turns out that it can also
be applied to a more accurate automatic summarization
system. Moreover, the experiment of this research showsthat the effect of the data augmentation was small under
the same conditions as the previous research [4]. However,it has also been found that reducing the number of articles
increases the effect of the data augmentation.
A future task is to increase the application range of thismethod. There are several models developed from the
Pointer-Generator model [9],[10]. I would like to examine
whether proposed method is effective. Also, in thisresearch, we used the topic model to determine theimportance of sentences, therefore, I would also like toexamine whether the data augmentation is more effective
when we use ROUGE to determine the importance ofsentences. Furthermore, in this research, the sentence withthe lowest importance was extracted, therefore, it ispossible to extract only the sentence with the highest
importance and make it an extended article. I would like toconsider such a new extension method as a future researchsubject.

Table 5. Results on 90,000 articles
original
argmented

ROUGE-1-f ROUGE-1-r ROUGE-1-p ROUGE-2-f ROUGE-2-r ROUGE-2-p
0.35380.35070.38300.14040.13870.1531
0.36270.35650.39450.14950.14670.1636

original
argmented

ROUGE-L-f ROUGE-L-r ROUGE-L-p
0.32880.32580.35620.33720.33130.3671

Table 6. Results on 180,000 articles
ROUGE-1-f ROUGE-1-r ROUGE-1-p ROUGE-2-f ROUGE-2-r ROUGE-2-p
original
0.37580.38840.38930.15870.16350.1655argmented 0.3827
0.40270.38740.16280.17110.1655original
argmented

ROUGE-L-f ROUGE-L-r ROUGE-L-p
0.34750.35880.36040.35150.36980.3561

Table 7. Results on 287,226 articles
ROUGE-1-f ROUGE-1-r ROUGE-1-p ROUGE-2-f ROUGE-2-r ROUGE-2-p
original
0.39180.41210.39580.17030.17840.1733argmented 0.3941
0.42880.38590.17210.18690.1693

original
argmented

10,000articles 45,000articles 90,000articles 180,000articles 287,226articles
0.01090.01080.00890.00690.00230.00660.00740.00910.00410.00180.00910.00940.00840.00400.0032

ROUGE-L-f ROUGE-L-r ROUGE-L-p
0.35930.3776
0.36330.36250.39410.35514.6. Discussion
Tables 3,4,5,6,7 show that the data augmentation iseffective in all articles. However, as the number of articles
increases, the data augmentation effect fades. Table 8shows the value obtained by subtracting the F value of
ROUGE of ‘original’ from that of ‘augmented’.
Since all the values in Table 8 are positive, it is better toextend in each the number of articles. Also, looking at
ROUGE-1, it gradually decreases as the number of articles
increases. Looking at ROUGE-2, it goes up to 90,000
articles and decreases from there. Looking at ROUGE-L,
it goes up to 45,000 articles and decreases from there.
The reason why the data augmentation effect is lesslikely to appear as the number of articles increases is asfollows. In a sufficient number of articles, learning
became saturated and the data augmentation is not very
effective. On the other hand, in the case where there wereReferences
1. Data Growth, Business Opportunities, and the IT
Imperatives
2. Sergey Edunov, Myle Ott, Michael Auli, David Grangier,“Understanding
Back-Translation
at
Scale”,
arXiv:1808.0938lv2[cs.CL] (2018).

99

3. T. Ouchi, M. Tabuse, “Effectiveness of Data Augmentationin Automatic Summarization System”, ICAROB2019, 177180 (2019)
4. Abigail See, Peter J. Liu, Christopher D. Manning, “Get To
The Point: Summarization with Pointer-Generator
Networks”, arXiv:1704.04368v2(2017).
5. H. Sigematsu, I. Kobayashi, “Generation of abstracts
considering importance of potential topics”, Proceedings of
the Annual Meeting of the Association for Natural
Language Processing (2012) (In Japanese).
6. DeepMind Q&A Dataset
7. Chin-Yew Lin, “ROUGE: A Package for Automatic
Evaluation of Summaries”, Proceeding of the Workshop on
Text Summarization Branches Out, Post-Conference
Workshop of ACL 2004, Barcelona, Spain (2004).
8. pointer_summarizer
9. Mike Lewis, Yinhan Liu, Naman Goyal, et al., “BART:
Denoising Sequence-to-Sequence Pre-training for Natural
Language Generation, Translation, and Comprehension”,
arXiv:1910.13461v1(2019).
10. Li Dong, Nan Yang, et al., “Unified Language Model Pretraining for Natural Language Understanding andGeneration”, arXiv:1905.03197v3(2019).
Authors introduction
Mr. Tomohito Ouchi
He received his Master's degree from
DepartmentofEnvironmental
Science, Graduate School of Life andEnvironmental
Sciences,
Kyoto
Prefectural University, Japan in 2019.
He is currently a Doctoral course
student
in
Kyoto
Prefectural
University, Japan.
Dr. Masayoshi Tabuse

He received his M.S. and Ph.D.
degrees from Kobe University in1985 and 1988 respectively. From
June 1992 to March 2003, he had
worked in Miyazaki University.
Since April 2003, he has been in
Kyoto Prefectural University. His
current research interests aremachine learning, computer
vision and natural language processing. IPSJ, IEICE and
RSJ member.

100