言語処理学会 第26回年次大会 発表論文集 (2020年3月)

Data Augmentation Technique for Process Extraction in
Chemistry Publications
Yuni Susanti Hikaru Yokono Hiroaki Yoshida
Fujitsu Laboratories, Ltd.
{susanti.yuni, yokono.hikaru, yoshida.hiro-15}@fujitsu.com

1

2

Introduction

In the field of materials science and chemistry, deep
understanding of the relation between structure andproperties is required for creating new materials.
Academists and scientists often seek for that information in scientific publications, since experimental observations have been the primary means to
get the various chemical and physical properties ofmaterials. Another important information from experimental observations includes the process of conducting the experiment or manufacturing procedure,
because they can refer to that to conduct similar
experiments, or to understand how some materials
are made. Thus, access to a structured informationof experiment process or manufacturing procedure isneeded to improve the eﬃciency of their work.
The process extraction task aims to extract information such as materials, conditions, apparatus
and other process-specific information from text describing some process. This task suﬀers from insufficient number of labeled resource, and data annotation is much harder for domain-specific tasks such as
in chemistry or material science. One of the popular
techniques for increasing the size of labeled training
sets is data augmentation. In this paper, we explore
simple data augmentation technique focused on process extraction task in chemistry domain.
Previous work has proposed some techniques fordata augmentation in NLP. One popular study generated new data by backtranslation [7]. Other work
has used data noising as smoothing [6] and predictive language models for synonym replacement [3].
Easy Data Augmentation, or EDA [5], comprehensively explores simple text editing-based data augmentation techniques for text classification task andshowed a strong performance gain despite the simplicity of the technique. Inspired by their work,in this paper we explore simple data augmentation
technique for process extraction in chemistry domain without using any dictionary. We also explore
the possibility to create meaningful new data frompresent data by preserving the meaning of the original information of the present data.

Process Extraction

Process extraction task aims to extract entities froma process sentence and label them with processspecific entity-type labels such as operation/process,material, condition, apparatus, and so on. Theselabels define the roles of the entities in the processsentence. In this paper, we focus only on processsentence, which we define as a sentence containing
process predicates. The following shows an example
of a process sentence, consists of one process predicate biotinylated.
Antibodies were biotinylated using a 5fold molar excess of biotin-LC-NHS ester.
In this study, process predicate is defined as an
expression representing the operation of a process,e.g. the word biotinylated in the above example. We
assume S = P1 , P2 , ...Pn where n is the number ofprocess predicate P in sentence S.
Given a chunked text as the input, the process
extraction task is to identify all the entities and theirtypes/ roles in the process sentence. In this work, we
represent entity as a word or sequence of words; thetask is thus identifying the span of the entities. We
cast this problem as a sequence labeling task.

3

Data Augmentation

EDA [5] explores four simple text editing-based data
augmentation techniques for text classification task:
synonym replacement, random swap, random insertion, and random deletion. Those techniques can beeasily applied to process extraction task, but in our
experiment, it led to performance decrease. This result is expected, considering the diﬀerent nature of
the task. For instance, it is natural to think thatthe order of the words in text classification task
might not be as important as in a sequence labeling task such as process extraction. Thus, swapping
the words or randomly insert and delete the entities
could be the main reason of the decrease in the performance.
In this work, we introduce simple text editingbased data augmentation method for process extraction task. For this task, the general idea is replacing
an entity with entity having the same entity type

― 1041 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

to create a new augmented sentence. Therefore, we
do not have to worry about the entity label in the
new sentence since the entities are replaced with theentity having the same label (the original labels aremaintained).
Similar to EDA [5], our proposed method also
does not require a domain-specific dictionary or any
dictionary at all1 . Suppose a sentence has a setof entities and a pattern (Sentence = SentenceN E ,Sentencepattern ). The pattern here refers to thegrammatical structure of the sentence including theorder of words, the stop word choices, and the like.
The main idea of our proposed method is for a set ofentities in an input sentence (InputN E ), a new sentence using the entity set is created using a sentence
pattern from a source sentence (Sourcepattern ).
This new sentence, or the augmented sentence,
would have the entities from the input sentence andthe pattern from the source sentence (Augmented =InputN E , Sourcepattern ).
In the following, we explain in detail of our proposed data augmentation method. Given an input
sentence, the data augmentation method is twofold:
1. Source sentence selection: search for sentence in the training data to be the pattern for
the new sentence. The source sentence candidates are all sentences in the training data except the input sentence. To make k-new sentences, k-number of source sentences are selected.
2. Entity replacement: create a new sentence,
i.e. the augmented sentence, by substituting theentities in the source sentence having the same
entity types with the corresponding entities in
the input sentence.
In this paper, we introduce several methods onsource sentence selection.
Label-similarity method (LSIM). In the first
method, we simply choose the sentence having thebiggest number of label overlap with the input sentence as the source sentence. The idea is sentences having high label overlap should be similar in structure. For instance, the following input and source sentences have 100% label overlaps:
MATERIAL MATERIAL O PP O MAT-DESC MATERIAL .
Input sentence: Oxalic acid were dissolved
in deionized water.
Source sentence: Borac acid was added toboiling alcohol.
As we can see, both sentence indeed has the samestructure, and it would be safe to use the structure
1 Nonetheless, EDA [5] uses WordNet lexical dictionary tofind synonymsof the source sentence with the set of entities from input sentence, and vice versa. In our implementation,
we rank the source sentence candidates according totheir label overlap, and choose the k-highest scoring
sentences as the source sentences to make k-new augmented sentences. For above input-source sentences
pair, the augmented sentence would be
Augmented sentence: Oxalic acid wasadded to deonized water.
The augmented sentence uses the pattern fromsource sentence and entities from input sentence, e.g.
oxalic acid substitutes borac acid since they havethe same entity type, i.e. material. As we can see,the newly-created augmented sentence still somehow
makes sense grammatically even though the meaningslightly diﬀers.
Process-similarity method (PSIM). In thelabel-similarity method, the generated sentence often loses the meaning of the original sentence, orsometimes they form a non-sense new sentence. This
could be problematic since instead of giving a useful
information to the model, they can be a noise to thetraining data. To be able to preserve the meaningof the original sentence, we choose the sentence describing similar process as the source sentence in thissecond method of source sentence selection.
In a process sentence, to preserve the meaning of
the original sentence, the most important part is theprocess predicate. Therefore, we focused on the process predicate similarity to find similar process sentence as the source sentence. We used Word2vec
pretrained model2 to calculate the vector similarityof the process predicate of the input sentence andthe source sentence candidates. The sentence having
the highest process predicate similarity with those of
the input sentence is chosen as the source sentence.
Since one sentence could have more than one process predicates, we make pair combinations of process predicates from the sentence pair and calculate
the similarity between them. For example,
Input sentence: The pH of the mixed aqueous solution was adjusted to 6 with aqueous
ammonia (28%).
Source sentence candidate: The pH valueof this K2HPO4 was adjusted to 0.1 using
0.1 M phospate solutions.
For above example, all pairs of process predicatesare (mixed, adjusted), (mixed, using), (adjusted, adjusted), (adjusted, using). In our implementation,
the final similarity score for a candidate would be1) all pairs average (PSIM), which is the average
2 Available on https://github.com/olivettigroup/materialsword-embeddings― 1042 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

Table 1. Performance (F1 scores) of data augmentation methods on the process extraction model by varying
the training set fractions and the number of augmented sentences per original sentence (k). Bold indicates thebest score for each training set fraction; italic indicates decrease in performance over the original model.

fractionstraining set size
original (org) org+augmented

LSIM

PSIM

PSIM-A

189
379
568
758
947
1136
1326
1517
1706
1896

1004
1989
2963
4063
5027
6021
7056
8077
9121
10106

5
5
5
5
5
5
5
5
5
5

46.3
53.2
57.8
62.2
63.2
64.5
66.2
66.7
66.0
68.2

52.2
53.7
59.3
62.4
61.8
66.1
68.0
67.4
67.4
68.9

56.9
57.6
60.2
66.0
66.2
66.1
69.7
69.4
69.6
71.7

52.3
58.4
62.1
66.2
64.0
65.8
68.0
67.4
70.0
72.1

10%
40%
90%

189
758
1706

1493
6046
13570

8
8
8

46.3
62.2
66.0

55.9
62.6
67.9

57.2
67.5
69.4

57.9
67.3
70.1

10%
40%
90%

189
758
1706

2797
11334
25434

16
16
16

46.3
62.2
66.0

54.0
63.6
68.3

55.6
67.1
71.2

55.0
67.6
70.1

sentences test set) annotated by domain experts for
named entity recognition task such as identifying reaction conditions and materials [2, 4]. The dataset
includes the labeled entity mentions associated withentity types which specify a category/kind for theentity mention. The dataset is collected from paragraph describing inorganic material synthesis procedure selected from 2.5 million publications. There
are in total 21 entity types in the dataset, including: material, operation, amount-unit, conditionunit, material-descriptor, apparatus etc.

4.2

Augmented sentence: The pH value of thissolution was adjusted to 6 using 28% ammonia aqueous.
As we can see, the original meaning of the input sentence is kept in the augmented sentence to some extent. In addition, by keeping the original process
predicates (the process predicates of the source sentence), the augmented sentence could maintain thesentence structure of the source sentence.

Experimental Setup

To test the eﬀectiveness of the data augmentation,
we conducted experiments on process extraction task
model trained with and without data augmentation
method. The following describes the dataset and themodel used in the experiments.

4.1

original model

10%
20%
30%
40%
50%
60%
70%
80%
90%
100%

similarity scores of all pairs combined, or 2) aligned
pair average (PSIM-A), which is the average of thehighest scores for each of the input sentence’s process
predicate. For example, the highest score for mixed
would be either the similarity score of (mixed, adjusted) or (mixed, using), while for adjusted would
be either (adjusted, adjusted) or (adjusted, using).
In the entity replacement step, we keep the process
predicates of the source sentence in the augmented
sentence. The augmented sentence as the result of
the above input-source sentence pair would be

4kDatasetIn the experiment, we used a dataset of 235 synthesis procedures (1896 sentences in training set and 210

Process Extraction Model

We conducted the experiments for one of the state-ofthe-art models in sequential labeling task: Bidirectional LSTM network with CRF model (BI-LSTMCRF), which shows good performance for sequencetagging [1]. We run the model without data augmentation as the original model for experiment incurrent study.

5

Result and Discussion

We run both the training using only original data
and training with data augmentation for the following training set fractions (%): 10, 20, 30, 40, 50,
60, 70, 80, 90, 100. For all experiments, we used
the same hyperparameters settings. We generatek : 5, 8, 16-augmented sentences per each original
sentence. Table 1 shows the results (F1 scores, in%) of the process extraction model described in section 4.2 equipped with or without the data augmentation methods described in section 3.

― 1043 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

To sum up the results, almost all of the models
trained using data augmentation outperformed theoriginal model trained without data augmentation.
The improvement varies depending on the size of thedata used in the training, ranging from as small as
0.2 to 10.6 points improvements. The best F1 scoreswithout data augmentation, 68.2, was achieved using
100% of the training data. The model PSIM, which
was trained using data augmentation, surpassed thisnumber by achieving F1 score of 69.7 while only using
70% fraction of the training data.
Table 1 shows that the data augmentation has themost significant improvements (up to 10.6 points) for
training on the smaller dataset fractions (10%, 20%).
Nevertheless, the improvements were quite reasonable, up to 4 points, using the higher training set
fractions (90%, 100%). Since overfitting tends to be
more severe when training on small data, this result
is encouraging since it shows that the data augmentation could reduce the risk of overfitting, especiallyon smaller size of training set.
While the improvements across all data augmentation methods are more or less the same, PSIM and
PSIM-A data augmentation methods constantly contributed to the performance gain on the process extraction model on all training set fractions. LSIM
model gave performance improvement on all experiments except on experiment using 50% fraction of
the training data. It showed lower F1 scores than theoriginal model on that experiment. Since in all otherexperiments LSIM model showed improvements over
the original model, we considered this as a peculiarcase and more thorough investigation is needed tounderstand the cause of this case.
Does k aﬀect performance? We also conducted
experiments where we varied the number of augmented sentences, k, per each original input sentence(k: 5, 8, 16). The result is shown in the lower part ofTable 1. To summarize, the performance gains are
not significantly diﬀerent across diﬀerent numbers ofk in our experiments.
Adding pretrained model. EDA [5] stated thatit might not yield substantial improvements when using pretrained models, so we conducted experiments
where we added a pretrained Word2vec model. The
result is summarized in Table 2. It showed somesmall improvements (up to 4.2 points) on the PSIM
and PSIM-A, while we can notice some decrease inperformance on the LSIM model. We may consider
this decrease negligible since they are very small.

6

Conclusion

We have introduced simple data augmentation methods to be used on process extraction task. In thechemistry domain, process extraction task aims toTable 2. Performance (F1 scores) of the models after
adding a pretrained model.
% trainoriginal

LSIM

PSIM

PSIM-A

10%
20%
30%
90%
100%

57.7
65.7
67.9
73.0
73.4

60.7
64.7
66.4
72.6
73.6

61.6
65.8
68.7
73.5
74.1

61.9
65.9
68.4
73.0
73.6

extract entities from a process sentence and label
them with process-specific entity-type labels such as
operation/process predicate, material, condition, apparatus, and so on. This task suﬀers from insuﬃcient number of labeled resource, and data annotation is much harder for domain-specific tasks such as
in chemistry or material science.
In this paper, we have shown that simple data
augmentation method can boost performance on theprocess extraction task, which is a sequence tagging task. Our proposed method attempts to create meaningful augmented sentences by utilizing process information (the process predicate). We showedthat the proposed data augmentation substantially
improves performance, up to 10.6 points, and could
potentially reduce overfitting especially when training on small dataset. Future work includes exploring
more sophisticated data augmentation methods andevaluating on more general sequence labeling tasks.

References[1] Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional LSTMCRF models for sequence tagging. CoRR, abs/1508.01991,
2015.
[2] Edward Kim, Zach Jensen, Alexander van Grootel, Kevin
Huang, Matthew Staib, Sheshera Mysore, Haw-Shiuan
Chang, Emma Strubell, Andrew McCallum, Stefanie Jegelka,
and Elsa Olivetti. Inorganic materials synthesis planningwith literature-trained neural networks.
arXiv preprint
arXiv:1901.00032, 2018.
[3] Sosuke Kobayashi. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings ofthe 2018 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 452–
457, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
[4] Sheshera Mysore, Zachary Jensen, Edward Kim, Kevin
Huang, Haw-Shiuan Chang, Emma Strubell, Jeﬀrey Flanigan,
Andrew McCallum, and Elsa Olivetti. The materials science
procedural text corpus: Annotating materials synthesis procedures with shallow semantic structures. In Proceedings ofthe 13th Linguistic Annotation Workshop, pages 56–64, Florence, Italy, August 2019. Association for Computational Linguistics.
[5] Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance on text classification tasks.
In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing(EMNLP-IJCNLP), pages 6381–6387, Hong Kong, China,
November 2019. Association for Computational Linguistics.
[6] Ziang Xie, Sida I. Wang, Jiwei Li, Daniel Lévy, Aiming Nie,
Dan Jurafsky, and Andrew Y. Ng. Data noising as smoothingin neural network language models. CoRR, abs/1703.02573,
2017.
[7] Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao,
Kai Chen, Mohammad Norouzi, and Quoc V. Le. Qanet:
Combining local convolution with global self-attention forreading comprehension. CoRR, abs/1804.09541, 2018.

― 1044 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.