言語処理学会 第25回年次大会 発表論文集 (2019年3月)

Word Embeddings in place of Dictionary Lookup in thecontext of Academic Writing
Chooi Ling GOH
Yves LEPAGE
The University of Kitakyushu
Waseda University{goh@kitakyu-u.ac.jp, yves.lepage@waseda.jp}

1

Introduction

Researchers who are non-native speakers of English
always face some problems to compose a scientific article in this language. Most of the time, it is due tothe lack of vocabulary or knowledge of alternate waysof expression. They can use machine translation systems to translate from their mother tongues to English when writing English articles. However, sometimes the translation output is not correct, or does
not comply with the academic writing style. Some
researchers may also use a bilingual dictionary or athesaurus to search for suitable words.
Word embeddings have been applied to many natural language processing tasks such as information
retrieval, sentiment analysis, question answering anddocument classification. In this paper, we propose to
use word embedding models as an alternative to dictionary lexicon or term bank look-up. As opposed
to a thesaurus, which usually provides only semantically similar words or expressions, a word embedding
model may not only show semantically similar words
but also other words that have similar word vectors.
Hopefully, a word embedding model trained on a collection of academic articles should comply with theacademic writing style and contain terms which aresimilar in the domain considered. Such dictionary
lookup can be of help to non-native speakers of English to search for vocabularies that are suitable for
writing an article in style and in lexicon. For example, a lower level proficiency person may know
the easy word “but”, but word vectors may propose“however” or “although” as alternative expressions.
Following a trend in natural language processing(NLP), we first apply our methods to the NLP research field, i.e., we apply NLP methods on NLP
data. This has been called NLP4NLP [2]. We use
the ACL Anthology Reference Corpus1 (ACL-ARChereafter) as our NLP domain corpus. ACL Anthology is a digital archive of research papers in the premium conferences in NLP and the English language
quality of the papers is reputed. The goal of this paper is to run a preliminary experiment. We will use
1 https://acl-arc.comp.nus.edu.sg/

ACL-ARC to build a word embedding model, andcompare the word similarity results with some otherlarge pre-trained models.

2

Large Pre-trained Models

There exist three standard models for word embeddings at the moment2 : Word2vec [3], GloVe [5] andfastText [1].
The above word embedding models allow us tocompute the semantic similarity between two words,
or to find the most similar words given a target
word. The ability to obtain word vectors for out-ofvocabulary words is featured in fastText [1] by capturing the subword information. While Word2vec [3]is limited to a vector space locally, GloVe [5] considers also word co-occurrence globally.
We will use large pre-trained models available fromthe three methods above to compare with our wordembedding model trained on ACL-ARC.
• Word2vec3 :trainedonGoogleNews,
GoogleNews-vectors-negative300.bin.gz,
3billion tokens, 3 million word vectors.
• GloVe4 : trained on Wikipedia 2014 + Gigaword
5, glove.6B.zip, 6 billion tokens, 400K word vectors.
• fastText5 : trained on Wikipedia 2017 + UMBC
webbase corpus + statmt.org news dataset,wiki-news-300d-1M.vec.zip [4], 16 billion tokens,
1 million word vectors.
GoogleNews model contains compound words(e.g. “ANTARA News PRNewswire AsiaNet” and“eerily similar”), whereas in other models, no compound word is to be found. Besides, GoogleNews andfastText models have more uncleaned texts, such as
erroneous spelling, than other models (e.g. “baed”,“similiar”, “infomation”, see Table 4).
2 We leave aside the more recent ELMo [6] which is basedon deep context.
3 https://code.google.com/archive/p/word2vec/
4 https://nlp.stanford.edu/projects/glove/
5 https://fasttext.cc/docs/en/english-vectors.html― 1161 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

3

Specific Word Embedding
Model Trained on ACL-ARC

We use ACL-ARC to build our word embedding
model. ACL-ARC is a subset of ACL Anthology6 .
The corpus consists of publications about computational linguistics and natural language processing
from selected conferences and journals since 1979 until 2015. It consists of 22,878 articles.
We use gensim7 implementation of Word2Vec tobuild our model. The parameter settings are as follows.
• Dimensionality of the word vectors (size=300)• Distance between the current word with the predicted word (window=5)• Minimumcount(min count=5)

ofwordoccurrence

As pre-processing, we extract the texts from theXML output generated by the commercial opticalcharacter recognition (OCR) software, Nuance Omnipage. The front pages from the conferences areexcluded. There exists some noise or uncleaned text.
We did not care about it, and just used them as it is.
Most of the noise is coming from conference names,
mathematical equations and references. All the text
is lower-cased, and words containing numbers, symbols or punctuations are removed. Table 1 showssome statistics on the corpus used for building ourword embedding model. From 88 million tokens, webuilt a model containing 77K word vectors.
# of articles used
# of tokens# of distinct word
# of word vectors

21,63688,006,598
578,960
77,311

Table 1: Some statistics on the word embeddingmodel built on ACL-ARC.

4Experiments

We choose 60 highly frequent words from ACL-ARC,and extract similar words using the four models presented above. From the highest 200 frequent words,
we omit some functional words like “the”, “of”,“and”, single character words like “a”, “x”, “e”, andwords that are too specific for the NLP field like “semantic”, “dependency”, “discourse”. We further filter words that do not look like having more choices
6 https://aclanthology.coli.uni-saarland.de/7 https://radimrehurek.com/gensim/

with, by, which, model, each, data, system,using, used, information, features, results,
also, corpus, text, different, some, based, approach, work, given, english, algorithm, evaluation, most, method, performance, new, machine, parsing, however, structure, methods,
paper, knowledge, research, processing, possible, while, following, phrase, because, problem, since, experiments, annotation, many, accuracy, form, well, see, very, similar, classification, human, thus, process, best, score, shows
Figure 1: 60 highly frequent words selected from
ACL-ARC used for evaluation.

of alternative expressions. Figure 1 shows the 60
base words left finally. For each word, we extract
the 10 nearest neighbor words based on cosine similarity from each word embedding model (see Sections
2 and 3).

4.1

Evaluation Guidelines

For each base word, if a proposed word could be
used to replace the original word, by any form ofrephrasing, then it is considered as a possible substitute (1 point), or else it is not (0 point)8 . We counthow many possible substitutes have been proposedby each model.

4.2

Human JudgementWe asked three master second year students (S1, S2,S3) to evaluate the results. These students are nonnative speakers of English, but they have some experience in writing at least one international English
research article. Their English proficiency levels asof TOEIC (Test of English for International Communication) are shown in Table 2.

Education level
English education (yrs)
TOEIC level# of scientific English
papers published

S1

S2

S3

M2
11625
1

M2
11450
2M2
14430
1Table 2: Information about the evaluators.
stands for master’s student 2nd year)(M28 For GoogleNews and fastText pre-trained models, thesame possible substitute may be proposed several times with
only differences in case (lower/upper), e.g. “show” and“Show”. In this case, we count 0.5 point. The Word2vec
model trained on ACL-ARC and the large pre-trained GloVe
model lowercase the text before training.

― 1162 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

5Results

Table 3 shows the result of the manual evaluationby the three evaluators. In average, each word has
about two possible substitute proposals. Although
the ACL-ACR model is much smaller than the large
pre-trained models, it gives comparative results forfinding similar words.
Model

S1S2

S3

Avg

Word2vec(ACL-ARC)

2.07

1.53

1.92

1.84

Word2vec(GoogleNews)

1.58(1.88)

1.72(2.01)

2.22(2.52)

1.84(2.14)

GloVe
1.63(wiki+Gigaword)

2.38

2.55

2.19

fastText (wikinews)

1.97(2.16)

2.23(2.48)

2.02(2.26)

1.87(2.14)

Table 3: Evaluation results by three evaluators.
Brackets show the results where a same word withdifferent case is counted as 0.5 point.
Table 4 shows some examples of proposed similarwords. Double underline shows mutual agreements
among the three evaluators. Single underline shows
words selected by at least one judge. There is notmuch mutual agreements among the evaluators.

6Discussion

Based on the evaluation in Table 3, ACL-ARC provides a slightly lower number of possible substitutes
compared to other models. However, it exhibits alarger variety of proposals which conform to the academic writing style. From the authors’ point of view,
the proposals for “using” and “used” are almost perfect for the ACL-ARC model if the writers are able
to rephrase the sentence properly. However, due tothe low proficiency level of English of the evaluatorsin this experiment, they do not have any idea on how
to use the words. Since our purpose is to help thewriters to compose an article, in the case where thewriters do not have the ability to use the proposed
substitutes, we need to look for other ways to help
them. This experiment was useful to help us in thedesign of a writing aid tool: we understood that justproposing a list of possible substitutes is not enough,we also need to provide writers with usage samplesof the possible substitutes.
As usually observed with human evaluators, there
exists some inconsistency, where evaluators have chosen a word in a model, but did not choose it inan other model. For example, from the base word“shows”, the word “show” should be mutually agreed
upon, but one of the evaluators did not choose it in
the ACL-ARC model. The same goes for “similar”,
where “identical” should be chosen from the GoogleNews model as well.
We also spot another problem: the different formsof spelling, especially between American and British
English. For example, evaluators have chosen “utilized” but not “utilised”.
As shown in Table 4, GoogleNews and fastText
models exhibit a lot of erroneous words. In reality,
we should not propose this kind of substitutes to thewriters as they may not know that these words areincorrect and they may misuse them. For example,
one of the evaluators has mistakenly chosen “infromation” although it is a mis-spelled word.

7Conclusion

The purpose of this paper was to survey on various word embeddings models, in order to look foralternative expressions for a certain word in place ofdictionary lookup in the context of academic writing. As a preliminary experiment, we focused here
on proposing words for writing articles in the natural language processing field, by using the ACL-ARC
as our search corpus. Compared to large pre-trainedmodels, a specific model proposed more words which
conform with the academic writing style.
We conclude that word embeddings are useful forsuggesting substitute words for writing academic articles. They can help to transform a low level proficiency writing into academic style writing, especiallyfor non-native speakers of English. In the future,
we will explore into suggesting different expressions,
not only at word level, but also at phrase, sentenceor even paragraph level.

Acknowledgment
This work was supported by JSPS KAKENHI Grant
Number JP18K11446.

References[1] P. Bojanowski, E. Grave, A. Joulin, andT. Mikolov. Enriching word vectors with subword
information. Transactions of the Association for
Computational Linguistics – Volume 5, Issue 1,pages 135–146, 2017.
[2] G. Francopoulo, J.-J. Mariani, and P. Paroubek.
NLP4NLP: Applying NLP to scientific corpora
about written and spoken language processing.
In Proceedings of the First Workshop on Mining Scientific Papers: CLBib@ISSI, pages 5–11,
2015.

― 1163 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

base word

Word2vec (ACL-ARC)

Word2vecNews)(Google-GloVe (wiki+Gigaword)

fastText (wiki-news)

using

employing,utilizing,applying, uses, via,
combining, use, employs, used, utilizes

use, utilizing, Using,used, uses, incorporating, applying, Use, employing, Utilize

used, use, uses, instead,or, types, allows, can,directly, intended

utilizing,
employing,utilising,
applying,
incorporating, use, constructing,combining,
creating, substitutingusedutilized,
employed,
exploited,applied,adopted, leveraged, designed, utilised, reused,
usefulutilized, using, use,
uses, intended, Used,
Using, misused, designed, beusedusing, use, uses, or,types, instead, example,
such, as, similarutilized,employed,utilised, uses, designed,applied, relied, referred,devised, allowed

basedrelies, relying, basis,rely, relied, depends,depending,focuses,depend, focusing

Based,
headquartered, baed, headquarted, basing, basedin,bsed, Basing, basd,
ANTARA News PRNewswire AsiaNetcompany, group, new,its, which, firm, business, also, part, researchbasing, Based, predicated, relying, relies, derived, centered, rely, focusing, premised

however

although,moreover,furthermore,
indeed,nevertheless,unfortunately,but,though,because,nonethelessthough,although,nevertheless, nonetheless, that, meanwhile,also, but, only, notalthough, though, as,both, this, but, be, also,latter, .

although,though,nevertheless,but,therefore, nonetheless,indeed, unfortunately,consequently, yetsimilaranalogous,
identical,
dissimilar, close, competitive,
comparable,
related,divergent,promising, complementary

similiar,strikingly similar, Similar,
identical,
dissimilar,
eerily similar,
virtually identical, different,
simliar, same

example,instance,
such, same, this, unusual, which, although,particular, common

similiar,comparable,
identical,
analogous,
dissimilar, same, different, related, akin,simlarshows

illustrates,
reveals,demonstrates, showing,
depicts,
indicates,
displays,suggests,
show, summarizes

show, shown, showed,
showing,
indicates,
reveals, demonstrates,suggests,
illustrates,
Shows

show, shown, picture,feature, showing, seen,
appearing,featured,appeared, thisshow,
showing,
indicates,
shown,
demonstrates, showed,
illustrates,
reveals,
displays, Shows

information

clues, meta-information,knowledge, meta-data,shifters,
metadata,worry, cues, generalisations, exclusivity

info, infomation, infor mation,
informaiton, informa tion,informationon, informationabout, Information,
informaion, details

source, data, sources,
provided,
documents,
search, web, provide,knowledge, intelligence

infomation, informaton,
info, informtion, informatin, infromation,
information-,
informaion,
inforamtion,
infomration

Table 4: Examples of proposed similar words sorted by descending order of cosine similarity. Double underline
shows mutual agreements among the three evaluators. Single underline shows words selected by at least one
judge.

[3] T. Mikolov, K. Chen, G. S. Corrado, and J. Dean.
Efficient estimation of word representations invector space. In Proceedings of Workshop atICLR, 2013.
[4] T. Mikolov, E. Grave, P. Bojanowski, C. Puhrsch,
and A. Joulin. Advances in pre-training distributed word representations. In Proceedings ofLREC, 2018.

[5] J. Pennington, R. Socher, and C. D. Manning.
GloVe: Global vectors for word representation. In
Proceedings of EMNLP, pages 1532–1543, 2014.
[6] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep
contextualized word representations. In Proceedings of NAACL-HLT, pages 2227–2237, June2018.

― 1164 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.