VCDM: Leveraging Variational Bi-encoding and Deep Contextualized
Word Representations for Improved Definition Modeling
Machel Reid

Edison Marrese-Taylor
Yutaka Matsuo
Graduate School of Engineering
The University of Tokyo
machelreid2004@gmail.com{emarrese, matsuo}@weblab.t.u-tokyo.ac.jp
Abstract

word embedding or other representations of context. Current approaches for this task (Noraset
et al., 2017; Gadetsky et al., 2018; Ni and Wang,2017; Ishiwatari et al., 2019) are mainly encoderdecoder based, in which one encodes a contextual
representation for a word/phrase w using a varietyof features such as context or character composition, and uses the contextual representation(s) togenerate the definition d.

In this paper we tackle the task of definition
modeling, where the goal is to learn to generate definitions of words and phrases. Existing approaches for this task are discriminative,
combining distributional and lexical semantics
in an implicit rather than direct way. To tackle
this issue we propose a generative model forthe task, introducing a continuous latent variable to explicitly model the underlying relationship between a phrase used within a context and its definition. We rely on variational
inference for estimation and leverage contextualized word embeddings for improved performance. Our approach is evaluated on four
existing challenging benchmarks with the addition of two new datasets, C AMBRIDGE andthe first non-English corpus ROBERT, which
we release to complement our empirical study.
Our Variational Contextual Definition Modeler (VCDM) achieves state-of-the-art performance in terms of automatic and human evaluation metrics, demonstrating the effectivenessof our approach.1

1

Introduction

In most current NLP tasks, fixed-length vector representations of words, word embeddings, are used
to represent some form of the meaning of the word.
In the case of humans, however, oftentimes we will
use a sequence of words known as a definition —astatement of the meaning for a term— to express
meanings of terms (words, phrases, or symbols). It
is with this in mind that the question of “Can machines define?” is aimed to be answered with thetask of definition modeling (Noraset et al., 2017).
Definition modeling can be framed as a task of
conditional generation, in which the definition dof the word or phrase is generated given a conditioning variable w such as a word’s associated1
We release the code at: https://github.com/
machelreid/vcdm

Despite the relative success of existing approaches
for definition modelling, their discriminative nature —where distributional-derived information isat one end of the model and lexical information
is at the other— limits their power as the underlying semantic representations of the distributional
and lexical information are learned in an implicit
rather than direct way. For example, although Ishiwatari et al. (2019) successfully showed that both
local and global contexts are useful to disambiguatemeanings of phrases in certain cases, their approachheavily relies on an attention mechanism to identify
semantic alignments between the input phrase andthe output definition, which may introduce noise
and ultimately be insufficient to capture the entire
meaning of each phrase-definition pair.
To tackle this issue, we propose to explicitly model
the underlying semantics of phrase-definition pairs
by introducing a continuous latent variable z overa definition space, which is used in conjunction
with w to guide the generation of definition d. The
introduction of this latent representation enables us
to treat it as a global defining signal during the generation process, complementing existing alignment
mechanisms such as the attention.
Although the latent definition variable enables us to
explicitly model underlying semantics of contextdefinition pairs, the incorporation of it into the task
renders the posterior intractable. In this paper werecur to variational inference to estimate this in-

6331
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6331–6344,
November 16–20, 2020. c 2020 Association for Computational Linguistics

tractable posterior, effectively making our model a
Conditional Variational Autoencoder and evolving
the generation process from p(d|w) to p(d|w, z).
We also note that existing approaches for definition modelling heavily rely on word embeddings,
which due to their fixed nature can only capture
so much of the semantics, being known to offer
limited capabilities when dealing with polysemy.
Considering the success of pretrained deep contextualized word representations which by specificallyaddressing these limitations have been shown to
improve performance on a variety of downstream
NLP tasks (Peters et al., 2018; Devlin et al., 2018),
in this paper we propose a mechanism to integrate
deep contextualized word representations in thedefinition modelling task. Specifically, we successfully leverage BERT (Devlin et al., 2018) as ourcontextual encoder and our definition encoder toproduce representations for w and d respectively.
Finally, we develop two new datasets for this task,
one derived from the Cambridge Dictionary 2 , andthe other derived from Le Petit Robert3 . In summary, our contributions are:
• Model: We propose a novel approach for thetask of definition modeling, leveraging deep
contextualized word representation and thevariational encoder-decoder architecture. We
achieve new state-of-the-art performance on
the definition modeling task, outperforming
the previous state-of-the-art by as much as 9
BLEU points on the OXFORD dataset and 22
BLEU points on the ROBERT dataset.
• Datasets: We develop two new datasets C AM BRIDGE and ROBERT for this task. WithROBERT, a French dataset, being the first nonEnglish dataset developed for this task.
Datasets and pre-trained models will be publicly
released to the greater NLP community to help
facilitate further advances on this task upon acceptance of this paper.

2

Related Work

Our work is related to the seminal paper by Hillet al. (2016), who proposed using the definitions
found in everyday dictionaries as a means of bridging existing gaps between lexical and phrasal semantics. Effectively, they train a language model
2
3

https://dictionary.cambridge.org/
https://dictionnaire.lerobert.com/

to map dictionary definitions to lexical representations of words, presenting the task of reverse
dictionaries, where the goal is to return the nameof a concept given a definition.
Noraset et al. (2017) later introduced the task ofdefinition modeling, in which a model is tasked
with generating a definition for a given word, given
its respective embedding. The authors argued that,compared to other related tasks such as word similarity or analogical relatedness, definition generation can be considered a more transparent view of
the information captured by an embedding. However, this method does not incorporate contextual
information, preventing it from generating appropriate definitions for polysemic words. Addresing
this, Gadetsky et al. (2018) studied the problem ofpolysemy in definition modeling, introducing anattention-based model which uses contextual information determine components in the embedding
which may refer to a relevant word meaning.
Ni and Wang (2017) explore a different but related
problem, proposing an approach for automaticallyexplaining non-standard English expressions (i.e.
slang) in a given sentence. They present a hybrid word-character sequence-to-sequence modelthat directly explains unseen non-standard expression, garnering reasonable definitions of expressions given their context.
More recently, Ishiwatari et al. (2019) have tackled some of the limitations of previous works on
definition modelling and non-standard English expression explanation. Concretely, they note thatwhenever it is not possible to figure out the meaning of a given expression from its immediate local
context, it is common to consult dictionaries fordefinitions or search documents or the web to findother global context to help in interpretation. In
light of this, they introduce the task of describing
a given phrase in natural language, based on its
local and global contexts. To tackle this the authors
introduce a model which consists of two context
encoders (one for the local context, and one forthe global context) as well as a description decoder.
Our proposed model, uses a more practical variational encoder-decoder framework, allowing us totake advantage of explicitly modeling the phrasedefinition relationship, while also leveraging deep
contextualized word representations for more informative context representations.

6332

Finally, our model is also related to Sohn et al.
(2015), in which Conditional Variational Autoencoders (CVAEs) —an extension of the original variational autoencoder (VAE) (Kingma and Welling,2014)— were proposed for generating diverse
structured output, mainly in the context of image
generation, and visual object segmentation and labeling. Our work is also related to CVAE models
that have been developed for the domain of natural language processing, specifically Zhang et al.
(2016) who proposed a CVAE in the context of
Neural Machine Translation (NMT). As the usage
of VAEs has become relatively common, we will
omit a detailed explanation of these models, referring readers to Kingma and Welling (2014).

3

Proposed Approach

In a way analogous to previous work (Noraset et al.,2017), our proposed approach is a generative probabilistic model for word definitions, in which thegoal is to estimate the probability of generating
a definition d, given an input w. Concretely, we
propose to directly capture the joint semantics ofthe (w, d) pairs by introducing a latent variable
z to model the underlying definition space. Our
proposed generative process can be formulated asfollows:ZZp(d|w) = p(d, z|w)dz = p(d|z, w)p(z|w)dzz

and d respectively, and utilize neural networks toestimate the following components.
• q(z) ≈ qφ (z|w, d) is our variational approximation for the intractable posterior (the recognition network), which we model with a neural network with parameters φ. This makesqφ (z|w, d) a neural definition inferer.
• p(z|w) ≈ pθ (z|w) is a (conditional) prior
network, parameterized by θ, which in ourcase can be regarded as a neural definition
prior.
• p(d|w, z) ≈ pθ (d|w, z) is a generation network, parameterized by θ which acts as a variational definition modeler.
In the following subsections we give details on how
we specifically model each one of these components. With this in mind, we develop the following
architecture comprised of 3 major components:• Encoders - This component is comprised of
two encoders - one context encoder to produce a representation for w and another definition encoder to produce a representation ford (Section 3.1).
• Neural Definition Inferer - This component
infers the latent representation z from the representation of the word/phrase —the explicitly
modeled prior pθ (z|w)— and in conjunction
with the definition (the approximated posterior qφ (z|d, w) ) (Section 3.2).

z(1)
where the conditional probability p(d|w) evolves
into p(d|w, z), and the generation of the definitiond is now conditioned on both the input variable w
and our introduced continuous latent variable z.
Since the introduction of our latent variable makes
posterior inference intractable, in this paper we resort to variational inference to perform posterior approximation. Effectively, this makes our proposed
generative model a CVAE (Sohn et al., 2015) suchthat the variational lower bound can be formulated
as follows:
log p(d|w) ≥ Ez∼q(z) log p(d|w, z)− DKL q(z)||p(z|w) (2)where q(z) is the introduced variational approximation to the intractable posterior p(z|w, d) andp(z|w) is the prior distribution. Following previous
work (Sohn et al., 2015; Zhang et al., 2016) we letz, w and d be random vectors associated to z, w

• Variational Definiton Modeler - This component can be viewed as a decoder whichtakes in latent representation z to guide thegeneration of the target sentence, essentiallypθ (d|w, z) (Section 3.3).
Notation For clarity when explaining the approach, we define the notation conventions we will
follow, namely: d refers to dimensions, c refers
to the context vectors produced by the attention
mechanism, g refers to projections/activations, andh refers to sets of vectors.
3.1
3.1.1

Encoders
Context Encoder

To encode the sequence in which the word in question is used, we adopt the BERT (Devlin et al.,2018) architecture, which is comprised of multiple
Transformer (Vaswani et al., 2017) encoder layers

6333

pretrained on a masked-language modeling task to
encode deep contextual word representations for agiven sequence. BERT has also shown to be able
to model the relationship between two tasks onpair-wise natural language understanding tasks. It
is to this end that we propose the construction ofphrase-context pairs to leverage this property inthe context encoding process.

the case that wt is split into multiple subword tokens by the BERT tokenizer, we set the word representation to be the mean of each of its subword
representations. Namely, in the case that wt is comprised of the nth to the mth subtokens,

Inspired by context-gloss pairs (Huang et al., 2019)
for the task of Word-Sense Disambiguation (WSD),
we construct phrase-context pairs for our task ofdefinition modeling. Often, there are differences in
the word or phrase that we aim to define, and thelexeme form that is used in the context sentence.
For example, the lemma run has the following
forms: run, runs, ran and running, which all represent the same lexeme. To account for these discrepancies between the lemma and the lexeme form,
we construct the aforementioned phrase-context
pairs, which are constructed by simply insertinga separator token, denoted as [SEP], between theword/phrase and the context sentence. Below weshow how this process would work for an example
taken from Cambridge dictionary dataset for thelemma leave:

3.1.2 Definition Encoder
The definition encoder, which we denote as Td , isalso initialized with BERT. This encoder takes inthe definition sequence d = [d0 , d1 , . . . , dMe ] as
input and represents d as:

He left a wife and two children.
,→ leave [SEP] He left a wife and two children.
This form of construction for the phrase-context
pairs comes with the added benefit of querying asentence for a definition by simply prepending theword/phrase and a seperator token to the context
sequence. As we use BERT as our encoder, we areable to leverage its self attentive nature to produce
a representation of the word or phrase in question
with respect to the context sentence.
As we initialize our context encoder with
BERT, the phrase-context pair sequencec=[wt , [SEP], c2 , . . . cMc ], containing
word or phrase wt , is prepended by a [CLS]
token and is appended by a [SEP] token, making
c0 = [CLS] and cMc = [SEP].
We define this context encoder as Tc , which takes
in the context sequence c, and returns a sequenceof annotation vectors for each token in c. We
denote these annotation vectors as as hc , where(i)dcc{hc }Mi=0 ∈ R and
rwt = Tc (c)[t](3)

is the tth representation in hc , representing wt . In

rwt =

1 Pm
Tc (c)[i]m − n i=n

rd = Td (d)[0]

(4)

(5)

where rd ∈ Rde . We take the representation (corresponding to the preprended [CLS] token) as arepresentation for the entire definition sequence.
3.2

Neural Definition Inferer

We formulate the posterior distribution qφ (z|d, w)
and prior distribution pθ (z|w) as multivariate Gaussians with a diagonal covariance matrices. To
model these distributions we make use of neuralnetworks, following Zhang et al. (2016).
3.2.1 Neural Definition Posterior
As modeling the true posterior p(z|d, w) is generally intractable, to approximate this true posterior,
we use a variational distribution, formulated as thefollowing multivariate Gaussian:qφ (z|d, w) = N (z; µ(d, w), σ(d, w)2 I)(6)which is parameterized by the mean and µ(d, w)
standard deviation σ(d, w), both which are treatedas functions of definition d and phrase w parameterized by neural networks.
From the neural encoding mechanisms, we gather
the definition representation rd , and the context
representation rwt . We then concatenate rd andrwt and project the resulting vector onto our latent
space, setting hz = g(Wz [rwt ; rd ] + bz ), whereWz ∈ Rdz ×(de +dc ) is a trainable weight matrix,bz ∈ Rdz is a trainable bias vector and g(·) represents a non-linearity activation. In our experiments
we set g(·) to be the tanh(·) activation function,
following previous work.
To attain the aforementioned mean and variancevectors parameterizing the variational distribution,

6334

setting µ = Wµ hz +bµ and log σ 2 = Wσ hz +bσ ,where Wµ , Wσ ∈ Rdz ×dz are trainable weight matrices parameterizing the projection and bµ , bσ ∈Rdz are bias vectors.
In order to make the parameters θ differentiable forgradient descent optimization, we use the “reparameterization trick” (Kingma and Welling, 2014)setting z = µ + σ · e, where e ∼ N (0, I) is anoise variable sampled from a multivariate Gaussian distribution to derive our latent vector z.

these components in the case that the previous hidden state and/or generated token does not provide
enough information or misleads the accurate generation of the next token. We refer to this modified
cell as the Variational Contextual Definition Modeler (VCDM) Cell, and the resulting decoder asa VCDM-RNN. The VCDM Cell calculates the
decoder hidden state sj as follows4 :

3.2.2 Neural Definition Prior
Our prior is a conditional distribution formulated
as the following multivariate Gaussian:002pθ (z|w) = N (z; µ (w), σ (w) I)(7)

which is parameterized by µ0 (·), and σ 0 (·) which
are both solely functions of phrase w. In a similar
fashion to the neural definition posterior, we makeuse of a linear projection to project rwt to the mean
vector µ0 and another linear projection to derive thelog variance vector. During inference (at test time)
when sampling from pθ (z|w), we set our latentvector z to be the mean vector µ0 .
To initialize the decoding procedure detailed in
the next subsection, we feed the latent representation z and project it to the decoding space setting
h0d = g(Wd z + bd ), where Wd ∈ Rdd ×dz andbd ∈ Rd d .
3.3

ij = σ(W Edj + U sj−1 + Acj−1 + V h0d )(10)

fj = σ(Wf Edj + Uf sj−1 + Af cj−1 + Vf h0d )oj = σ(Wo Edj + Uo sj−1 + Ao cj−1 + Vo h0d )C̃j = g(Wg Edj + Ug sj−1 + Ag cj−1 + Vg h0d )

(11)
(12)

Cj = σ(fj · Cj−1 + ij · C̃j )(14)

sj = g(Cj ) · oj

(15)

where Edj ∈ Rdw is the embedding forthe target word, W ,Wf ,Wo ,Wg ∈ Rdd ×dw ,U ,Uf ,Uo ,Ug ∈ Rdd ×dd , A,Af ,Ao ,Ag ∈ Rdd ×dd ,and V ,Vf ,Vo ,Vg ∈ Rdd ×dd are trainable weightmatrices parameterizing the RNN cell.
Additionally, at each decoder timestep j we attend
to the set of annotation vectors hc produced by
the last layer of the context encoder. To computecontext vector cj , we use general attention (Luonget al., 2015) shown below:(i)i=1 αi hc(i)
αi = softmax(s>j Wa hc )cj =Variational Definition Modeler

Given phrase w and latent representation z, theprocess of definition modeling can be formulated
as the following conditional language model:
Q dp(d|w, z) = M(8)
j=1 p(dj |d<j , z, w)
p(dj |d<j , z, w) = gd (sj , cj )

(9)

(13)

PT

(16)
(17)

where Wa ∈ Rdd ×dc , and αi can be viewed as analignment over hc and cj as a vector capturing theencoder hidden states scaled by this alignment.
3.4

Optimization challenges

During generation of the definition sequence, wewant the decoder to rely on all of the encoded components at each timestep. We modify the LSTM
Cell (Hochreiter and Schmidhuber, 1997) to encompass previous context vector cj−1 , and the projected latent definition representation h0d .

Despite the VAE’s appeal as a tool to learn unsupervised representations through the use of latent variables, these models are often found to ignore latentvariables when using powerful generators. To overcome this issue of “posterior collapse” (Bowman
et al., 2016), we incorporate the following heuristics: (1) annealing the KL term from 0 to 1 using
a sigmoid annealing schedule, following Bowman
et al. (2016) and (2) thresholding the KL term in
the objective function with a constant λ using the“free bits” technique (Kingma et al., 2016). With
these changes, our objective function is modified

Intuitively, at each timestep j, we want the generated token to have the ability to rely on each of

4Note: For clarity, we omit the bias terms in Equations
10-15

where gd is a feed-forward neural network which
returns a distribution over the elements in the decoder vocabulary given the context vector cj (seeEq. 16) and decoder state sj .

6335

C AMBRIDGE
Partition

Count

LengthPhrases Examples Phrase

Context

Train
Valid
Test

21,993
4,671
4,670

42,689
5,335
5,337

1.01
1.00
1.00

9.14 ± 4.27 11.64 ± 6.75
9.25 ± 4.24 11.69 ± 6.80
9.20 ± 4.32 11.60 ± 6.75

Overall

24,55753,361

1.01

9.16 ± 4.27 11.64 ± 6.76

ROBERT
Partition

Count

Definition

LengthPhrases Examples Phrase

Context

Definition

Train
Valid
Test

30,0496,992
6,985

71,073
8,884
8,884

1.00
1.00
1.00

10.51 ± 7.36 7.97 ± 4.95
10.55 ± 7.47 7.93 ± 4.90
10.46 ± 7.36 8.03 ± 5.00

Overall

33,507

88,842

1.00

10.51 ± 7.38 7.97 ± 4.94

Table 1: Statistics for C AMBRIDGE and ROBERT. The
number of individual phrases, number of examples, andthe mean and s.d. of the lengths of each partition of thedataset are reported

In addition to these datasets, we also develop
the C AMBRIDGE (English) and ROBERT (French)dataset. We collect this data from the online versionof the Cambridge Dictionary7 and Le Petit Robert8 .
Following the spirit of previously released datasets,
we include three components for each example: (1)the word or phrase being defined, (2) an example(context sentence) in which it is contained and (3)its corresponding definition. These datasets can beseen as an addition to the domain of “traditional”
dictionary definitions, with ROBERT being the first
non-English dataset. Please refer to Table 1 forstatistics regarding these datasets.
4.2
4.2.1

to become the following:L(θ, φ) = −Ez∼qφ (z|w,d) log pθ (d|w, z)X+γmax(λ, DKL (qφ (zi |w, d)||p(zi |w)))

(18)

i

Where γ denotes the annealing term that follows
the sigmoid schedule and λ denotes the target rate,
and the sub-index i denotes the ith dimension of
the latent vector z. In our experiments, we set the
total λ = 1.

4

Empirical Study

4.1

Data

The task of definition modeling with respect to eachof the aforementioned datasets can be regarded as
three separate domains, in which (1) OXFORD canbe viewed as a corpus of “traditional” dictionary
definitions, where most common words in a given
language are contained, (2) U RBAN can be viewed
as a corpus of “uncommon , slang words in which
one often has to use context and subword information to decipher the meaning, and (3) W IKIPEDIA
can be viewed almost as a description generation
task of named entities, conditioned on the given
context.
56

oxforddictionaries.com
urbandictionary.com

Our Model: VCDM

We initialize each of our encoders with BERT-baseuncased (or in the case of ROBERT, CamemBERTbase (Martin et al., 2019)), setting de , dc = 768.
We set latent dimension dz = 83, and the LSTM
decoder’s hidden size dd = 512 with an output vocabulary size of 10k, initializing embeddings with
Word2Vec (Mikolov et al., 2013). We perform gradient descent using the Adam optimizer (Kingma
and Ba, 2014) with its default hyperparameters.
During decoding, we use the beam-search algorithm, setting the beam size to 5. We implement allmodels in PyTorch (Paszke et al., 2019).
4.2.2

To evaluate our approach we make use of thefollowing previously released datasets: OXFORD(Gadetsky et al., 2018) built from Oxford Dictionaries5 , U RBAN built from the Urban Dictionary6 ,and W IKIPEDIA (Ishiwatari et al., 2019) built from
Wikipedia.

Experiments

Baselines

Local and Global Context-Aware Description
generator (LoG-CAD): proposed by Ishiwatari
et al. (2019), this model achieved the previous
state-of-the-art on existing datasets for this task.
The model makes use of a BiLSTM (Graves andSchmidhuber, 2005) to encode sentence-level context, a character-level CNN (Zhang et al., 2015) toencode character-level information, and pretrained
Google CBOW9 (Mikolov et al., 2013) vectors (forROBERT we use the French fasttext word vectors (Grave et al., 2018)). During decoding, thismethod makes use of a 2-layer attentional 300-dim
LSTM decoder with an additional gating mechanism to combine all these sources of encoding
information.
LSTM baseline (LSTM): To show the effect ofcontinous latent variable modeling for this task,
7

dictionary.cambridge.org
https://dictionnaire.lerobert.com/
9https://code.google.com/archive/p/
word2vec/6336

8

Our interest in BERTScore sparks from the factthat it has been recently shown to correlate betterwith human judgement in system evaluations, andto address the potential issue of coherent definition
generations being given low evaluation scores as aresult of having zero or low n-gram overlap withthe reference sentence.
Finally, in addition to our automatic evaluation
we also performed a human study, where threedifferent human annotators evaluated the output
generated by our proposed approach, as well as by
the LoG-CAD and BERT-ft baselines. We followed
the approach by Ishiwatari et al. (2019) and usedtheir 1-5 scale:
1. Completely wrong or self-definition
2. Correct topic with wrong information
3. Correct but incomplete
4. Small details missing

PBERT

RBERT

FBERT

LoG-CAD
LSTM
BERT-fr
BERT-ft
VCDM

18.63
21.02
18.26
27.26
27.38

86.40
85.58
85.95
87.36
87.47

80.57
85.51
85.11
87.07
87.11

83.38
85.52
85.50
87.19
87.27

U RBAN

LoG-CAD
LSTM
BERT-fr
BERT-ft
VCDM

10.65
11.10
9.89
11.45
13.90

78.73
84.27
84.04
84.91
85.15

81.77
83.54
82.36
82.65
83.70

80.09
83.87
83.12
83.71
84.36

W IKIPEDIA

LoG-CAD
LSTM
BERT-fr
BERT-ft
VCDM

36.65
38.86
35.97
42.97
42.27

89.51
90.09
89.51
90.48
90.89

88.17
88.44
88.11
89.54
88.97

88.83
89.21
88.77
89.97
89.87

C AMBRIDGE

LoG-CAD
LSTM
BERT-fr
BERT-ft
VCDM

16.87
16.44
17.90
20.04
22.46

86.09
86.21
87.17
87.81
88.16

85.32
85.43
85.95
86.88
87.46

85.68
85.81
86.53
87.24
87.70

LoG-CAD
LSTM
BERT-fr
BERT-ft
VCDM

22.94
39.76
23.61
41.50
44.97

69.77
78.89
73.74
81.82
82.80

68.09
79.18
71.90
80.54
81.96

68.80
78.90
72.63
81.02
82.24

OXFORD

Evaluation

When comparing our approach to our baselines
we make use of two automatic evaluation metrics, namely sentence-level BLEU (Papineni et al.,
2002; Koehn et al., 2007) and the recently proposed BERTScore (Zhang et al., 2019). While theformer is a well-known metric for machine translation, based mainly on n-gram matching between
source and target, the latter is a rather new approachthat leverages BERT’s pretrained contextual embeddings, matching words in candidate and reference
sentences by way of cosine similarity. Concretely,
BERTScore computes 3 metrics, namely precision(denoted as PBERT ), recall (denoted as RBERT )and F1 score (denoted as FBERT ).

BLEU

DATA

BERT Baselines: This baseline is a single-layer attentional 512-dim LSTM-LM decoder conditioned
on rwt . We use two variants: (1) BERT-fr whererwt is produced by a a frozen BERT-base encoderand (2) BERT-ft where rwt is produced by a BERTbase encoder finetuned during training.
4.3

Model

ROBERTand for a more direct comparison to LoG-CAD,
we implement an LSTM version of our proposedarchitecture. Following LoG-CAD, use a 2-layer
300 dimensional BiLSTM as each encoder and use
a 10k Byte-Pair tokenized (Sennrich et al., 2016)
encoder vocabulary. The neural definition inferer
and the variational definition modeler are kept thesame as our proposed method.

Table 2: Results on the test set for U RBAN , OXFORD ,W IKIPEDIA , C AMBRIDGE, and ROBERT.

5. Correct
to evaluate 100 randomly sampled instances fromOXFORD.
To compare the values obtained for each example
across two models, we utilized t-tests and pair-wisebootstrap resampling tests with 10,000 samples(Koehn, 2004), controlling for the random seed (setto 2 in our experiments).

5

Results

Automatic Evaluation: Table 2 shows the resultson the test set for each reported metric and dataset.
Firstly, we note that the LSTM Baseline is able
to consistently outperform LoG-CAD in terms of
BERTScore, although with mixed results in termsof BLEU. We think this difference is mainly due tothe n-gram matching nature of BLEU, which tends
to give better scores for longer but incorrect generations, as the example in Table 3 shows, while also
being unable to adequately handle cases where thedefinitions are expressed using words not present in
the gold standard. We believe these results validate
the usage of a metric such as BERTScore on thistask, ultimately showing that tackling definition

6337

Word

Frankenstein
In arming the dictator, the US was creating aContext
Frankensteinsomething that destroys or harms the person
Referenceor people who created it
Generated

BL

P

R

F

something that you say or dothat you think someone ofsomething is ridiculous

12.5 83.41 84.78 84.09

an extremely frightening or
offensive person

8.13 87.00 84.78 85.88

Model

Model

p-value bootstrap

p-value t-test

VCDM
BERT-ft
VCDM

LoG-CADLoG-CADBERT-ft

0.005
0.006
0.797

1.4 × 10−8
4.2 × 10−5
1.6 × 10−2

Table 5: Exact p-values of the performed statisticaltests, to compare the scores obtained during our humanevaluation.

Table 3: An example from the C AMBRIDGE test set,showing an evaluation issue caused by BLEU. The generated outputs of the LoG-CAD baseline are shown
above, and ours below. BL stands for sentence BLEU
and P, R and F stand for PBERT , RBERT , and FBERT .
Configuration

FBERT (∆)

BLEU (∆)

VCDM
Decoder LSTM Cell
Frozen definition encoder
Tied encoders
Frozen encoders
Frozen context encoder

87.70 ( — )87.68 (-0.02)
87.14 (-0.56)
87.14 (-0.56)
86.35 (-1.35)
85.19 (-2.51)

22.46 ( — )22.27 (-0.19)
20.70 (-1.76)
20.59 (-1.87)
17.42 (-5.04)
13.49 (-8.97)

Table 4: Results of the ablation study performed on
C AMBRIDGE.

modeling with a generative approach can lead to
improved results, and suggesting that the incorporation of a latent variable that models the underlying
definition space is beneficial for this task.
Results on Table 2 also show that the inclusion of
BERT significantly improves generation quality interms of BERTScore and BLEU on most datasets.
This suggests that the inclusion of pretrained deep
contextual word representations is beneficial forthe task, which is expected given its contextual
nature. We also see that VCDM is able to successfully leverage BERT, as our model is able to offer
improved results compared to BERT baselines inall datasets except W IKIPEDIA. We think these results offer additional empirical evidence to support
the effectiveness of our generative approach. Improvements provided by our model are particularly
significant in the case of U RBAN, a dataset which
there are many rare words and the context is arguably less informative due to its noisy properties.
We surmise that the subpar performance of VCDM
over BERT-ft in W IKIPEDIA is related to the properties of the dataset domain (i.e. description generation of named entities). With this in mind, it
could be argued that a completely context- focusedarchitecture (such as that of our finetuned BERT
baseline) has properties that are more beneficial
in this setting. Contrary to findings in Ishiwatari
et al. (2019) which argue for the inclusion of a
global context during generation, we find that acontextually-focused (local context) architecturewith a strong context encoder (such as BERT) results in better performance within this domain.
Ablation Study: To further evaluate the contribution of each introduced component in our approach
we performed an ablation study on C AMBRIDGE.
Results of these experiments are summarized in Table 4, where it is possible to see that each of our introduced components is beneficial to the task. Notethat the VCDM-Cell vs LSTM-Cell improvement isminimal on this dataset. The purpose of the integration of the latent variable in the decoder LSTM cell
is for it to act as a ”global definition signal” so we
can rely on the properties of the latent variable. As
this property is especially useful in cases in which
there is noisy context, we think it is reasonable
to assume that as context here is more informative, the performance gain from including a global
definition signal is relatively small. We also seethat freezing the context encoder has a extremley
negative impact on performance. We believe thisis because the context-encoder hasn’t effectively
learned to use the phrase-context pairs (Sec. 3.1.1).
Human Evaluation: Average human scores obtained are 2.51 for LoG-CAD, 3.08 for BERT-ft
and 3.31 for VCDM. When tested for statistical significance (Table 5), we observed that both
VCDM and BERT-ft were superior to LoG-CAD
with 99% confidence, using both paired t-tests orpair-wise bootstrap resampling tests (Koehn, 2004),
and that the difference between VCDM and BERTft was statistically significant at 95% for the t-test.
Qualitative Evaluation: Finally we provide a
qualitative evaluation by showing an example ofthe output of our model and of two of our base-6338

Word:

Present(VB)

Context:

Within a sexist ideology In
additiontoand a male-dominated this, think of thecinema, the woman is presents, the toys,presented as what she gift sets, and mostrepresents for man.
importantly, all thatwrapping paper.

Present(NN)Reference: To represent (someone A thing given toor something) to others someone as a gift
in a particular way
LoG-CAD: a person who is present a person’s mind
in a particular way
BERT-ft:

Portray or regard (some- An item of furnitureone) as a particular per- presented to resemson, idea or action
ble a bride <unk>

VCDM:

To portray or describe A thing kept as a gift(someone or something) for children
in a particular context

Table 6: Example showing the generated definitions fortwo senses of the word “present”, taken from OXFORD.

lines, in Table 6. In the OXFORD dataset, which
this example is taken from there are 7 senses of the“present”, showing VCDM’s ability to effectively
disambiguate between a large amount of senses.

6

Conclusion

In this paper we have introduced a generative modelthat directly combines distributional and lexical semantics via a continuous latent variable for the taskof definition modeling. Empirical results on multiple corpora, including two new datasets released,show that our model is able to outperform previous work by a consistent margin, also successfully
being able to leveraging contextualized word representations. For future work we are interested in exploring how definition modeling could be adapted
to a multilingual or cross-lingual setting.

Acknowledgments
We are grateful for the support provided by the
NVIDIA Corporation, donating two of the GPUs
used for this research. We thank Victor Zhong
for insightful discussions, and thank Pablo Loyola,
Cristian Rodriguez-Opazo, and Jorge Balazs for
proofreading the work and providing useful suggestions.

References
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016.
Generating Sentences from a Continuous Space. In

Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 10–21,
Berlin, Germany. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2018. BERT: Pre-training of deep
bidirectional transformers for language understanding.
Artyom Gadetsky, Ilya Yakubovskiy, and Dmitry P.
Vetrov. 2018. Conditional generators of words definitions. CoRR, abs/1806.10090.
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning
word vectors for 157 languages. In Proceedings of theInternational Conference on Language Resources and
Evaluation (LREC 2018).
Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional lstm andother neural network architectures. Neural networks,18(5-6):602–610.
Felix Hill, Kyunghyun Cho, Anna Korhonen, andYoshua Bengio. 2016. Learning to understand phrases
by embedding the dictionary. Transactions of the Association for Computational Linguistics, 4:17–30.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long
short-term memory. Neural computation, 9(8):1735–
1780.
Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing
Huang. 2019. Glossbert: Bert for word sense disambiguation with gloss knowledge. Proceedings ofthe 2019 Conference on Empirical Methods in Natural
Language Processing and the 9th International Joint
Conference on Natural Language Processing (EMNLPIJCNLP).
Shonosuke Ishiwatari, Hiroaki Hayashi, Naoki Yoshinaga, Graham Neubig, Shoetsu Sato, Masashi Toyoda,
and Masaru Kitsuregawa. 2019. Learning to Describe
Unknown Phrases with Local and Global Contexts. In
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1(Long and Short Papers), pages 3467–3476, Minneapolis, Minnesota. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A
method for stochastic optimization.
Diederik P. Kingma and Max Welling. 2014. AutoEncoding Variational Bayes. arXiv:1312.6114 [cs,stat]. ArXiv: 1312.6114.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz,
Xi Chen, Ilya Sutskever, and Max Welling. 2016. Improved Variational Inference with Inverse Autoregressive Flow. In D. D. Lee, M. Sugiyama, U. V. Luxburg,
I. Guyon, and R. Garnett, editors, Advances in Neural
Information Processing Systems 29, pages 4743–4751.
Curran Associates, Inc.

6339

Philipp Koehn. 2004. Statistical Significance Tests for
Machine Translation Evaluation. In Proceedings ofthe 2004 Conference on Empirical Methods in Natural Language Processing, pages 388–395, Barcelona,
Spain. Association for Computational Linguistics.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris
Callison-Burch, Marcello Federico, Nicola Bertoldi,
Brooke Cowan, Wade Shen, Christine Moran, Richard
Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin,
and Evan Herbst. 2007. Moses: Open source toolkit
for statistical machine translation. In ACL.
Minh-Thang Luong, Hieu Pham, and Christopher D
Manning. 2015. Effective approaches to attentionbased neural machine translation. arXiv preprint
arXiv:1508.04025.
Louis Martin, Benjamin Muller, Pedro Javier Ortiz
Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, and Benoı̂t
Sagot. 2019. Camembert: a tasty french language
model.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey
Dean. 2013. Efficient estimation of word representations in vector space.
Ke Ni and William Yang Wang. 2017. Learning to Explain Non-Standard English Words and Phrases. In
Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2:
Short Papers), pages 413–417, Taipei, Taiwan. AsianFederation of Natural Language Processing.
Thanapon Noraset, Chen Liang, Larry Birnbaum, andDoug Downey. 2017. Definition modeling: Learning
to define word embeddings in natural language. In
AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence.

Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237,
New Orleans, Louisiana. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016. Neural machine translation of rare words withsubword units. In Proceedings of the 54th Annual
Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725,
Berlin, Germany. Association for Computational Linguistics.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. 2015.
Learning Structured Output Representation using Deep
Conditional Generative Models. In C. Cortes, N. D.
Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 3483–3491. Curran Associates, Inc.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all you
need.
Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and
Min Zhang. 2016. Variational Neural Machine Translation. In Proceedings of the 2016 Conference on
Empirical Methods in Natural Language Processing,pages 521–530, Austin, Texas. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q.
Weinberger, and Yoav Artzi. 2019.
Bertscore:
Evaluating text generation with BERT.
CoRR,
abs/1904.09675.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.
Character-level convolutional networks for text classification. In Advances in neural information processing
systems, pages 649–657.

Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th
Annual Meeting of the Association for Computational
Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam
Lerer, James Bradbury, Gregory Chanan, Trevor
Killeen, Zeming Lin, Natalia Gimelshein, Luca
Antiga, Alban Desmaison, Andreas Kopf, Edward
Yang, Zachary DeVito, Martin Raison, Alykhan Tejani,
Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie
Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In
H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlchéBuc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8026–
8037. Curran Associates, Inc.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt
Gardner, Christopher Clark, Kenton Lee, and Luke
Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conferenceof the North American Chapter of the Association for

6340

VCDM: Leveraging Variational Bi-encoding and Deep Contextualized
Word Representations for Improved Definition ModelingApackage2 to calculate BERTScore, with hashDatasets

Tables 1, 2 and 3 provide a summary of the sizesof each partition for the datasets OXFORD, U RBAN
and W IKIPEDIA, respectively.
Count

Partition
Train
Valid
Test

Phrases

ExamplesPhraseContext

Definition

33,1288,867
8,850

97,855
12,232
12,232

1.00
1.00
1.00

17.74
17.80
17.56

11.02
10.99
10.95

Train
Valid
Test

Count

LengthPhrases

ExamplesPhraseContext

Definition

190,696
26,876
26,875411,384
57,88338,371

1.54
1.54
1.68

10.89
10.86
11.14

10.99
10.95
11.50

Table 2: Statistics for U RBAN. The number of individual phrases, number of examples, and the mean lengthsof each partition of the dataset are reported.

Partition
Train
Valid
Test

Count

LengthPhrases

ExamplesPhraseContext

Definition

151,995
8,361
8,397

887,455
44,00357,232

2.10
2.11
2.10

18.79
19.21
19.02

5.89
6.31
6.94

Table 3: Statistics for W IKIPEDIA. The number of individual phrases, number of examples, and the mean
lengths of each partition of the dataset are reported.

B

Evaluation

We make use of the sentence-bleu.cpp1
script in the MOSES (Koehn et al., 2007)
GitHub repository to compute sentence-level
BLEU, and use the bert-score Python
1

https://github.com/moses-smt/
mosesdecoder/blob/master/mert/
sentence-bleu.cpp(hug_trans=2.8.0)

Englishdatasets

for
and

the
hash

bert-base-multilingual-cased_L9_no-idf_

for
ROBERT which is in French.
For datasets
where there are multiple examples for a given
word sense, such as W IKIPEDIA, we note that
results provided by (Ishiwatari et al., 2019)are obtained by first averaging the evaluation
metrics for multiple examples for a given sense—although this is not reported on the paper—
which tends to inflate the final values of the metrics.
Instead, in this paper report example-wise metricaverages, which provide more realistic values of
the aggregated evaluation metrics.
version=0.3.3(hug_trans=2.10.0)

LengthTable 1: Statistics for OXFORD. The number of individual phrases, number of examples, and the mean lengthsof each partition of the dataset are reported.

Partitionroberta-large_L17_no-idf_version=0.3.2

C

Model Details

We train each of our models using a batch size of64, and set 1e-3 as the initial learning rate for theAdam optimizer. However, when finetuning BERT(or CamemBERT) in any circumstance, we set theinitial learning rate for the BERT parameters to
be 5e-5 and use a linear warmup schedule, warming up for the first epoch. We train all models in
PyTorch (Paszke et al., 2019), and use the HuggingFace3 (Wolf et al., 2019) implementation ofCamemBERT-base and BERT-base-uncased.
Additionally, we re-implement LoG-CAD (Ishiwatari et al., 2019) using the authors’ GitHub repository4 .
2https://github.com/Tiiiger/bert_score
https://github.com/huggingface/transformers
4
https://github.com/shonosuke/
ishiwatari-naacl2019

6341

3

D

Infrastructure and Environment

Experiments for different datasets were run in two
different machines:• A server machine with an Intel Xeon E5-2630
CPU, and two NVIDIA RTX-2080 (Driver
418.56, CUDA 10.1) GPUs, running Ubuntu
16.04• An additional server machine with an Intel
Core i7-6850K CPU and two NVIDIA Titan
Xp (Driver 430.50, CUDA 10.1) GPUs, also
running Ubuntu 16.04

E

Additonal Output Examples
Word:

Yen

Context:

If Koizumi has enjoyed some economic success,say critics, it has been through a combination ofgood luck and what many believe has been an
artificial weakens of they yen against the dollar.

Reference: the basic monetary unit of japan.
LoG-CAD: a longing or yearning
BERT-ft:a monetary in a foreign country

VCDM:

the basic monetary unit of japan, equal to 100
cents.

Table 4: Descriptions for the rare word “yen"Word:doucheturdContext:

Brad, you’re such a doucheturd.

Reference: insulting noun, being both a douche or douchebag
and a turd
LoG-CAD: a person who is a douchebag
BERT-ft:a person who is a douchebag

VCDM:

a person who is a mix of a douche and a turd

Table 5: Descriptions for the slang word “doucheturd"

6342

Gold
LoG-CAD
VCDM( especially of a political party ) sponsor ( a candidate ) in an election
a candidate or candidate in a race or election
set ( a person or team ) in an electionGold
LoG-CAD
VCDM( of a batsman ) run from one wicket to the other in scoring or attempting to score a run .
a race or contest in which a race is run( of a sports team ) run by hitting at the reach of the three runsGold
LoG-CAD
VCDM

a large open stretch of land used for pasture or the raising of stock( of a horse ) be <unk>a specially <unk> area for cattle or cattle

Gold
LoG-CAD
VCDM

a large open stretch of land used for pasture or the raising of stock( of a horse ) be <unk>a specially <unk> area for cattle or cattle

Gold
LoG-CAD
VCDM

a preliminary test of a procedure or systema person or thing that is <unk> or <unk>
a continuous search or undertaking

Gold
LoG-CAD
VCDM

a race between candidates for elective officea series of people who are <unk>
be the charge ofGold
LoG-CAD
VCDM

a row of unravelled stitchesa short , narrow <unk>
a <unk> in a careless or <unk> way

Gold
LoG-CAD
VCDM

a track made or regularly used by a particular animal
move or cause to move in a specified directiona <unk> or <unk> run over a treeGold
LoG-CAD
VCDM

become undone
be <unk>
become undone by being undoneGold
LoG-CAD
VCDM

cause something to pass or lead somewheremove or move in a <unk>cause something to pass or lead somewhere by constant strengthGold
LoG-CAD
VCDM

cause to performmake a series of facts or plans
put in an areaGold
LoG-CAD
VCDM

diarrhoea .
a person ’s <unk>
a <unk> of <unk>Gold
LoG-CAD
VCDM

emit or exude a liquid( of a person ’s eyes ) move or cause to move in a specified direction( of a person ’ s <unk> ) become <unk> with <unk>Gold
LoG-CAD
VCDM

extend or continue for a certain period of time
be a result ofpass for a certain time of timeGold
LoG-CAD
VCDM

fail to stop at ( a red traffic light )( of a vehicle ) move or move in a specified direction
run at or at a particular <unk>Gold
LoG-CAD
VCDM

move about in a hurried and hectic way
be in a specified way
go or run somewhere in a particular placeGold
LoG-CAD
VCDM

move or cause to move between the spools of a recording machine
move or move in a specified directionuse or enable ( a container ) to a desired point in a specified directionGold
LoG-CAD
VCDM

publish or be published in a newspaper or magazine
a <unk> or <unk>( of a newspaper or a public station ) publish or broadcast ( a television programme )Gold
LoG-CAD
VCDM

put ( a form of public transport ) in service( of a vehicle ) be <unk> or <unk>provide ( an undertaking , train , or service ) for a service

Gold
LoG-CAD
VCDM

the act of running ; traveling on foot at a fast pacea run in a race
the act of running ; traveling on foot at a fast paceGold
LoG-CAD
VCDM

the act of testing something
the act of <unk> something
the act of testing somethingGold
LoG-CAD
VCDM

travel a route regularly
move or move in a specified direction
travel a route regularlyGold
LoG-CAD
VCDM

the after part of a ship ’s bottom where it rises and narrows towards the stern .
a <unk> or <unk> .
an act of <unk> a boat ’ s footGold
LoG-CAD
VCDM

the average or usual type of person or thinga person ’s existence or belief
the general aspects of something , especially a language

Table 6: Comparison of the outputs6343
for the 23 senses of “run’ on OXFORD.

References
Shonosuke Ishiwatari, Hiroaki Hayashi, Naoki Yoshinaga, Graham Neubig, Shoetsu Sato, Masashi Toyoda,
and Masaru Kitsuregawa. 2019. Learning to Describe
Unknown Phrases with Local and Global Contexts. In
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1(Long and Short Papers), pages 3467–3476, Minneapolis, Minnesota. Association for Computational Linguistics.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris
Callison-Burch, Marcello Federico, Nicola Bertoldi,
Brooke Cowan, Wade Shen, Christine Moran, Richard
Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin,
and Evan Herbst. 2007. Moses: Open source toolkit
for statistical machine translation. In ACL.
Adam Paszke, Sam Gross, Francisco Massa, Adam
Lerer, James Bradbury, Gregory Chanan, Trevor
Killeen, Zeming Lin, Natalia Gimelshein, Luca
Antiga, Alban Desmaison, Andreas Kopf, Edward
Yang, Zachary DeVito, Martin Raison, Alykhan Tejani,
Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie
Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In
H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlchéBuc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8026–
8037. Curran Associates, Inc.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien
Chaumond, Clement Delangue, Anthony Moi, Pierric
Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, andJamie Brew. 2019. Huggingface’s transformers: Stateof-the-art natural language processing.

6344