A Self-Supervised Integration Method of Pretrained Language Models and Word Definitions
Year: 2023

Abstract:
We investigate the representation of pretrained language models and humans, using the idea of word definition modeling–how well a word is represented by its definition, and vice versa. Our analysis shows that a word representation in pretrained language models does not successfully map its human-written definition and its usage in example sentences. We then present a simple method DefBERT that integrates pretrained models with word semantics in dictionaries. We show its benefits on newly-proposed tasks of definition ranking and definition sense disambiguation. Furthermore, we present the results on standard word similarity tasks and short text classification tasks where models are required to encode semantics with only a few words. The results demonstrate the effectiveness of integrating word definitions and pretrained language models.1

heading: 1 Introduction
text: A word embedding vector maps a word into a fixeddimensional vector as a distributed representation. The word vectors are trained by looking at their context words and aggregating their representations in supervised ways (Turney, 2013) or unsupervised ways (Mikolov et al., 2013; Pennington et al., 2014). More recently, the representations have been learned as a form of pretrained language models (Peters et al., 2018; Devlin et al., 2019). The huge success of these pretrained language models on various NLP tasks is achieved by capturing a rich semantic representation of words from their context in huge data.
On the other hand, for centuries, lexicographers and linguists have created dictionaries that contain general definitions of words and examples of their usage. With these sophisticated data, there have been many applications for NLP tasks (e.g., machine translation (Hill et al., 2016), semantic
1https://github.com/hwiyeoljo/DefBERT
relatedness classification (Bahdanau et al., 2017)). Some recent works have used WordNet (Miller, 1995) for fine-tuning BERT for word sense disambiguation (Huang et al., 2019; Guo et al., 2020), whereas our work uses up-to-date dictionary definitions and usage examples to fine-tune pretrained language models.
In this work, we study the difference between machine-learned definitions and human-written definitions. Table 1 shows the mean squared distance between the vanilla BERT representation (the last hidden layer of [CLS]) for the word ‘love’ and the sentence representation (by [CLS]) for its definitions in dictionaries. The closest word of ‘love’ in the pretrained model is ‘number’ in our data collection. This indicates a potential risk of using pretrained representations as the only means to measure the semantic similarity between words
14
or short sentences, where the context words are insufficient to get good representations.
Furthermore, it is important to make general and self-indicated embeddings. For example, if we do not have pooling layers to the pretrained embedding, we need additional training data to fine-tune the pooling layer and the pretrained model. On the other hand, if we can do the same task by using the pretrained model only (without fine-tuning), this means good generalization.
Lastly, some researchers believe that target word token representation is better than [CLS] token when the input text is short. However, we do not know ‘what the short text is’ or ‘when the model gets short text as inputs.’ Thus, the fact that we can use [CLS] token for a single word or short text is beneficial in that we do not need to consider the input length. To do so, we attempt to inject worddefinition-example (its usage) information into the model.
To overcome the deficiency and get such a generalized model, we propose a new joint representation that combines the human-written word definition with its usage example in a dictionary entry. We show the effectiveness of this new representation on several downstream tasks.
The main contributions are:
• Performed extensive analyses of how close the representations of pretrained language models are to the one of collected human-written definitions; our analyses show that the representations of BERT do not reflect the humanwritten definitions.
• Incorporated the dictionary definitions into the pretrained language models in embeddinglevel–As a new model called DefBERT (§4), showing significant performance improvements where tasks lack contextual information.
• Proposed two semantics-related ranking tasks: DefRank aims to find the correct definition given the word, and SenseRank is to find the proper sense from a word’s definitions given the word’s usage. Unsurprisingly but interestingly, DefBERT shows significant improvements in both tasks.

heading: 2 Related Work
text: Using dictionaries for NLP tasks. Dict2vec (Tissier et al., 2017) learned word
embeddings through word-definition pairs. They designed strong and weak word pairs within dictionaries and made the word pairs close. Bahdanau et al. (2017) utilized dictionaries to solve out-of-vocabulary (OOV) problems by encoding the definitions of OOV words and generating the word’s embeddings. Hill et al. (2016) suggested a dictionary-based learning task using neural networks. They also suggested reversed dictionary evaluation tasks that choose the most related word to a given description. Like dictionaries, WordNet (Miller, 1995) has been widely used to enrich word representations (Faruqui et al., 2015). However, the prior works were biased to inject relation knowledge, such as synonyms, rather than general word definitions.
More recently, GlossBERT (Huang et al., 2019) used definitions for disambiguation tasks, but the approach needs context-gloss pairs and a classifier even at inference. In this work, we attempt to build a generalized model which does not require additional classifiers.
Definition Modeling. The definition modeling task was proposed by Noraset et al. (2017) that generates a word definition from the word’s embedding. The authors considered the definition modeling as a special case of language modeling and used it for word embedding evaluation. However, Gadetsky et al. (2018) found that the prior definition modeling tasks could not resolve word disambiguation because it is conditioned on only a single word. To address the issue, they also extended Noraset et al. (2017)’s model to process context.
Chang and Chen (2019) investigated whether contextual representations can capture word definitions. Unlike the prior works on definition modeling, they suggested a general framework that maps the contextualized representation into a definition embedding space and then selects top-N closest definitions. This retrieval-based approach can resolve the problems in the generative approach of definition modeling, such as the difficulty in evaluation.
The major differences between the prior works and our study are as follows: First, we compare representations from pretrained language models and definitions from a lexical dictionary at embeddinglevel. Second, we use word-definition pairs and definition-example pairs from the dictionary. The use of words in a sentence is similar to GlossBERT, but its objective is not to make definition-injected
representation. Rather it is to solve sense disambiguation tasks. The method also requires an additional classifier. Lastly, we propose two tasks that can measure the capability of model representations on human-written definitions (and examples): DefRank and SenseRank. Compared to other benchmark datasets that predict how similar two words or sentences are, we expect these tasks to be a straightforward benchmark.

heading: 3 Preliminary Analyses
text: The central motivation behind our analysis is to check whether a word representation in pretrained language models (in this work, BERT) can indicate the representation of its definition and vice versa.
3.1 Definition Dataset Collection: Oxford+
In prior work, Chang et al. (2018); Chang and Chen (2019) collected an online dictionary from lexico2 (Oxford University Press, 2020). Since our work requires up-to-date definitions, we recollected the dataset based on the vocabulary of the original work.
Table 2 shows the comparison and statistics of the dictionary data. The number of unique vocabulary is slightly different from the previous one. However, when considering the number of definitions and the number of examples increases, we think that the difference is due to the updates of lexico dictionary. Dictionaries usually order word senses by how frequently the senses are used, so the order information is important for investigating major versus minor definitions. Due to the more extensive coverage of usage and definitions, and additional information, we call our dataset Oxford+.
2https://www.lexico.com but recently it is redirected to https://www.dictionary.com
From Oxford+, we take two sets of pairs and calculate the distances: one is a pair between a word and its definition (W-D), and the other is a pair between a definition and its usage (D-E) where the pairs are embedded by pretrained language model.

heading: 3.2 Distance Measures
text: Embedding scheme. We use bert-base-uncased in HuggingFace (Wolf et al., 2019) as a backbone model.3 Although there are several different ways to represent a word or sentence using BERT (e.g., averaging [CLS] in every hidden layer, concatenating [CLS], etc.), we use the [CLS] token in the last hidden layer, as the original BERT paper proposed.
For all definition-example pairs, we first input the example through BERT and then use the target word tokens in the example instead of using the [CLS] token (see Figure 1). We average their vectors if the target word is tokenized by more than one token.
Let i be the word index, i j be the definition index of the j-th definition of the i-th word, and i jk be the index of the k-th example of the j-th definition for word i. Following our central motivation, the distance |Wi − Di j| between a word Wi and one of its definitions Di j is calculated by mean squared distance. Likewise, the distance |Di j −Ei jk| between a word used in an example Ei jk and its definition Di j is calculated by mean squared distance.4
In order to compare BERT’s ability to capture human-written definitions, we need to control BERT’s inputs and weights. We thus use (1) [PAD] masked inputs on the target word and (2) BERT with random weights. or example, suppose the embedding of empty input (BERT([PAD] ... [PAD])) is closer to the definition embedding (BERT(D)) than a single word embedding (BERT(W))). In that case, BERT does not seem to capture definition information through the inputs. The controls by [PAD] will be denoted as W[PAD] for word and E[PAD] for usage example, respectively. The [PAD] controlled inputs are also illustrated in Figure 1. Likewise, if BERT with random weights performs better, BERT’s pretrained weights do not have information about human-written definitions. We denote the controlled model as Rand. With this idea, we can define distance types.
3In the experiment (§5.1), further fine-tuning through masked language modeling loss over collected definition datasets improves model performances a little. However, we focus on the pretrained representation of vanilla BERT.
4Likewise, |Wi − Ei jk | can be measured but it is out of our
Distance Types. For each word-definition pair and each definition-example pair, • |W-D| : computes the distance between the
original input vector (INPUT in Figure 1) and the definition vector for each layer. • |W[PAD]-D| : computes the distance between the
padded input vector (PAD INPUT in Figure 1) and the definition vector for each layer. • |Rand W-D| : is the same as |W-D|, but all the
model weights are randomly initialized. • |D-E| : computes the distance between the
definition vector and the target word vector
motivation since the pair does not use definition.
used in the example sentence. • |D-E[PAD]| : computes the distance between
the definition vector and the padded target word vector in the example. • |Rand D-E| : is the same as |D-E|, but all the
model weights are randomly initialized. To sum up, the padded inputs and the randomized weights are used to contaminate the model representation. If the contaminated embeddings are closer to definitions than the vanilla input or model embeddings, the model representation is not meaningful.

heading: 3.3 Findings
text: Distribution of Distances. We visualize the distances between a target word ‘love’ and all definitions in Oxford+ (Figure 2). As we showed in §1, the closest (or most similar) word to ‘love’ was ‘number’. The definitions of ‘love’ are scattered over the distribution, indicating how BERT’s representation of ‘love’ is far from its human-written definitions. We observe similar patterns in most words in the dictionary.
The pretrained representation of a word alone is not indicating its human-written definitions. Figure 3 (top) shows the averaged word-definition distances according to hidden layers. The distance of |W-D| is smaller than |W[PAD]-D| distance across all the hidden layer depth. Since the difference between them is only the input, the word itself in-
cludes information about the human-written definitions definition. In the same plot, however, the distance of randomized BERT |Rand W-D| is much lower than |W-D| and |W[PAD]-D| at upper layers, which casts doubt about BERT’s pretrained weights whether they can represent human-written definitions. We thus conjecture that using a word alone is not appropriate for a contextualized representation since a single word lacks context.
To provide more context for the model, we conduct a second experiment to compare the definition’s representation to its usage in the example sentence where pretrained language models have shown strong performances.
BERT can self-indicate better by using surrounding words but it still fails to capture the human-written definitions. Figure 3 (bottom) shows the definition-example distances. The distances of |D-E| and |D-E[PAD]| shows similar trends but |D-E[PAD]| is smaller at the last hidden layer. The result shows the tokens are less selfindicated in the sentences, while the averaged distance of the randomized model is much smaller than in ordinary settings.
From this analysis, the pretrained language model (especially BERT) seems unable to encode human-written definitions, as |Rand W-D| and
|Rand D-E| show lower distance than |W-D| and |D-E|, respectively. Also, the distances between the vanilla BERT and the padded models are small, which tells us that it might have potential benefits by adding semantic information.
4 DefBERT: Definition Induced BERT
Using lexical resources for fine-tuning word embeddings is a typical solution to take advantage of both lexical semantics and distributional semantics. However, as seen in §3, the lexical relations, such as antonyms and synonyms, are unnatural to be integrated with pretrained language models. On the other hand, dictionary definitions and examples are expressed as complete sentences, leading to better settings for optimizing the pretrained models.
Based on the analysis (§3), we present a simple yet effective method to integrate general definitions from a dictionary with pretrained representations while keeping the nature of contextualization. The setup of BERT for fine-tuning is the same as Figure 1; we then fine-tune BERT using the distances as a loss function.
By doing so, we optimize BERT’s representation to be close to its human-written definitions (W-D) and its word representation used in the examples (D-E). The loss functions used for each pair are as follows:
LW-D = 1 #W × #D ∑
i
∑
j
√ (Wi − Di j)2 (1)
LD-E = 1 #W × #D × #E ∑
i
∑
j
∑
k
√ (Di j − Ei jk)2
(2)
where i is the word index, j is the definition index of the j-th definition of the i-th word, and i jk is the index of the k-th example of the j-th definition for word i. The number of words, definitions, and examples are denoted as #W, #D, and #E, respectively. We use Adam optimizer (Kingma and Ba, 2014) with learning rate 5e-6, and 32 batch size. The maximum token length from our definition data is 191, including special tokens (e.g., [CLS] and [SEP]), but we utilize the model’s maximum capacity, which is 512.
However, as we observed in the analysis (§3), the pretrained embeddings of source and target words (i.e., W, D, and E) might not be appropriate to be trained. Therefore, we additionally design loss functions, which utilize the other dictionary information: the distance between [CLS] token of W and
W tokens itself (W’) to align the token embedding(s) to [CLS] token. Likewise, the distance between W and E is used.
LW-W’ = 1
#W
∑
i
√ (Wi − W’i) (3)
LW-E = 1 #W × #D × #E ∑
i
∑
j
∑
k
√ (Wi − Ei jk)2
(4)
We use the additional loss functions for calibration of DefBERT. As a result, we can provide all the information in the dictionary self-supervised way.
In the training process, we prepare two BERT models in order to make the training fast and keep BERT’s original properties; One of BERT model makes prediction and update its weights by the loss(es), while the other BERT model only makes prediction used for target embedding. The target BERT is copied in every epoch. After the training, the fine-tuned BERT is selected.

heading: 5 Experiments

heading: 5.1 DefRank: Definition Ranking Task
text: Setup. To evaluate the ability of pretrained word vectors to capture human-written definitions at embedding-level (i.e., without classifiers), we present a task called Definition Ranking (DefRank).
Given a word, the model predicts the closest word definition among four candidate definitions. The main idea is similar to Chang and Chen (2019), but DefRank looks at only a word and does not require additional mapping function in the evaluation framework, which corresponds to our goal–get a general embedding model. We assign approximately 10% of data to test set5.
DefRank has two sets based on task difficulty: Easy set and Challenge set. The candidate definitions in the easy set are randomly sampled from Oxford+. On the other hand, candidate definitions in the challenge set are selected by the closest three definitions except for the gold definitions. We use Sentence-BERT (Reimers and Gurevych, 2019) to choose similar and negative examples as an adversarial constraint. Therefore, models are supposed to capture the subtle differences in meaning among the definitions of words such as love, dear, endearing, and adoration. Table 3 (top) and Table 3 (middle) show the examples.
Furthermore, the easy set has a sub-set called Neologism set, which consists of a newly coined word or expression. Thus, we can evaluate the models’ ability even when the words never appear in the (pre-)training data.
To collect neologisms, we refer to the update notes of Oxford Dictionary and consider ‘new word entries’ as neologisms. We then process them by removing words that require a subscription to see the full definition and references in definitions to other similar words (e.g., See, Cf. and explanations after ‘;’). The number of collected neologisms is 345. Table 3 (bottom) presents the example of neologism.
We compare BERT variations, such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), Sentence-BERT (Reimers and Gurevych, 2019), and GlossBERT (Huang et al., 2019). Besides, we report the performance fine-tuned by masked language modeling on the definition data. In the masked language training, we set an artificial template that “the definition of W is D and its example is E.” As we mentioned in §4, W-W’ pairs and W-E pairs are used for model calibration, denoted as [+W’] and [+E], respectively.
We also empirically find the optimized pair selection of DefBERT, which shows the best performances in DefRank, denoted as BestSelect6).
5After we post-process to clean the test data, the ratio becomes approximately 9%
6The best sequence of training is [+E]+W-D+D-E+[+W’].
Finally, we will report the BestSelect model performance.
Results. Table 4 shows the performance on the DefRank task. Considering the high performance of Sentence-BERT, our tasks are well-designed to examine the semantics incorporated in model representations. The results show that fine-tuning by masked language modeling is ineffective in the performances. Besides, GlossBERT does not perform well on these tasks, which implies that the word disambiguation model largely depends on the classifiers at the end of the architecture.
Our variations of DefBERT show much better performance since we train models with a similar distribution. However, it is interesting that D-E pairs increase the model performances more, even though W-D pairs are directly related to the tasks.
The performance gaps between baselines and our variations are small for the challenge set. Therefore, the challenge set is very hard to distinguish the subtle variation of semantics, requiring a deeper understanding of definitions.
Lastly, we can find several properties of definition pairs. For example, calibrations with only [+W’] or [+E] make significant improvements to the model. The models starting with calibration perform much better than the models without calibrations. We guess that BERT’s self-attention successfully normalize the model. Moreover, DefBERT
proves to be effective in neologisms. We conjecture that DefBERT learns unseen words (and their tokens) through other words’ definitions. We also report the performance of ConceptNet vector (Speer et al., 2017). The representation is a strong baseline since the embeddings are fine-tuned and specialized in a number of tasks regarding word semantics. When evaluation, the sentence vectors are made by averaging the word vectors. ConceptNet shows good performance on the easy set and the challenge set, which also tells us that DefRank correlates with word semantics tasks while hardly being correct in neologism. The combination of various types of lexical resources (e.g., dictionary, relation, WordNet) remains an interesting direction for future work.

heading: 5.2 SenseRank: Sense Disambiguation Task
text: Setup. Extending from DefRank, we propose another task SenseRank that distinguishes the different senses of definitions for the same word. In this setting, we provide a word and its usage, an example sentence. Then, models select the most appropriate sense of definitions among the word’s definitions. Compared to Chang and Chen (2019), SenseRank has to choose a gold definition among the candidate definitions from the same target word. Therefore, the task can be used to measure the model’s ability to do fine-grained sense disambiguation.
Table 5 shows four definitions for the target word ‘love’. Given an example sentence, DefBERT correctly predicts the most similar sense of the definitions, while BERT fails. Similar to the challenge set of DefRank, the candidate definitions in SenseRank are semantically very similar (i.e., the variation of their senses), but this task has more contexts than DefRank.
We filter out the words for which the number of definitions is fewer than four. We then sample 10% (115,849) as a test set.
Results. Table 6 shows the performances on SenseRank. Similar to the DefRank, the accuracies from BERT variants are relatively low except for Sentence-BERT, which is good at encoding semantics. Apart from D-E pairs that is closely related to SenseRank, other types of data pairs (i.e., W-D), and +W’ and +E for calibration) increase the model performances. Also, DefBERT with best selection shows the largest improvement. The results indicate that the setup of DefBERT learns the sensespecific patterns between definitions and examples. Moreover, ConceptNet performs worse than most of the BERT-variants, showing that context is an important factor in this task.

heading: 5.3 Downstream Task 1: Word-Similarity
text: Setup. Word similarity tasks can be used to evaluate word representations. They make use of Spearmann correlations to assess agreement between human ratings and computational representations of the similarity between word pairs. We use the evaluation tasks–WordSim (Finkelstein et al., 2001; Agirre et al., 2009), RareWord (Luong et al., 2013), MEN (Bruni et al., 2012), SemEval (CamachoCollados et al., 2017), SimLex (Hill et al., 2015), and SimVerb (Gerz et al., 2016). For DefBERT, we choose the best selection model in DefRank. Note that there is no additional training on the word similarity datasets.
Results. Table 7 shows performances on the word similarity tasks. The other embeddings, except for DefBERT show poor performances. Additional masked language modeling fine-tuning increases the performance only a little. We conjec-
ture that word similarity/relatedness tasks are very challenging for pretrained and contextualized models because no context is given (see §6 for further discussion). The result is the same as what we found in our preliminary distance analysis on word-definition pairs. On the other hand, DefBERT largely closes the gaps among word, definition, and usage, which leads to significant improvements from BERT in all the datasets.

heading: 5.4 Downstream Task 2: Short Text
text: Classification
Setup. As we mentioned in §1 and showed in the previous experiments §5, BERT embedding for a word or short text did not make good representations. In order to generalize the effect of our integration, we employ text classification datasets– TREC (Hovy et al., 2001), SST-2 (Socher et al., 2013), and IMDB (Maas et al., 2011). All the datasets are relatively small, and the text length is short in TREC and SST-2, whereas IMDB is rather long. We report IMDB performance to show the performance of long text.
As the original paper did, we use a [CLS] token at the last hidden layers. The hyperparameters are 2e-5 for learning rate, 32 for mini-batch size. We use Adam optimizer (Kingma and Ba, 2014). If the dataset does not have a validation set, we assign 15% of the training set and use them for earlystopping. The maximum length of tokens is 512.
Results. We present the performance of text classification in Table 8. Compared to other methods, DefBERT shows comparable performance with other baselines. Although the performance gap is small (we guess that the baseline is already strong), DefBERT shows the best performance on the shortest dataset TREC, which has only maximum 37 words (by space). On the other hand, IMDB has approximately maximum 3000 words. Though GlossBERT is also fine-tuned by external data (specifically, gloss), the result indicates that word disambiguation tasks are not related to representing a single word or short sentence.

heading: 6 Conclusion and Further Discussion
text: We present a novel way of combining pretrained contextualized representations and human-written definitions from a dictionary. We first collect definitions and examples from an online dictionary Oxford+. Our analyses with the dictionary show that BERT’s representations do not incorporate human-written definitions. Motivated by the findings, we develop a new representation DefBERT, by constraining BERT to human-written definitions in the dictionary. In the experiments, we first proposed definition ranking (DefRank) and sense disambiguation tasks (SenseRank) and DefBERT outperforms other baselines. We also presented the effectiveness of DefBERT in downstream tasks: word similarity benchmark and short text classification tasks.
One of the contributions of this paper is to make researchers revisit the old and traditional resource, dictionaries. While resources, including synonyms, antonyms, and other relations, are widely used to improve models as a constraint, dictionaries are less frequently used. However, the dictionary is the basic form of word semantics and is a relatively objective resource compared to relational resources.
Furthermore, word-related resources are hard to align with pretrained language models because the weights are dynamic according to contexts. Therefore, pouring resources can occur catastrophic forgetting that the information previously trained disappears. On this problem, we suggest a potential approach to enhance semantics on the pretrained weights, maintaining the nature of contextualized encoder.

heading: 7 Limitations
text: The performances except for the proposed tasks. We presented the result of neologism and the performances on two downstream tasks (i.e., word similarity task and short text classification), which are closely related to the understanding of word semantics. The selected downstream tasks are challenging for the contextualized models; they can use only a few contexts to make a representation.
The performance in general benchmarks (e.g., GLUE) is almost the same as the vanilla BERT because our model suffers catastrophic forgetting while learning definition information. Sophisticated modeling and training processes to overcome the problem could be interesting future work.
The use of other models Other pretrained models like RoBERTa could be a base model of our method (e.g., DefRoBERTa). However, we think that BBPE tokens scarcely have semantic meanings, which makes it hard to find appropriate tokens to inject definition information. Therefore, integrating human-written definitions with other types of tokens (e.g., Byte-Pair Encoding and Byte-level BPE) is also a future direction.
The use of all the loss function& Collect more definition data. Presenting more experiments with other models, other collections of definition data, and other loss functions will further support our idea. Nevertheless, we want to show the performances with the widely-used basic model of pretrained language models (i.e., BERT), using definition data from the previous work, with various loss functions (e.g., W-D, D-E, [+W’], [+E]) as many as possible. A fine-grained combination of all the loss functions could make further improvements.

heading: Acknowledgement
text: The author would like to thank previous co-workers who discussed this idea long ago, including reviewers in several rounds of submission. Also, PAUST gave helpful advice on experimental techniques and distributed software engineering. Lastly, I am grateful to Alice Lee for help in writing this work.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
7 A2. Did you discuss any potential risks of your work? Overall process has no potential risk
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 5
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? We used well-known BERT.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Sophisticated modeling and training processes to overcome the problem could be interesting future work. The use of other models Other pretrained models like RoBERTa could be a base model of our method (e. g. , DefRoBERTa). However, we think that BBPE tokens scarcely have semantic meanings, which makes it hard to find appropriate tokens to inject definition information. Therefore, integrating human-written definitions with other types of tokens (e. g. , Byte-Pair Encoding and Byte-level BPE) is also a future direction. The use of all the loss function& Collect more definition data. Presenting more experiments with other models, other collections of definition data, and other loss functions will further support our idea. Nevertheless, we want to show the performances with the widely-used basic model of pretrained language models (i. e. , BERT), using definition data from the previous work, with various loss functions (e. g. , W-D, D-E, [+W’], [+E]) as many as possible. A fine-grained combination of all the loss functions could make further improvements