Balaur: Language Model Pretraining with Lexical Semantic Relations
Authors: Andrei Mircea
Jackie CK Cheung
Venue: EMNLP 2023 Findings (2023)
Keywords: language model
lm
pretraining
lexical semantics
lexical semantic relations
hypernymy
semantic specialization
wordnet

Abstract: Lexical semantic relations (LSRs) characterize meaning relationships between words and play an important role in systematic generalization on lexical inference tasks. Notably, several tasks that require knowledge of hypernymy still pose a challenge for pretrained language models (LMs) such as BERT, underscoring the need to better align their linguistic behavior with our knowledge of LSRs.
In this paper, we propose Balaur, a model that addresses this challenge by modeling LSRs directly in the LM's hidden states throughout pretraining. Motivating our approach is the hypothesis that the internal representations of LMs can provide an interface to their observable linguistic behavior, and that by controlling one we can influence the other. We validate our hypothesis and demonstrate that Balaur generally improves the performance of large transformer-based LMs on a comprehensive set of hypernymy-informed tasks, as well as on the original LM objective. Code and data are made available at https://github.com/mirandrom/balaur

BALAUR: Language Model Pretraining with Lexical Semantic Relations
Andrei Mircea
Jackie C. K. Cheung
Mila/McGill University
Mila/McGill University
mirceara @mila. quebec
jcheung @cs. ecs.mcgill.ca
Abstract
Language Model
BALAUR
Token
Contextual Enbeddiings
Transforred Erbeddings
BALAUR Constraaints
Lexical semantic relations (LSRS) character- ize meaning relationships between words and play an important role systematic generaliza- tion on lexical inference tasks. Notably, sev- eral tasks that require knowledge of hypernymy still pose challenge for pretrained language models (LMs) such BERT, underscoring the need better align their linguistic behavior with our knowledge of LSRs. In this paper. we propose BALAUR, a model that addresses this challenge by modeling LSRs directly in the LM's hidden states throughout pretraining. Motivating our approach is the hypothesis that the internal representations of LMS provide an interface to their observable linguistic behav- ior, and that by controlling one we can influ- ence the other. We validate hypothesis and demonstrate that BALAUR generally improves the performance of large transformer-bant based LMs comprehensive set of hypernymy- informed tasks, as well as on the original LM objective. Code and data are made available at github. com/m /mi randrom om / laur.
Inputs
the
BALAUR Heads
Hypeary vel Lok space
young
LM Encoder
brown
young pige group!
fox
dog (untmnal
jumpe
over
the
Hyporfyyyy vec Los space
mask
Inass] SHIDPY
LM Vocab
Token Poediction Objective
SALATIR Concept
Enberiitings
Esowddd ogl
acreen
ligh group!
dog
dog (msak]
janimal
(agedd)
mesidants
reaidents
48ja
Icorgi
tyounsl bid
Figure BALAUR learns transform LM hidden states (contextual token embeddings) in LSR-specific vector spaces, modeling relatedness as similarity constraints.
they provide a dimension along which LMs can perform class-based generalization to new events in sample-efficien manner. For example, knowing that a Xoloitzeuintli is dog breed allows model to infer many plausible properties about them, even without many occurrences of this word in training. This improved modelling of LSRSs could in turn improve generalization and data efficiency in LMs. We propose BALAUR (Figure 1), an approach to Transformer LM pretraining which directly mod- els LSRs in the latent representations of the LM. BALAUR consists of a modular neural architecture with multiple heads, each head modeling distinct LSR (e.g. hypernymy, synonymy, antonymy). Each LSR is modelled as learned vector transfor- mation, with constraints injected in the resulting vector space reflect structural properties of the LSR. Concretely, a BALAUR head transforms con- textual token embeddings (e.g. fox) and static concept embeddings (e.g. (canine)) such that related pairs are similar in the corresponding LSR vector space (c.g. hypemnyyy..
1 Introduction
Pretrained language models (LMs) trained on ever- increasing compute and data have achieved state- of-the-art performance on a wide variety of NLP benchmarks. However, they are still known to struggle certain tasks, notably those involving reasoning and world knowledge (Liu et al., 2021). In particular, previous work has found that these models make errors on tasks involving lexical se- mantic relations (LSRs) such as hypernymy: no- tably cloze completions such as "A fox is a type of where the hypernym "canine" is a valid com- pletion (Ettinger, 2020); and monotonicity-basee inferences such as "Bob saw fox" entailing "Bob saw canine" (Geiger et al., 2020).
This represents important limitation of current LMs. Besides the specific classes of inferences in which they are involved, LSRs are crucial because

---

Previous work has proposed to integrate LSRs with LMs, for example augmenting LM pretraining with hypernym class prediction (Bai et al., 2022). However, these approaches do not encode known structural properties of LSRs, modeling these in- directly without accounting for c.g. hypernymy being transitive. antisymmetric with hyponymy. In contrast, our method draws inspiration from work on semantic specialization, which models LSRs as constraints in the vector space of static word embeddings (e.g. Mrkšić et al., 2017). We show how similar insights can be applied to mod- ern Transformer LM architectures such that the resulting LM's hidden states have inductive biases useful for LSR-informed tasks. More broadly, we demonstrate that our method creates an interface between the model's 's latent representations and its linguistic behaviour, and that we can control the latter by injecting LSR information into the former.
2 Related Work
2.1 LM Pretraining with Hypernymy
Incorporating LSRs into LM pretraining, particu- larly hypernymy, has been approached from differ- ent angles. Lauscher et al. (2020) create supple- mental training instances consisting of two words, where the model must predict whether they are semantically related using the next sentence predic- tion objective of Devlin et al. (2019). In contrast to our work, this approach combines synonymy, hypernymy and hyponymy into one relation and requires a large number of additional training ex- amples during pretraining. Levine et al. (2020) avoid the need for additional training data by modi- fying the LM objective to jointly predict word's supersense in addition to the word itself, while Bai et al. (2022) create a curriculum where LMs learns to predict a word's hypernym before predict- ing the word itself. However, these methods aim to improve LMs more broadly and do not target specific hypemymy-inffrmed tasks or attempt to disentangle hypernymy from other relations during pretraining. In contrast, our work presents a novel method based on semantic specialization, where hypernymy and other LSRRs are jointly modeled in the latent representations of LMs to improve per- formance on targeted evaluations of hypernymy. To the best of our knowledge there has been no work successfully using finetuning to incorporate musk-agnostic knowledge of LSRs into LMs such as BERT, and adapting our method to finetuning consistently led catastrophic forgetting.
We evaluate BALAUR on several tasks which require knowledge of hypernymy. These include cloze completion, monotonicity-informed natural language inference (NLI) and finetuning effi- ciency on the previous two tasks in a transfer learn- ing setting. We find that BALAUR generally im- proves performance these hypernymy-informed tasks, as well as the on the original LM objective.
Contributions
We introduce BALAUR, method aimed at im- proving the generalization of pretrained language models on tasks involving lexical semantic rela- tions, specifically hypernymy and hyponymy.
By modeling hypernymy and other lexical seman- tic relations in the hidden states of language models during pretraining, BALAUR consistently improves performance on language modeling and in com- prehensive set of ceyperyymyiifforme tasks.
2.2 Lexical Semantic Specialization
Prior to the advent of pretrained LMs, distributional word embeddings were augmented with LSRs us- ing a class of techniques known as semantic spe- cialization (Yu and Dredze, 2014; Glavaš and Vulić, 2018; Vulić et al., 2018, i.a.). These methods learn transformation of the original distributional vector space that better captures relational knowl- edge such LSRs, modeling relatedness as con- straints in the resulting vector space. Our work most closely resembles that of Arora et al. (2020) which learns multiple cllation-specffic subspaces the original vector space of word embeddings. and Gajbhiye et al. (2022) which use BERT-basc bi- encoders to predict whether commonsense concept- property pairs related based on the similarity of their transformed embeddings.
Our evaluation brings together previous work on evaluating hypernymy in language models, pro- viding comprehensive view of how well hyper- nymy is captured the linguistic behavior of LMs on tasks involving prompt completion, natural lan- guage inference and transfer learning.
Finally, as part of this evaluation, we create and HYPCC, a dataset of hypernymy-inforne cloze complction prompts improving on previous datasets with a better coverage of hypernymy and hyponymy. We also identify important challenges in creating such datasets from lexical resources.

---

2.3 Evaluating Lexical Semanties in Models There have been various approaches at evaluating how well a model's representations capture lexical semantics. In particular, there exists a plethora of intrinsic evaluations which probe representa- tions directly, particularly for hypernymy relations. These typically take the form of relation prediction between term-pairs (e-g. Bordea al., 2015; San- tus et al., 2015; Bordea et al., 2016; Shwartz et al., 2016) with variants such as hypernym wordsense prediction (Espinosa-Ande Anke et al., 2016), graded relation prediction (Vulić et al., 2017), hypernym retrieval from corpora (Camacho-Colladee et al., 2018), and probing of LM representations (Vulié et al., 2020). However, such probing methods suffer from several limitations in the context of LMs, discussed by Rogers et al. (2020). Notably, probes tell us only what information can be recov- ered from LM representations, not how (or even if) the LM uses it in practice (Tenney et al., 2019). Because of these limitations and the fact that our approach explicitly models LSRs in LM represen- tations, we do not conduct such intrinsic evalua- tions. In contrast, extrinsic evaluations measure LM performance on downstream inference tasks requiring knowledge of hypernymy (e.g. Geiger et al., 2020; Rozen et al., 2021). A fundamental challenge for such approaches is understanding whether performance is attributable to a model's learned representations or to finetuning. To ad- dress this, recent work has evaluated pretrained LMs in zero-shot prompt completion and finetun- ing efficiency (Talmor et al., 2020). Building on this work Ettinger (2020), Ravichander et al. (2020) and Hanna and Mareček (2021) demonstrate that modern LMs fail to generalize systematically on zero-shot cloze-style prompts informed by hyper- nymy. Our paper builds on this line of work in two ways. First, we present HYPCC, dataset of cloze prompts with better coverage of hypernymy and hyponymy. Second, bring together inference and prompt completion tasks into comprehensive evaluation of hypernymy in LMs.
tempted inform lexical semanties with distribu- tional semanties. Of particular relevance to our work the extraction from corpus data of hyper- nymy (Caraballo, 1999; Snow et al., 2004) and meronymy (Poesio et al., 2002) relations, typically based on Hearst patterns (Hearst, 1992). Simi- larly, Mohammad et al. (2008) leverage the co- occurrence hypothesis (Charles and Miller, 1989) to identify antonymy. Bridging the gap between lexical and distributional semantics, there is work like Agirre et al. (2009) which combines both ap- proaches, noting that while distributional methods help alleviate out-of-vocabulle resources, they struggle to distinguish semantic similarity from relatedness. Our work attempts to address this issue, explicitly modeling LSRs in LM representations so they can be distinguished.
issues in lexical
3 The BALAUR Head Architecture
In this section, we present the neural architecture of BALAUR heads, describing how they model LSRs in the hidden states of LMs, how this is translated into optimizable loss function, and how they interface with LMs during pretraining.
Assumptions We take the term (neural) language model (LM) to refer to a neural network that pre- dicts a token given its context, including commonly used masked LMs like BERT (Devlin et al., 2019) and auto-regressive LMs like GPT (Radford et al., 2018). These LMs encode a sequence of tokens into latent representations known as hidden states or contextualized token embeddings T, using these as inputs to a classification head that predicts the target token. Meanwhile, we represent a lexical semantic relation R as a set of related lexical item pairs (X; - X,) C R, noting that LSRs can be directed, e.g. (corgi dog) resides in hypernymy while (dog corgi) resides in hyponymy.
3.1 Modeling Lexical Semantic Relations
Our goal is model LSRs in T. where T; the LM hidden state for token i. However, modeling LSRs between pairs of in-context tokens is challenging because related tokens often do not co-occur in the same context. One solution would be to model LSRs for related token pairs across different con- texts, however this becomes intractable as the num- ber of possible context pairs grows combinatorially. Instead, we model LSRs (T; -> C₃)) E R, where C' set of comtext-independent concept embeddings learned during pretraining. Modeling
2.4 Lexical and Distributional Semanties
More broadly, these lines of work explore the inter- play between lexical and distributional semantics, specifically how the first (in the form of LSRs) helps inform the second (in the form of training and evaluating LMs or word embeddings). Con- versely, there is rich body of work that has at-

---

LSRs between token-concept pairs not only ad- dresses the issue of related token co-occurrence, but also enables us to capture concepts that would otherwise fall outside the model's vocabulary.
3.3 Interfacing with Language Models
During LM pretraining, is computed in the for- ward pass and used as input to the LM's classifica- tion head for token prediction. Each BALAUR head also takes T as imput, along with concept embed- dings C and relation-specific sets of indices (i. j) where i indexes T and the corresponding token in the training batch, while j indexes concept in C related to T: by the corresponding relation R. Each head then computes its loss L and these are averaged before being added to the LM loss.
For a given relation R, the corresponding BAL- AUR head learns to transform T and C such that related token-concept pairs are similar in the result- ing relation-specifie vector space. We implement these as two-layer neural networks with GELU ac- tivation functions (Hendrycks and Gimpel, 2020):
T 8 (GELU x WRT + BBRT)) 1xd
1xb bxb
dxb
1x6
(I)
Method
C Wo (GELU C x WRC. + BR,)), dxb 1xb
exb 5xb
exd
In this section, we detail our methods for LM pre- training with BALAUR, using LSRs and concepts extracted from WordNet. We also present the ar- chitecture and hyperparameters for the LM in our experiments, a variant of BERTLARGE suitable for academic budgets. While our experiments are lim- ited to masked language modeling and LSRs, our method can be extended to autoregressive language modeling and other forms of relational knowledge.
where & and the number of token and con- cept embeddings, and d and b are their original and transformed dimensionalities. W", WR.,, W B B are learned projection and bias matri- ces that parameterize the transformations for R. These learned transformations enable BALAUR heads to model and disentangle multiple LSRs in the vector space of T, i.c. ensuring token's 's re- lated concepts be predicted from its contextu- alized embedding, distinguishing across different relations. Moreover, by parametrizing LSRS as learned transformations, our approach can model LSRs inductively; i.e. generalize from instances of related pairs to a functional representation that can extrapolate to unseen pairs (Vulić et al., 2018).
4.1 Extracting LSRs from WordNet
As a first step, we extract related token-concept pairs for hypernymy hyponymy, antonymy and synonymy from WordNet's noun hierarchy (Miller, 1995). To do this, we begin by mapping the model's vocabulary to corresponding WordNet synsets (re- ferred to throughout this paper as concepts) us- ing NLTK (Bird and Loper, 2004), For example, the token dog maps to the concept of pet dog hot dog (frank.n. nn. 02).
3.2 Optimizing a BALAUR Head
To translate our similarity constraint into learn- ing objective, we adapt the supervised contrastive loss of Khosla al. (2020) which maximizes the inner product similarities S between each related token-concept pair (i, (), while minimizing it for unrelated pairs (i, k). Optimizing this loss thus enables us to predict a token's related concepts from its contextual embeddings, encoding the cor- responding LSR in the LM's hidden states:
(dog n. 01),
Next, using the resulting set of concepts, we extract related concept-conceye pairs from Word- Net and convert these to token-concept pairs. For example (canine.. n. 01) is a hypernym of (dog n. 01). while (sausage .n.ge.n.00 01) is a hypernym of (frank n. 02); but both are ex- tracted hypernyms of the token dog. To im- prove coverage of WordNet, hop hypernymy up to depth
consider multi- such that e.g. (2) (animal, n. 01) is extracted as a hypernym of both dog and c ne. The resulting set of token- concept pairs contains 15, 612 unique concepts. Manual sampling and inspection of the resulting pairs revealed several known issues associated with WordNet, including inaccurate lemmatization (Mc- Crae et al., 2019), and too fine-grained word senses (McCarthy, 2006), further discussed in $7. To help address these potential sources of noise in BAL-
1
exp (S
L
log
R
Σ exp (SRR) ksc
(1,J)
SR - T x (CR)T
(3)
Ixe txb
exb
where indexes the set of token embeddings, while j and & index the set of concept embeddings.

---

AUR, we filter tokens, concepts and token-concept pairs using the criteria described in A.I.I.
els that were pretrained with and without BALAUR heads. Throughout this section, we refer to these BERT+BALAUR and BERT (OURS) respectively. Drawing from the observation that LSRs con- strain language production, understanding, and learning in humans (Nagy and Gentner, 1990; Fass, 1993); we assemble gauntlet of hyperyyy-- informed evaluations that broadly mirror these three capabilities, providing comprehensive view of different ways LMs can capture hypernymy their linguistic behavior:
4.2 Incorporating BALAUR into Pretraining We then use these token-concept pairs to optimize a BALAUR head for each LSR throughout LM pretraining. First, we randomly initialize C' as a 15612 x 768 embedding layer. Second, training examples are annotated with relation-specifer sets of indices (i,j). where i indexes token in the training sequence and j indexes related concept in C. Lastly, the hidden states T are computed in the LM's forward pass, and fed into each BAL- AUR head, along with C and the sets of indices (1,j), to compute _ as described in $3.3, using a transformed dimensionality b of 768. To prevent se- quentially iterating over BALAUR heads, we adopt the parallelization technique from multi-head atten- tion (Vaswani et al., 2017). Specifically, we learn one set of transformations (1) but multiply b by RR] so the resulting transformed vector space can be partitioned across relations. To reduce memory overhead, we only input the subset of containing LSRs into BALAUR, reindexing on this subset.
Language modeling ($5.1): to complement our evaluation, we verify the effect BALAUR has the original LM objective, with particular attention performance on tokens with hypernymy relations. Prompt completion ($5.2): models must predict the correct token given cloze-style prompt de- scribing a hypernymy or hyponymy relation, e.g.
"a dog is a type of [mask]".
Monotonicity NLI ($5.3): models must predict whether sentence entails another, when hyper- nymy and monotonicity determine entailment, e.g. "drive a taxi" entails "drive car".
Finetuning Efficiency ($5.4): we compare how efficiently models transfer-learn when finetuned on the two previous tasks, disambiguating what is learned during pretraining versus during finetuning.
4.3 Language Model Pretraining Setup
Our LM architecture, pretraining procedure, and hyperparameters are based on 24hBERT (Izsak et al., 2021) which enables rapid pretraining with limited resources, while reaching comparable per- formance with the original BERT models (Devlin al., 2019). Specifically, we pretrain a BeRTLarge architecture to perform masked language modeling (MLM) on 128-token scquences for 25,000 steps with a batch size of 4,096 and using 16-bit preci- sion. We optimize using AdamW (Loshchilov and Hutter, 2019) and peak learning rate of 2e-3 with warm-up over the first 1,500 steps and linear de- cay. The pretraining data snapshot of English Wikipedia from 2022-03-01, and Book Corpusonunnnnn (Bandy and Vincent, 2021), with 0.5% withheld for validation. These datasets were downloaded from and preprocessed with the datasets library (Lhoest et al., 2021) which provided licenses such as CC-BY-SA 3.0 and GFDL for Wikipedia.
5.1 Language Modeling
In Table I, we see that incorporating BALAUR into the LM pretraining procedure of Izsak et al. (2021) increases both negative log likelihood (NLL) and mean reciprocal rank (MRR) for the original masked language modeling objective. We observe similar improvements when masking random to- kens as when masking only tokens with LSRs, in- dicating the improvements introduced by BALAUR extend beyond the modeling of LSRs. Lastly, we note that improvements in the original MLM ob- jective begin early and are consistent throughout pretraining, as seen in Figure 2.
RANDOM TOKENS
LSR TOKENS
MODEL
NLL
MRR
NLL
MRR
BERT (OURS)
1.659
0.733
3.359
0.482
BERT+BALAUR
1.587
0.743
3.201
0.503
5 Evaluating BALAUR Language Models
A(%)
4.3
1.4
4.5
4.1
this section, we evaluate whether BALAUR heads can improve performance on tasks that are in- formed by LSRs, specifically hypernymy and hy- ponymy. To this end, we compare BERTLARGE mod-
Table I: Validation MLM performance, shown for mask- ing random tokens and for only masking tokens with ISRRs (i.e. modeled by BALAUR during pretraining).

---

3.5
Results and Discussion In Table 2, we find that BALAUR improves performance on hypernymy- informed prompt completion across settings and metrics, even outperforming the original BERTLARGE implementation of Devlin et al. (2019). However, we note that both of our models strug- gle with Acc @ when compared to BBERT despite general improvements of BALAUR over our baseline. A closer inspection of model predictions reveals that, similar to findings of Ettinger (2020), models often repeat the hypernym or hyponym in the context (c.g. predicting "a daisy is a type of daisy"). In Table 3, we find that our baseline pre- training procedure exacerbates this problem, ex- plaining the discrepancy in Acc @ performance. Moreover, qualitative analysis of selected clozes similar to Arora al. (2020), shown in Ta- ble 4, suggests that BALAUR better disentangles hy- pernymy from other forms of semantic relatedness, These results agree with Agirre al. (2009), who showed similar improvements combining lexical and distributional semantics in word embeddings.
BERT BERT (OURS)
Eval MLM Loss
3-
2.5-
5k
10k
1.kc
20%
25k
Train Steps
Figure 2: Validation MLM loss throughout pretraining.
5.2 Prompt Completion
Task Description We create HYPCC: a dataset of cloze-style prompts taking the form "In the context of hypernymy, a(n) is a type of y." where z, y are hyponym-hypernyen pairs of tokens in our model's vocabulary, and either is masked out be predicted by the model. This evaluation builds on the work of Ettinger (2020) and Ravichander et al. (2020), which draws from human psycholinguistic tests to create cloze prompts. In contrast to previous work, our evaluation includes hypernyms beyond Fischler categories, evaluates hyponym prediction, consid- ers tokens with multiple word senses, and includes clozes with multiple valid completions. The result- ing dataset contains 17,556 hyponym-hypernym pairs; 5,217 hypernym prediction prompts; and 4,115 hyponym prediction prompts. We report ad- ditional details for the creation of HYPCC in A.1.2, and discuss several limitations in $7.2.
It is also interesting to note that BALAUR spreads its probability mass more evenly across predictions, better capturing the one-to-many nature of hyper- nymy relations. However, we observe that many of the seemingly valid completions are actu- ally gold-standard completions HYPCC. This is because HYrCC considers only direct hypernymy relations WordNet, while several completions are indirect hypernymy relations or not in WordNet. We further discuss these limitations in $7.
Evaluation Method In line with previous work, models are evaluated on HYPCC in a zero-shot manner (i.c. using masked language modeling to complete the cloze prompt); and performance measured with accuracy and mean reciprocal rank (MRR) for both the open and closed vocabulary settings. In the closed setting, metrics are calcu- lated using only the set of possible hypernyms or hyponyms in HYrCC, while the open setting con- siders the model's 's entire vocabulary. Importantly, these metrics are adjusted to account for multi- ple valid completions in a prompt: ignoring other valid completions when computing a completion's rank (i.e. if a model's top three predictions are valid, the average accuracy will be 100% instead of 33%). To prevent a skewing of results by prompts with a larger number of completions, metrics are first averaged over completions, then averaged over prompts.
CLOSED VOCAB
OPEN VOCAB
MODEL
@cc@ 1/5 MRR
Acc@ 1/5
MRR
HYPERNYM PREDICTION
LBERT
/
0.092
1.78 / 11.77
0.071
BERT (OURS)
1.18/1861
0.121
0.88 / 14.72
0.080
BERT+BALAUS
5,31/ 19.65
0.128
1.60
15.44
0.089
HYPONYM PREDICTION
1BERT
3.60 14.95
0.097
2.76 / 12.87
0.083
BERT (OURS)
2.69 12.22
0.081
2.03 10.65
0.069
BERT+BALAUR
3.49/17.99
0.110
1.85 14.56 0.084
Table 2: Zero-shot results on HYPCC. BALAUR gener- ally improves performance across metries when com- pared to baseline BERT model with the same 24TBERT pretraining procedure, as well as the published check- point of t (Wolf et al., 2020). More extensive comparisons are included 1A.2.1.

---

MODEL
HYPERNYM REPETTTION
HYPONYM REPETITION
Results and Discussion In Table 5, we replicate the results of Geiger et al. (2020), finding that mod- els finetuned on SNLI only generalize to PMoNLI but fail completely on NMONLI. Unexpectedly, we find BALAUR improves both SNLI and PMoNLI performance in this setting, suggesting some exam- ples in SNLI also benefit from the representations learned with BALAUR pretraining.
tBERTLARGE
50.17
47.08
BERT (OURS)
87.81
64,20
BERT+BALAUR
69.59
69.38
Table 3: Rates of repetition on HYPCC. BALAUR re- duces repetition for hypernym prediction, with compa- rable rates of repetition for hyponym prediction.
the context of hypernymy, church is type of [mask].
MODEL
SNLI PMONLI NMONLI
BERT
church
religion
structure
building
worship 0.86
SNLI FINETUNING ONLY
(OURS)
74.78
2.83
1.25
1.11
BERT (OURS)
85.44
65.51
0.50
BERT+ BALAUR
church
building 21.45
structure 15.57
place 2,41
object 1.83
BERT+BALAUR
86.49
76.92
0.10
27.33
SNLI MONLI FINETUNING
In the context of hyperyymy a [maak] is type of poem.
BERT (OURS) 85.43
48.90
BERT (OURS)
pocm
poet
poctry 0.55
verse 0,50
word
BERT+BALAUR
86.38
56.50
91.72
0.84
0.35
BERT+
verse
poctry 2.47
Table 5: SNLI and MoNLI accuracies.
BALAUR
66,23
3,80
3.46
1.67
In the context of hyperyymy. volcano type of [maak].
However, we conversely find that BALAUR de- grades performance on the withheld test set of NMoNLI. While BALAUR may help LMs better capture hypernymy, the fact that it does not count for negation may help explain this result. Furthermore, visualizing performance across seeds in $A.3.3, we observe markedly larger variance on NMoNLI compared to SNLI and PMoNLI, making this result more difficult to interpret reliably.
BERT
volcano
lava
come
ruck 0.94
eruption 0.88
(OURS)
88.30
1.59
BERT+
volcano
mountain 13.27
structure 2.55
object 0.80
rock 0.69
BALAUR
60.54
Table Top-5 completions and probability percentages for selected clozes, showcasing how BALAUR can help disentangle hypernymy from other forms of semantic relatedness (related but invalid completions are bolded).
5.3 Monotonicity NLI
5.4 Finetuning Efficiency
Task Description Our second evaluation is taken from Geiger et al. (2020), who create MoNLI: a challenge NLT dataset where entailment is deter- mined by hypernymy. For instance, "A man is talking to someone in taxi" entails "A man is talk- ing someone in a car". While models finetuned on SNLI (Bowman al., 2015) perform well on such examples, they fail to generalize on examples where negation reverses entailment. For instance, "A man is not talking to someone in a car" now entails "A man is not talking to someone in a taxi". MoNLI is divided into PMoNLI and NMoNLI to distinguish between positive and negated examples.
Task Description Our final evaluation reframes $5.2 and $5.3 not in terms of zero-shot or final performance, but in terms of performance through- out the finetuning of pretrained model - pro- posed by Talmor al. (2020) in oLMpics. This ap- proach was originally proposed because finetuning pretrained LMs makes hard to disentangle what is captured in the pretrained representations from what is learned during finetuning. A key assump- tion underlying our use of this evaluation is that models which better capture hypernymy in their pretrained representations will be finetuned more efficiently (i.e. with better finetuned performance relative to finetuning steps, throughout finetuning).
Evaluation Method We adopt the evaluation pro- cedure of Geiger et al. (2020), reporting test set ac- curacies for models finetuned on SNLI, and models also finetuned on MoNLI. We follow the NLI fine- tuning procedure of ERT (Izsak et al., 2021) which model is based. However, we found that performance is sensitive to random seeds, so we report results averaged across seeds.
Evaluation Method We finetune our models us- ing the hyperparameters and finetuning procedures from Talmor et al. (2020) for prompt completion, and from Geiger et al. (2020) for MoNLI. This in- cludes freezing model parameters for the prompt completion task (leaving only the language mod- eling head unfrozen); while unfreezing the entire model for the NLI task. We perform 5-fold cross-

---

validation with a 20% split, and average validation set results. Importantly, the splits are systematic to ensure that no hypernyms or hyponyms occur in both train and validation sets.
0.15
BERTI BALAUR
Intruston Rate
RERT (OURS)
0.1-
0.05
Results and Discussion (Prompt Completion)
In Figure 3, we see that BALAUR'S improvement on hypernym prediction extends throughout finetuning, indicating better transfer learning abilities. We show similar results for the hyponym subset of HYPCC in $A.2.2.
10
100
Class Frequency HypCC
Figure 5: Average intrusion rate and frequency of classes in the final models finetuned on the hypernym prediction subset of HYPCC.
Instead, in Figure 5, we find that the intrusion rate of a class grows with its frequency in the finetuning dataset. Given that intrusion rates in- crease with finetuning and that frequent classes have higher intrusion rates, this suggests that LMs struggle to discriminate single token differences in prompts, and instead conflate learning signal across prompts with more frequent classes dominating.
MRR (Hyporym))
0.15-
BERT+ BALAUR
(OURS)
20k
40k
60%
Number training examples
Results and Discussion (MoNLI) In Figure 6, we observe similar results for MoNLI, indicating that BALAUR improves finetuning efficiency. In con- trast to the results in $5.3, we also observe in Ta- ble 6 that BALAUR improves final performance on systematic validation splits for both PMoNL and NMONLI These improvements are consistent even when stratifying by BALAUR coverage of the hy- pernym and hyponym in given MoNLI example, i.e. whether or not BALAUR models hypernymy or hyponymy relations for these tokens.
Figure 3: Average open-vocab MRR throughout fine- tuning on the hypernym prediction subset of HYPCC.
However, it is puzzling that performance re- mains relatively low despite extensive finetuning. A closer look at the outputs of the final model reveals that many of the erroneous entries in the model's top-10 open vocabulary predictions were in fact other classes in the HYPCC dataset (i.e. tokens from the closed vocabulary). In Figure 4, we quan- tify the class intrusion rate as the proportion of top-10 predictions which are both erroncous and a class in HYPCC, finding that it increases signifi- cantly throughout finetuning.
0.9
Acouracy (MoNLL))
0.8
0.7-
Class Intrasion Rate
0.8
0.6-
BAIT+BAI.AA
0.5-
BERT (OURS)
5k
10k
0.7-
Number of training examples
BERT+ BALAUR (OURS)
0.65
Figure 6: Average accuracy throughout finetuning.
20k
40k
60k
Number of training examples
Overall Accuracy
Accuracy by BALAUR Coverage
Hyper+Hypo
Hyper Only
Neiteer
Figure 4: Average class intrusion rate throughout fine- tuning on the hypernym prediction subset of HYPCC.
PMONLI
BERT (OURS)
82.14
80.69
85.07
58.33
BERT+BALAUR
86.78
86.19
88.27
72.22
One possible explanation is that models learn to predict indirect hypernyms or hyponyms not accounted for in HYPCC, similar to examples in Table 4. However, a manual inspection of model predictions showed that this was not often the case.
NMoNLI
BERT (OURS)
80.31
78.00
81.81
BERT+BALAUK
93.01
91.44
94.02
Table 6: Final performance on MoNLI subsets, averaged over five systematic validation splits and stratified by coverage (subsets with no coverage are omitted).

---

6 Conclusion
the compute and data required. While this approach seems more practical and accessible, in practice we found that effectively finetuning pretrained LMs to improve their linguistic behavior across a range of tasks without loss of generality is more difficult from an optimization perspective than pretraining different model from scratch.
this work, we set out to align the linguistic be- havior of LMs with our knowledge of LSRs and im- prove their performance on hypernymy-informed tasks. We presented BALAUR, an approach that aims to guide the linguistic behavior of LMs in such way by modeling LSRs directly in their hidden states throughout pretraining.
While previous work has succeeded in finetun- ing pretrained LMs without catastrophic forgetting (e.g. for downstream tasks (Chen al., 2020), domain adaptation (Gururangan al., 2020), and de-biasing (Gira et al., 2022)), these do not address the issue of incorporating external knowledge pretrained LMs to guide their linguistic behavior across variety of tasks. To the best of our knowl- edge, achieving this remains open question.
Underlying this proposed approach was the hy- pothesis that LM latent representations can provide an interface to their linguistic behavior, and that controlling one can help guide the other. To verify our hypothesis, we characterized the effect of BAL- AUR on series of evaluations, targeting several distinct ways in which LMs might capture hyper- nymy in their linguistic behavior.
Our findings show that BALAUR can robustly improve performance on diverse hypernymy- informed tasks, validating the effectiveness of our method while supporting our original hypothesis. Notably, we demonstrated that BALAUR also im- proves performance on the original language mod- cling objective, indicating method's improve- ments not limited ampernymy-informen tasks and can extend to more general linguistic behav- ior. However, we found that aligning the linguistic behavior of LMs with BALAUR still poses several challenges. We further discuss these limitations in $7 and outline implications for future work.
7.2 Noise, bias and coverage in WordNet
When using knowledge bases such as WordNet, is important account for their inherent limitations. In particular, we identify three prevalent issues in WordNet that can negatively affect what LMs learn in our experiments.
First the problem of noise. Due to issues with lemmatization, word sense granularity and idiomaticity, we found many questionable relations being extracted when creating training data for BALAUR and examples for HYPCC. For example, find that the token cat is lemmatized the concept -eat--'-nine-taille 01), implying cat has the hypernym (whip wwhippn.01) Conversely, word sense granularity can lead to questionable relations like chair (professorship.nn) .n....shhip......). being hyponym of si situation (position.n.o0l 06) Lastly, idioms like "taking crack at something" lead (unlikely when taken out of context) relations like "crack" having the hypernym "endeavor". These limitations are exacerbated our experiments, as we do not disambiguate word senses, considering all possible meanings of a given token instead.
More broadly, BALAUR is ceneral-purposee ar- chitecture for modeling relations in the latent rep- resentations of neural network models. While our work has focused on modeling LSRs in LM hid- den states throughout pretraining, BALAUR can in principle applied to different modalities, archi- tectures, relations and optimization settings. How well our results and hypothesis generalize to such different settings remains an open question.
7 Limitations
7.1 Pretraining from seratch
Our work attempts to improve LMs by pretrain- ing from scratch, adopting relatively efficient pretraining approach geared towards academic bud- gets (Izsak et al., 2021). While our approach can in principle generalize other LMs such as GPT (Radford et al., 2018), we could not pretrain these models due limited computational resources.
Second is the issue of bias. We found WordNet to encode several harmful biases and stereotypes, either directly via harmful relations, or indirectly by including certain relations for some groups but not others. For example, "man" has hyponyms "bach- clor", "officer" and "gallant";; in contrast to "mis- tress". "nurse" and "tease" for "woman". Despite removing these associations in our work, we want to note that these kind of biases can be difficult to
Adapting our method to effectively finetune ex- isting pretrained models could significantly reduce

---

comprehensively account for when expressed as selective inclusion or omission of associations for different groups.
References
Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasca, and Aitor Soroa. 2009. A Study on Similarity and Relatedness Using Distribu- tional and WordNet-based Approaches. In Proceed- ings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 19-27, Boulder, Colorado. Association for Computa- tional Linguistics.
Lastly, is the related issue of coverage. Many concepts and relations are simply not expressed in WordNet; limiting the knowledge of LSRs that can be incorporated in LMs with this resource. This lack of coverage is exacerbated in our experiments, are limited single token words (i.e. words in the model's vocabulary). Despite trying to alle- viate this by also modeling extra-vocabull con- cepts, effectively controlling the representations of multi-token expressions in LMs remains an open problem. We note that, due to its reliance on expert lexicographers, WordNet has had limited updates and developments to increase its coverage: this is in contrast to the open sourced English WordNet 2019 (McCrae al., 2019). We suggest future work consider this resource to mitigate coverage issues.
Kushal Arora, Aishik Chakraborty, and Jackie C. K. Cheung. 2020. Learning Lexical Subspaces in a Dis- tributional Vector Space. Transactions of the Associ- ation for Computational Linguistics, 8:311-329.
He Bai. Tong Wang, Alessandro Sordoni, and Peng Shi. 2022. Better Language Model with Hypernym Class Prediction. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1352-1362, Dublin, Ireland. Association for Computational Linguistics.
John Bandy and Nicholas Vincent. 2021. Addressing "Documentation Debt" Machine Learning: Ret- rospective Datasheet for BookCorpus. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 1.
Acknowledgements
We would like to thank Samsung Electronics Co., Ldt. and Fonds de recherche du Québec for funding this research. We would also like thank the Mila IDT team for computational resources and support. The authors acknowledge the material support of NVIDIA in the form of computational resources.
Steven Bird and Edward Loper. 2004. NLTK: The Nat- ural Language Toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214-217, Barcelona, Spain, Association for Compu- tational Linguistics.
Georgeta Bordea, Paul Buitelaar, Stefano Faralli, and Roberto Navigli. 2015. SemEval-22015 Task 17: Tax- onomy Extraction Evaluation (TExEval). In Proceed- ings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 902-910, Denver, Colorado. Association for Computational Linguis- tics.
Georgeta Bordea, Els Lefever, and Paul Buitelaar. 2016. SemEval-2016 Task 13: Taxonomy Extrac- tion Evaluation (TExEval-2). In Proceedings of the 10th International Workshop on Semantic Evaluation (SemFvval20166, pages 10811,09,, San Diego Cali- fornia. Association for Computational Linguistics.
Samuel R. Bowman, Gabor Angeli, Christopher Potts. and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Jose CCamachoCoollaoos Claudio Delli Bovi, Luis Espinosa-Anke, Sergio Oramas, Tommaso Pasini, Enrico Santus, Vered Shwartz, Roberto Navigli, and Horacio Saggion. 2018 SemEval- 2018 Task 9: Hy- pernym Discovery. In Proceedings of the 12th Inter- national Workshop on Semantic Evaluation, pages

---

712-724, New Orleans, Louisiana. Association for Computational Linguistics.
Michael Gira, Ruisu Zhang, and Kangwook Lee. 2022. Debiasing Pre- Trained Language Models via Effi- cient Fine-Tuning. In Proceedings of the Second Workshop on Language Technology for Equality, Di- versity and Inclusion, pages 59-69, Dublin, Ireland. Association for Computational Linguisties.
Sharon A. Caraballo. 1999. Automatic construction of a hypernym-labeled noun hierarchy from text. In Proceedings of the 37th Annual Meeting of the As- sociation for Computational Linguistics, pages 120-
126, College Park, Maryland, USA. Association for Computational Linguistics.
Goran Glavaš and Ivan Vulić. 2018. Explicit Retrofitting of Distributional Word Vectors, In Pro- ceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), pages 34-45, Melbourne, Australia, Associ- ation for Computational Linguistics.
Walter G. Charles and George A. Miller. 1989. Contexts of antonymous adjectives. Applied Psycholinguistics 10(3):357-375.
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che,
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't Stop Pretraining: Adapt Language Models to Domains and Tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguisties.
Ting Liu, and Xiangzhan Yu. 2020. Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), P). pages 7870-7881, Online. As- sociation for Computational Linguistics.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Michael Hanna and David Mareček. 2021. Analyzing BERT's s Knowledge of Hypernymy via Prompting. In Proceedings of the Fourth BlackboxNI.P Workshop Analyzing and Interpreting Neural Networks for NLP, pages 275-222220 282, Punta Cana, Dominican Repub- lic. Association for Computational Linguistics.
Marti A. Hearst, 1992. Automatic Acquisition of Hy- ponyms from Large Text Corpora. In COLING 1992 Volume 2: The 14th International Conference on Computational Linguistics.
Luis Espinosa-Anke, Jose Camacho-Collee Collados. Clau- dio Delli Bovi, and Horacio Saggion. 2016. Super- vised Distributional Hypermym Discovery via Do- main Adaptation. In Proceedings of the 2016 Con- ference on Empirical Methods in Natural Language Processing, pages 424-435, Austin, Texas, Associa- tion for Computational Linguistics.
Dan Hendrycks and Kevin Gimpel. 2020. Gaussian Error Linear Units (GELUs). LxXiv1606.08815 [cs].
Peter Izsak, Moshe Berchansky, and Omer Levy. 2021. How to Train BERT with an Academic Budget. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10644-106522 Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Allyson Ettinger. 2020. What BERT Is Not: Lessons from New Suite of Psycholinguistic Diagnostics for Language Models. Transactions of the Association for Computational Linguistics, 8:34-48. Publisher: MIT Press
Dan Fass. 1993. Lexical Semantic Constraints. In James Pustejovsky, editor, Semantics and the Lexicon, Studies in Linguistics and Philosophy, pages 263- 289. Springer Netherlands, Dordrecht.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020, Super- vised Contrastive Learning. In Advances in Neural Information Processing Systems, volume 33, pages 18661-18673. Curran Associates, Inc.
Amit Gajbhiye. Luis Espinosa-Anke, and Steven Schockaert. 2022. Modelling Commonsense Prop- erties Using Pre-Trained Bi-Encoders. In Proceed- ings of the 29th International Conference on Com- putational Linguistics, pages 3971-3983, Gyeongju, Republic of Korea. International Committee on Com- putational Linguistics.
Anne Lauscher, Ivan Vulić, Edoardo Maria Ponti, Anna Korhonen, and Goran Glavaš. 2020. Specializing Unsupervised Pretraining Models for Word-Level Se- mantic Similarity. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 1371-1383, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics.
Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation. In Proceedings of the Third BlackboxNI.P Workshop on Analyzing and Interpreting Neural Net- works for NLP, pages 163-173, Online. Association for Computational Linguistics.
Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. 2020. SenseBERT: Driv- ing Some Sense into BERT. In Proceedings of the

---

58th Annual Meeting of the Association for Compu- tational Linguistics, pages 4656-4661 Online. Asso- ciation for Computational Linguistics.
Nikola Mrkšić, Ivan Vulić. Diarmuid Ó Séaghdha. Ira Leviant, Roi Reichart, Milica Gašié, Anna Korhonen, and Steve Young. 2017. Semantic Specialization of Distributional Word Vector Spaces using Monolin- gual and Cross-Lingual Constraints. Transactions of the Association for Computational Linguistics, 5:309- 324.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jemite, Abbishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis. Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger. Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cis- tac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A Community Library for Natural Lan- guage Processing. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
William Nagy and Dedre Gentner. 1990. Semantic con- straints on lexical categories. Language and Cogni- rive Processes, 5(3):169-200)
Made Nindyatama Nityasya, Haryo Wibowo, Al- ham Fikri Aji, Genta Winata, Radityo Eko Prasojo. Phil Blunsom, and Adhiguna Kuncoro. 2023. On "Scientific Debt" in NLP: A Case for More Rigour in Language Model Pre- Treiniain Research. In Proceed- ings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8554-85 -5722 Toronto, Canada. Association for Computational Linguisties.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du. Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretrain- ing Approach. ArXiv:1907.11699 [cs].
Ted Pedersen and Sanjeev Banerjee. 2009. A WordNet Stop List.
Massimo Poesio, Tomonori Ishikawa, Sabine Schulte im Walde, and Renata Vieira. 2002. Acquiring Lexical Knowledge for Anaphora Resolution. In Proceed- ings of the Third International Conference on Lan- guage Resources and Evaluation (LREC*02), Las Palmas, Canary Islands - Spain. European Language Resources Association (ELRA).
Zeyu Liu, Yizhong Wang, Jungo Kasai, Hannanch Ha- jishirzi, and Noah A. Smith. 2021. Probing Across Time: What Does RoBERTa Know and When? In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 242, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018 Improving language under- standing with unsupervised learning. Technical re- port, OpenAL.
Ilya Loshchilov and Frank Hutter. 2019, Decoupled Weight Decay Regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Abhilasha Ravichander, Eduard Hovy, Kaheer Sule- man, Adam Trischler, and Jackie Chi Kit Cheung. 2020. On the Systematicity of Probing Contextual- ized Word Representations: The Case of Hypernymy in BERT. In Proceedings of the Ninth Joint Confer- ence on Lexical and Computational Semantics, pages 88-102, Barcelona, Spain (Online). Association for Computational Linguistics.
Diana McCarthy. 2006. Relating WordNet Senses for Word Sense Disambiguation. In Proceedings of the Workshop on Making Sense of Sense: Bringing Psycholinguisties and Computational Linguistics To-
gether.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A Primer in BERTology: What We Know About How BERT Works. Transactions of the Asso- ciation for Computational Linguistics, 8:842-866. 866.
John P. McCrae, Alexandre Rademaker, Francis Bond, Ewa Rudnicka, and Christiane Fellbaum. 2019. En- glish WordNet 2019- An Open-Source WordNet for English. In Proceedings of the 10th Global Word- net Conference, pages 245-252, Wroclaw, Poland. Global Wordnet Association.
Ohad Rozen, Shmuel Amar, Vered Shwartz, and Ido Da- gan. 2021. Teach the Rules, Provide the Facts: Tar- geted Relational-knowledee tual Inference. In Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computa- rional Semantics, pages 89-98. Online. Association for Computational Linguistics.
Enhancement for Tex-
George A. Miller. 1995. WordNet: lexical database for English. Communications of the ACM, 38(11):39 41.
Saif Mohammad, Bonnie Dorr, and Graeme Hirst, 2008. Computing Word-Pair Antonymy. In Proceedings of the 2008 Conference on Empirical Methods in Natu- ral Language Processing, pages 982-991, Honolulu, Hawaii. Association for Computational Linguistics.
Enrico Santus, Frances Yung. Alessandro Lenci, and Chu-Ren Huang. 2015. EVAL 1.0: an Evoly- ing Semantic Dataset for Training and Evaluation of Distributional Semantic Models. In Proceedings of the 4th Workshop on Linked Data in Linguisties:

---

Resources and Applications, pages 64-69, Beijing, China. Association for Computational Linguistics.
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 38-45, Online. Association for Computational Linguisties.
Vered Shwartz, Yoav Goldberg, and Ido Dagan. 2016. Improving Hypernymy Detection with an Integrated Proceed-
Path-based and Distributional Method. ings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1389-2398, Berlin, Germany. Association for Computational Linguistics.
Mo Yu and Mark Dredze. 2014, Improving Lexical Em- beddings with Semantic Knowledge. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers). pages 545-550, Baltimore, Maryland. Association for Computational Linguistics.
Rion Snow, Daniel Jurafsky, and Andrew Ng. 2004, Learning Syntactie Pattems for Automatic Hyper- nym Discovery. In Advances Neural Information Processing Systems, volume 17. MIT Press.
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. oLMpics-On On What Language Model traitraining Captures. Transactions of the As- sociation for Computational Linguistics, 8:743-758.
lan Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4593- 4601, Florence, Italy. Association for Computational Linguisties.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. Advances Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Ivan Vulić, Daniela Gerz, Douwe Kiela, Felix Hill, and Anna Korhonen. 2017. HyperLex: Large-Scale Evaluation Graded Lexical Entailment. Computa- tional Linguistics, 43(4):781-83)
Ivan Vulić, Goran Glavaš, Nikola Mrkšić, and Anna Korhonen. 2018. Spstaapecialis:aion:: Retrofitting Vectors of Words Unseen in Lexical Resources. In Proceedings of the 2018 Conference of the North American Chapter the Association for Computa- tional Linguistics: Human Language Technologies. Volume 1 (Long Papers), pages 516-527, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing Pretrained Language Models for Lexical Semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222-7240, Online. Association for Computa- tional Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz. Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemite,, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Ar Natural Language Processing. In

---

A Appendix
lary. Specifically, our goal is to avoid modeling LSRs for synsets that only relate one item in our vocabulary, as these cannot provide any useful inductive bias to our model's 's representations of its vocabulary. We first keep any synsets which map to 2 or more tokens (i.e. capture synonymy). If remaining synset has antonymous synsets, we keep if both it and its antonym(s) have corresponding tokens in the model vocabulary. Lastly, if remain- ing synset belongs in a hypernymy or hyponymy relation, we keep it even if it does not map to a to- ken, as long as it relates to two or more hypemym or hyponym synsets that do. This enables us to indirectly model concepts not in the model vocabu- lary via co-hyponymy and co-hypemymy relations. Any remaining synset is removed, along with its related token-concee pairs. This filtering ensures that we model concepts relating multiple tokens in our vocabulary and prevents the degenerate case where a concept is indistinguishable from a token.
A.1 Additional method details
A.1.1 Filtering LSRs from WordNet
Filtering Tokens When mapping tokens to WordNet synsets using NLTK, we observed several potential sources of noise that could be addressed by filtering tokens. First, observed that many tokens with 3 fewer characters were often over-zealously lemmatized by NLTK as acronyms or abbreviations for unlikely synsets. For example cat may map to n. 01, while in maps to indium. n. 01, indiana.n. ... 01, and inch. n. 01. Originally, we attempted to filter any token with 3 or fewer characters, however our coverage of important concepts dropped significantly, so we limit ourselves to filtering tokens with 2 or fewer characters. We also filter out tokens which are wordpieces the model vocabulary (e.g. tokens prefixed by "##"" in the vocabulary of BERT, indicating these are not preceded by whitespace and occur in the middle of words) to ensure we only model LSRs for tokens that correspond entire words. We also WordNet stoplist (Pedersen and Banerjee, 2009) filter common function words that tend to be misrepresented by WordNet. Lastly, we limit ourselves to alphabetical tokens, as we found numerical and alphanumerical tokens to introduce a lot of noise.
A.1.2 HYPCC dataset creation
To create HYPCC, we first extract related token- concept pairs using the same procedure outlined in $4.1 and A.1.1. One notable difference is that we only consider direct hypemymss instead of multi-hop hypernyms up to depth 3. Further- more, we filter tokens such that they occur in the two most frequent English LM vocabularies: bert -l--baa base -usse-uncssed and 2pt as hosted by the transformers library (Wolf et al., 2020).
We then convert token-concept pairs to sets of token-token pairs, based on the concepts' surface forms which are present in our vocabularies. To convert these pairs to cloze-style prompts, we adopt the following template: "A(n) X is a type of Y". We use the inflect library to filter plural forms determine the adequate article ("a"" or "an"". While we do not account for uncountable nouns, we find that most prompts maintain their legibility.
Filtering Synsets Having filtered tokens, we then map each of these to all possible synsets using the NLTK interface WordNet. However, we found the quality, coverage and ambiguity of annotations to vary significantly across synset types. To reduce noise, we filtered synset categories based on manual inspection. We first limit ourselves to noun synsets, and filter what we found to be particularly noisy categories: quantity, motive, shape, relation, and process. Furthermore, we found that despite filtering tokens from our stoplist, NLTK was still lemmatizing other tokens to synsets in the stoplist, so we further filter any synset whose identifiers are in the stoplist.
Lastly, we found that several concepts and to- kens were disproportionately represented in this dataset as result of having multiple wordsenses or maintaining a high position in the WordNet hi- erarchy. These often lead to nonsensical prompts, which we attempted filter out using a manually curated stoplist for tokens and concepts.
Filtering Token- Token-Concept Pairs After mapping hy- peryyyy, hyponymy, synonymy and antonymy rela- tions between tokens and synsets, we filter synsets based on their coverage of our model's vocabu-
hhttps://githir com/ jtwp///aaaoo/ntiia

---

A.2 Additional results
A.2.2
Extended finetuning results on HYPCC
A.2.1 Extended zero-shot results on HYPCC
CLOSED VOCAB
OPEN VOCAB
MODEL.
Acc 1/5
MRR
A 1/5
MRR
0.1
HYPERNYM PREDICTION
BERTBASE
2.75 / 12.88
0.081
0.30/ 10.25
0.054
MIl
BERTLAKEE
3,53 / 14.13
0.092
1.78/ / 11.77
0,071
0.00
DERT+DALACR (OURS)
ROBERTABASE
15.54
0.103
1.90/ 12.14
0.074
ROBERTÁJ
7.01 /20.12
0.137
17.00
0.114
20k
40k
B0k
BERI (ours)
5.18/18.66
0.121
0.88 0.88/ 14.72
0.080
(ours)
5.31 19.65
0.128
15.4/
0.089
Number of training examples
Figure 7: Average open-vocab MRR throughout fine- tuning on the hyponym prediction subset of HYPCC.
BASE BASE
1,99 / 11.89
0.073
1 10,42
0.061
3.60 / 14.95
0.097
2.76/ 12.87
0.083
ROBERTABASE
2,94 / 12.06
0.080
2.24/9.92
0.066
ROBERIALARGE
3,89 12.90
0,091
3.37 / 11:55
0.081
BERT (ours)
12.22
0.081
2.03 / 10.65
0.069
BERT+BALAUR (ours)
3.49/17.91
0.110
1.85 / 14.56
0.084
Class Intrusion Rate
0.85
Table 7: Zero-shot results on HyPCC across MLMs. We note that RoBERTa was trained on order of magni- tude more data than the models used in our experiments (16GB versus GBIG which has significant impact downstream performance (Liu et al., 2019) and helps explain the discrepancy in performance. In contrast and in line with Nityasya et al. (2023), our main results aim to disentangle the effects of scaling data or compute.
0,75
0.7
BERT+BALAUU
BERT (OURS)
20k
40k
600k
Number of training examples
Figure 8: Average class intrusion rate throughout fine- tuning on the hyponym prediction subset of HYPCC.
0.04
BALAUR
Intrusion Rate
0.03
BERT (OURS)
0.02
Class Frequency in HypCC
Figure 9: Average intrusion rate and frequency of classes in the final models finetuned on the hyponym prediction subset of HYPCC.

---

A.2.3 MoNLI performance across random seeds
ENLI Test Accuracc
SNLI Accuracy (zoomed in)
SNLI Test Loss (zoomed lnn
HE
IOURS!
SERT SOUISE
(OURSI
3.43
0.300
0.333
0.633
PHoNLI
PMoNLI (soomed In)
PMONLI izoomed
CERTHBALAUM
BENT-BALAUX
M
H
BER IOUES!
EERT JOURSE
DERT IDURS!
00
D44
0.5
DUB
1.0
0.00
0.th
0.70
D.75
0.33
3.5
0.82
0.50
0.83
200
C:
Q
NMoNLITest Acourary
Accuracy (zoomed in)
DENTIOUSSI
EEIT
DERT TOUSS!
3000
0.6.3
0000
3:0
3.40
3.50
3.0
Figure 10: MoNLI performance across seeds when finetuned only SNLI.

---

SNLI Test Accuracy
SNLI Test Accuracy Iocemred In]
SMILI Test Loss (zoomed In)
DALAUR
DCTT
H
HH
PMoNLI Toutt Accuracy
PMM Teat Accuracy [zoomeed
PMoNiLi Test Losx (snomed
DEST ADAAUU
DCTT SALAU
DERT
DUTT
0.88
2.50
2.54
2.56
0.35
0.33
0.25
0030
605
Accuracy
Accuracy
DERT
DCTT
DCTT
033
240
0.3
Figure 11: MoNLI performance across seeds when finetuned SNLI and MoNLI.


Reviews:
[{"review_id": "8RObgW9pSc", "reviewer": "Reviewer_AUBR", "summary": "", "questions": "Why were the prompts chosen in this way? Why would the prefix \"in the context of\" be necessary, and did you test without it? How did this affect the results?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful and considerate review of our work. \n\n**To address your main reason for rejection**, we would like to clarify the main intended contribution of our work. The improvement in language modeling or obtaining SOTA was not our goal; rather we wanted to verify our hypothesis that knowledge of lexical semantics (or more precisely, *improvements on tasks requiring knowledge of hypernymy*) could be instilled in a language model by controlling its latent representations (rather than by e.g. providing it additional text to pretrain on or increasing its number of parameters). The observed improvement in language modeling performance was an interesting and unanticipated positive result, but not directly related to our original research question. We see the positive results for prompt completion and monotonicity as the results which most directly support our original hypothesis.\n\nWe agree it is unclear whether our approach generalizes to larger models, however we believe that models at the scale of BERT-Large are still useful and used in a variety of contexts; and that the generalizability of our method to even larger models makes for an interesting future research direction rather than a fundamental limitation of our work. \n\n**To answer your questions**, we added “in the context of hypernymy” as a prefix based on the observation that standard prompts like “An [X] is a [MASK]” or “An [X] is a type of [MASK]” could have valid completions unrelated to hypernymy, e.g. synonyms, which we wanted to disambiguate. Without this prefix, improvement from our approach persisted, although performance fell across all models. As we believe our use of a disambiguating prefix is better suited for our evaluation, and found no additional insights from these prefix-less results, they were not included.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is well written and a lot of work has clearly been completed. The analysis is thorough and done in a variety of settings and tasks. All necessary details are there, and particularly the discussion of results is clear, fair, and refers to previous work. \n\nThe paper addresses a weakness of language models in a targeted way, and I think that the architecture may be applicable to the injection of other kinds of knowledge as well.", "reasons_to_reject": "My main reason to reject this work would be that I'm not confident it has broad applicability. I have no doubts about the positive results for prompt completion and monotonicity, I'm not sure that the improved result on language modelling warrants such a complex architecture. I'm sympathetic with the author's constraints in terms of computational power and it makes sense that they therefore tested this on BERT, but it doesn't follow that larger models would still benefit from supervision like this, which makes my doubts about the broader impact of this work even larger. I however admit that it is possible that I'm underestimating this, as I'm not too familiar with the area.", "questions_to_authors": "Why were the prompts chosen in this way? Why would the prefix \"in the context of\" be necessary, and did you test without it? How did this affect the results?", "paper_topic_and_main_contributions": "The paper introduces a novel neural architecture that learns to transform output hidden states so that knowledge about hypernymy, hyponymy, and antonymy can be injected. They train a BERT model with this, and evaluate against a self-trained BERT, as well as BERTlarge, for language modelling as well as a variety of tasks specific to lexical semantic relations. They evaluate very thoroughly and find that their method generally improves performance on these specific tasks,", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QqCuh9L4uT", "reviewer": "Reviewer_WbqA", "summary": "", "questions": "A) If recent large transformer-based LMs still struggle with tasks that require knowledge of hypernymy, could you provide examples instead of just referencing older papers that discuss older LMs?\n\nB) In many places, \"BERT (OURS)\" is mentioned, but this is confusing. I think BERT+BALAUR is your main proposed approach, not BERT.\n\nC) Is the low MLM loss for BERT+BALAUR in Figure 2 due to BERT+BALAUR having more parameters than BERT(Ours)?\n\nD) Is the improvement achieved by BERT+BALAUR over BERT(Ours) in the prompt completion and NLI tasks because BERT+BALAUR was trained on additional data from WordNET?\n\nE) Why was BERT_large chosen as the base model when RoBERTA_large performs better for hypernymy prediction?\n", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and questions about our work. \n\n**To address your reasons for rejection**, we would first like to clarify that this work was conducted before ChatGPT, and that even if it wasn’t, we strongly believe methods which improve models without relying on ever-increasing amounts of data and compute are important contributions in and of themselves. \n\nIt’s no longer surprising that more data and more compute improves performance of LLMs; however there still exist many situations where more data and more compute are not viable solutions (e.g. under-represented languages, or compute-constrained applications).  We believe our work introduces a meaningful potential strategy for such cases.\n\nThis also explains why RoBERTa-Large outperforms the models used in our experiments: it was trained on an order of magnitude more data (16GB versus 161GB); which the original authors of RoBERTa also note has a significant impact on downstream performance. We decided to transparently include results for RoBERTa even if they could be interpreted as negative results, because we trust the community to look beyond scale and SOTA; and see our contribution as a method which improves models orthogonally to scaling data or compute. \n\nRather than demonstrating SOTA, we perform rigorous experimentation that removes potential confounds while demonstrating significant improvements with our method, inline with  [On “Scientific Debt” in NLP: A Case for More Rigour in Language Model Pre-Training Research](https://aclanthology.org/2023.acl-long.477) (Nityasya et al., ACL 2023).\n\n**However, we agree that the claim in the abstract is no longer accurate, or at least too ambiguous**, and that we should be more specific than “recent pretrained language models”, in addition to clarifying considerations of scale, as above. This edit and discussion is straightforward to incorporate and will be done for the camera-ready version if the paper is accepted. \n\n**Regarding the issue of finetuning which you raise**, we explored the option but were not able to resolve issues of catastrophic forgetting; even when interleaving the original pretraining objective. As all the related work we found on incorporating lexical semantics into LMs involved pretraining rather than finetuning, we adopted a similar approach. We believe this is a potential direction for future work rather than a fundamental limitation of this work, especially given the similar approach of previous published works on this topic. However, we agree this should be explicitly discussed and justified rather than glossed over, and can do so. \n\n**To answer your questions**:\n\n**A.** Please see the discussion above.  \n\n**B.** This is explained on line 378. To remove any confounds such as differences in training data, model implementation, and random seed induced differences, we pretrain two versions of the exact same 24h-BERT model: one with and without Balaur, keeping everything else the same. We use BERT(OURS) to distinguish from published checkpoints of BERT-Large. \n\n**C.** For all our evaluations, BERT+BALAUR and BERT(OURS) have the exact same architecture and number of parameters.  Balaur parameters are only used to compute a complementary loss function during pretraining, but are not part of the computation graph of MLM or any of our evaluations.\n\n**D.** Exactly, that was the hypothesis we were validating: that we could successfully incorporate knowledge (data from WordNet) into a model by modulating its latent representations. Importantly, there are no additional training examples; both BERT+BALAUR and BERT(OURS) see the exact same texts throughout pretraining; we use the data from WordNet to simultaneously compute a complementary loss without requiring additional forward or backward passes.\n \n**E.** See A and B.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written.", "reasons_to_reject": "1) I believe the following line in the abstract is not accurate:\n\n\"... tasks that require knowledge of hypernymy still pose a challenge for recent pretrained language models (LMs) ...\"\n\nRecent LMs like ChatGPT perform well on those tasks. This may be a challenge for older LMs like BERT, but this distinction should have been explicitly mentioned. The way the paper introduces the problem is misleading.\n\n2) It's unclear why pre-training LLMs with LSR constraints is beneficial. One could also fine-tune LLMs with LSR constraints extracted from WordNET. The comparison between the two is missing in the paper.\n\n3) The results do not clearly demonstrate the benefits of the proposed approach. For the prompt completion task considered in the paper, RoBERTA_large outperforms their approach in the Hypernym prediction sub-task.", "questions_to_authors": "A) If recent large transformer-based LMs still struggle with tasks that require knowledge of hypernymy, could you provide examples instead of just referencing older papers that discuss older LMs?\n\nB) In many places, \"BERT (OURS)\" is mentioned, but this is confusing. I think BERT+BALAUR is your main proposed approach, not BERT.\n\nC) Is the low MLM loss for BERT+BALAUR in Figure 2 due to BERT+BALAUR having more parameters than BERT(Ours)?\n\nD) Is the improvement achieved by BERT+BALAUR over BERT(Ours) in the prompt completion and NLI tasks because BERT+BALAUR was trained on additional data from WordNET?\n\nE) Why was BERT_large chosen as the base model when RoBERTA_large performs better for hypernymy prediction?\n", "paper_topic_and_main_contributions": "The paper proposes an approach to pre-train large transformer-based language models together with lexical semantic relation (LSR) constraints. Four relations have been considered: i) hypernymy, ii) hyponymy, iii) antonymy, and iv) synonymy. The main idea is to ensure that the related pairs are similar in the four relation-specific vector spaces while pretraining the language models. The paper claims that enforcing these four relation constraints between pairs of words leads to improvements in tasks that require knowledge of hypernymy.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sN8apoe0w6", "reviewer": "Reviewer_8X5a", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and suggestions.\n\n**Evaluating our model on the established benchmarks you suggest** was actually something we carefully considered and decided against. Because our central research question was whether or not LM internal representations provide an interface to their observable linguistic behavior, our reasoning was two-fold. First, we wanted our evaluations to reflect extrinsic tasks which had been identified in the literature as points of failure for models like BERT. Second, as our method directly models lexical semantic relations in BERT’s contextual embeddings, we believed such intrinsic evaluations might be positively biased towards our approach without necessarily being indicative of a language model’s ability on extrinsic evaluations. We hope this clarifies our choice of evaluation and makes it seem less ad-hoc. Nevertheless, we agree that these established benchmarks are important to our work, and intend to update our paper with a more comprehensive review of these and our reasoning above for not including these in our evaluation. \n\n**Regarding your point about not predicting y in the sentence 'y is a type of [mask]'.** \nWe mention on L466 that our results are consistent with previous findings. We do not believe this to be a major limitation of our approach, but rather of the base language model and pretraining procedure which our work adopts. We agree that autoregressive models can likely better handle these, but believe it is beyond the scope of this work and more appropriate for future work.\n\n**Regarding the negative results on NMoNLI.**\nWhile the reported average for our method is lower (both methods perform close to chance, 48.9% vs 56.5%), we explain on line 527 and show in Figures 10&11 that these results are likely too high variance to interpret reliably with only 5 samples.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- New dataset\n- Well motivated pretraining objective\n- Very well written, and technically sound paper\n", "reasons_to_reject": "- Evaluation seems too ad-hoc\n- Only using wordnet considering the many taxonomies, lexical databases and thesauri available feels like an important limitation.\n- Experimental results are mixed.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a novel method of pretraining a BERT-type LM on semantic relations (synonymy, antonymy and hypernymy) so that they perform better in tasks requiring modeling hypernymy. The main idea is to pretrained them so that hyponym-hypernym vector pairs are brought together in the internal representations of the model, whereas noisy samples are pushed far apart. The dataset used as a source for pretraining is WordNet. \n\nThe description of this method is very well explained, sound and carefully provides implementation details (even licensing) making this model replicable to the last detail. The paper also comes with a newly created dataset of hypernymy pairs, verbalized and where one of them is masked. \n\nThe main limitation I see is that, surprisingly (given the good account of related work), this model is not evaluated on a plethora of works that have proposed datasets and tasks relevant to hypernymy. I find this to be a major limitation of the paper, since instead, it proposes several novel tasks that are conceptually similar or analogous to works that have served as benchmarks for many years.\n\nThe second major limitation I can see is that the resulting model still struggles with what intuitively should be an easy task, i.e., not predicting y in the sentence 'y is a type of [mask]'. I find this surprising and, given that the authors hint at applying the proposed pretraining strategy to autoregressive LMs, perhaps these could handle this artifact better.\n\nFinally, in terms of technical contributions, while reporting a sort of negative results in NMoNLI is commendable, it feels more like an interesting piece of analysis instead of a core motivation for the value of this new model.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "While a bit dated, the below references should give an account of existing datasets and methods that have been used in the hypernymy modeling space in the past years.\n\nTasks and datasets that explicitly measure hypernymy\n\n\tTexEval 1 and 2\n\tGeorgeta Bordea, Paul Buitelaar, Stefano Faralli, and Roberto Navigli. 2015. SemEval-2015 Task 17: Taxonomy Extraction Evaluation (TExEval). In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 902–910, Denver, Colorado. Association for Computational Linguistics.\n\n\tGeorgeta Bordea, Els Lefever, and Paul Buitelaar. 2016. SemEval-2016 Task 13: Taxonomy Extraction Evaluation (TExEval-2). In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 1081–1091, San Diego, California. Association for Computational Linguistics.\n\n\tHypernym detection\n\n\tVered Shwartz, Yoav Goldberg, and Ido Dagan. 2016. Improving Hypernymy Detection with an Integrated Path-based and Distributional Method. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2389–2398, Berlin, Germany. Association for Computational Linguistics.\n\n\tEnrico Santus, Frances Yung, Alessandro Lenci, and Chu-Ren Huang. 2015. EVALution 1.0: an Evolving Semantic Dataset for Training and Evaluation of Distributional Semantic Models. In Proceedings of the 4th Workshop on Linked Data in Linguistics: Resources and Applications, pages 64–69, Beijing, China. Association for Computational Linguistics.\n\n\tEnrico Santus, Frances Yung, Alessandro Lenci, and Chu-Ren Huang. 2015. EVALution 1.0: an Evolving Semantic Dataset for Training and Evaluation of Distributional Semantic Models. In Proceedings of the 4th Workshop on Linked Data in Linguistics: Resources and Applications, pages 64–69, Beijing, China. Association for Computational Linguistics.\n\n\tHypernym discovery\n\n\tEspinosa-Anke, L., Camacho-Collados, J., Delli Bovi, C., & Saggion, H. (2016). Supervised distributional hypernym discovery via domain adaptation. In Conference on Empirical Methods in Natural Language Processing; 2016 Nov 1-5; Austin, TX. Red Hook (NY): ACL; 2016. p. 424-35.. ACL (Association for Computational Linguistics).\n\n\tCamacho-Collados, J., Bovi, C. D., Anke, L. E., Oramas, S., Pasini, T., Santus, E., ... & Saggion, H. (2018, June). SemEval-2018 task 9: Hypernym discovery. In Proceedings of the 12th international workshop on semantic evaluation (pp. 712-724).\n\n\tGraded hypernym detection\n\n\tVulić, I., Gerz, D., Kiela, D., Hill, F., & Korhonen, A. (2017). Hyperlex: A large-scale evaluation of graded lexical entailment. Computational Linguistics, 43(4), 781-835.\n\nSimilar pretraining strategy\n\nGajbhiye, A., Espinosa-Anke, L., & Schockaert, S. (2022, October). Modelling Commonsense Properties Using Pre-Trained Bi-Encoders. In Proceedings of the 29th International Conference on Computational Linguistics (pp. 3971-3983)."}}]

Author rebuttal:

Meta-review: The paper introduces a novel BERT-type pre-training model that learns to transform output hidden states with knowledge about hypernymy, hyponymy, and antonymy (from WordNet) injected. They trained the model from scratch on English Wikipedia and BookCorpusOpen, comparing to BERT. The proposed method is interesting, evaluated on prompt completion and monotonicity NLI (MoNLI). The amount of improvements while did not impress some of the reviewers, have be in reasonable or expected range of improvement as similar research that also attempts modifications for BERT-type of encoder models. At the same time, this work has demonstrated novelty and technical contributions that *ACL/EMNLP conferences are looking for, offers a good reference point for future work on pre-training models with external semantic knowledge, and should be encouraged for publication.
Decision: Accept-Findings