SlowGuess/NAACL_2022_OCR huggingface dataset:
Myeongjun Jang $^{1}$  Frank Mtumbuka $^{1}$  Thomas Lukasiewicz $^{2,1}$
$^{1}$  Department of Computer Science, University of Oxford, UK
$^{2}$  Institute of Logic and Computation, TU Wien, Austria
firstname_lastname@cs.ox.ac.uk
The logical negation property (LNP), which implies generating different predictions for semantically opposite inputs ( $p$  is true iff  $\neg p$  is false), is an important property that a trustworthy language model must satisfy. However, much recent evidence shows that large-size pre-trained language models (PLMs) do not satisfy this property. In this paper, we perform experiments using probing tasks to assess PLMs' LNP understanding. Unlike previous studies that only examined negation expressions, we expand the boundary of the investigation to lexical semantics. Through experiments, we observe that PLMs violate the LNP frequently. To alleviate the issue, we propose a novel intermediate training task, named meaning-matching, designed to directly learn a meaning-text correspondence, instead of relying on the distributional hypothesis. Through multiple experiments, we find that the task enables PLMs to learn lexical semantic information. Also, through fine-tuning experiments on 7 GLUE tasks, we confirm that it is a safe intermediate task that guarantees a similar or better performance of downstream tasks. Finally, we observe that our proposed approach<sup>1</sup> outperforms our previous counterparts despite its time and resource efficiency.
Contemporary large-size PLMs, such as BERT (Devlin et al., 2019), ELECTRA (Clark et al., 2020), and GPT-2 and -3 (Radford et al., 2019; Brown et al., 2020), have shown excellent results in many downstream tasks, even performing better than humans in the GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019b) benchmark datasets.
However, their reliability is recently being challenged. Many studies have conducted various probing tasks and observed that PLMs exhibit faulty behaviours, such as insensitivity to sentence ordering (Pham et al., 2021; Gupta et al., 2021; Sinha
et al., 2021b), incomprehension on number-related representations (Wallace et al., 2019; Lin et al., 2020; Nogueira et al., 2021), and lack of semantic content understanding (Ravichander et al., 2020; Elazar et al., 2021). These issues raise concerns about PLMs' stability and reliability, precluding them from applications in practice, especially in risk-sensitive areas.
Another critical problem of PLMs is their inaccurate behaviour on negation, which is a principal property in many language understanding tasks. For tasks where the LNP holds ( $p$  is true iff  $\neg p$  is false; see Aina et al. 2018), PLMs should make different answers for the original and negated inputs. However, several studies observed that PLMs violate this property. In masked knowledge retrieval tasks, PLMs frequently generate incorrect answers for negated input queries (Ettinger, 2020; Kassner and Schütze, 2020). In other studies, PLMs show a poor generalisation ability on negated natural language inference (NLI) datasets (Naik et al., 2018; Hossain et al., 2020).
Although the aforementioned studies produced promising analysis results, they limited the scope of the LNP only to adding negation expressions (e.g., "no" and "not"). However, other perturbations that generate the opposite meaning also can be applied to the property. Therefore, a consideration of such perturbation methods is necessary to fully assess whether PLMs satisfy the LNP.
Also, remedies to alleviate the problem have not been studied much yet. Hosseini et al. (2021) recently employed data augmentation and unlikelihood training (Welleck et al., 2020) to prevent models from generating unwanted words, given the augmented negated data during masked language modelling (MLM). However, this approach has several downsides. First, like previous works, Hosseini et al. (2021) only considered negation expressions. Second, the data augmentation method is contingent on many additional linguistic compo
<sup>1</sup>https://github.com/MJ-Jang/beyond-distributional
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2030 - 2042 July 10-15, 2022 ©2022 Association for Computational Linguistics
nents, which causes the dependency of a model's performance on certain modules and precludes applying the method to other languages where such resources are unavailable. Finally, the model should be pre-trained from scratch with the unlikelihood objective, which consumes considerable time and resources.
In this paper, we expand the boundary of the LNP to lexical semantics, i.e., synonyms and antonyms, and ascertain that PLMs are prone to violate the LNP. Next, we propose a remedy, called intermediate-training on meaning-matching  $(\mathrm{IM}^2)$ , which hardly employs additional linguistic components. We hypothesise that a leading cause lies in the MLM training objective, which assumes the distributional hypothesis for learning the meaning of the text (Sinha et al., 2021a). Instead, we design a model that directly learns the correspondence between words and their semantic contents. Through experiments, we verify that our approach improves the model's comprehension of the LNP, while showing a stable performance on multiple downstream tasks.
Our main contributions are as follows: (i) We extend the investigation of the LNP from negation to lexical semantics (Section 2), (ii) we reveal that PLMs are prone to violate the LNP (Section 3), (iii) we propose a novel remedy, named  $\mathrm{IM}^2$ , which is decoupled from the distributional hypothesis but learns meaning-text correspondence instead (Section 4), (iv) through experiments, we ascertain that the proposed approach improves the understanding of negation and lexical semantic information (Sections 5.1 and 5.2), and (v) we verify that meaning-matching is a stable and safe intermediate task that produces a similar or better performance in multiple downstream tasks (Sections 5.3 and 5.4).
2 Probing Tasks for Investigating the Logical Negation Property
We design three probing tasks to evaluate whether PLMs satisfy the LNP: masked knowledge retrieval on negated queries (MKR-NQ), masked word retrieval (MWR), and synonym/antonym recognition (SAR). Brief illustrations of each task are in Figure 1.
2.1 Masked Knowledge Retrieval on Negated Queries
The MKR-NQ task examines whether PLMs generate incorrect answers for negated queries. Following the work of Kassner and Schütze (2020), we
constructed the evaluation dataset by negating the LAMA dataset (Petroni et al., 2019), which contains masked free-text forms of ConceptNet (Speer et al., 2017) triplets and their corresponding answers (e.g., (bird, CapableOf, fly)  $\rightarrow$  ("A bird can [MASK]", fly)). The task aims to generate a correct word through MLM.
According to the LNP, a model must not generate the original answer if the query is negated. To measure how likely PLMs generate wrong predictions for negated queries, we collected pairs of (negated_query, wrongpredictions). We selected several relations in the LAMA dataset that ensure mutual exclusiveness between the original and negated queries. For negating sentences, we selected LAMA data points that contain a single verb using the Spacy parts of speech (POS) tagger (Honnibal and Johnson, 2015). Next, we added negation expressions, such as "not" and "don't", or removed such expressions if they existed. Finally, we collected the wrong predictions from ConceptNet by using the head entity and relation. As a result, we collected 3,360 data points for this task. The list of the relations that we used and examples of the data are in Table 10 in Appendix A.
2.2 Masked Word Retrieval
To expand the boundary of the LNP to lexical semantics, we design the MWR task, which generates an answer of a masked query, asking for the synonym/antonym of a target word through MLM (e.g., "happy is the synonym of [MASK]").
Let  $s_w$  and  $a_w$  denote masked queries that ask the synonym and antonym of the word  $w$ , respectively. Also, let  $A_s$  and  $A_a$  refer to the list of correct answers for  $s_w$  and  $a_w$ , respectively. Intuitively,  $A_a$  becomes the wrong predictions of  $s_w$ , because  $s_w$  and  $a_w$  have the opposite meaning. Therefore, we can evaluate the violation of the LNP by investigating whether a PLM generates wrong predictions.
To extract commonly-used words for our experiment, we first extracted nouns, adjectives, and adverbs that appear more than five times in the SNLI dataset (Bowman et al., 2015). Among the extracted candidates, we filtered words that have synonyms or antonyms in ConceptNet. Finally, we generated masked queries by employing templates used by Camburu et al. (2020). As a result, we collected about 27K data points for MWR. The
For example, the HasProperty relation is not suitable to use, because sentences like "Some adults are immature" and "Some adults are not immature" are not mutually exclusive.
(a) MKR-NQ
(b) MWR
(c) SAR
Figure 1: Illustration of the MKR-NQ, MWR, and SAR tasks.
templates and examples of the data are in Table 11 in Appendix A.
2.3 Synonym/Antonym Recognition
SAR is a classification that distinguishes whether two given words are synonyms or antonyms. It aims to evaluate whether the contextualised representations of PLMs reflect the lexical meaning of words. Therefore, we use a parametric probing model (Adi et al., 2017; Liu et al., 2019a; Belinkov and Glass, 2019; Sinha et al., 2021a) for the experiment. Specifically, the experiment is performed on the final layer of each PLMs, i.e., we only train the classifier while keeping the encoder frozen. We use ConceptNet to build the dataset. ConceptNet has much more synonym triplets compared to antonyms. As a result, we randomly sample the synonym triplets to maintain a balance. To that end, we collect 33K, 1K, and 2K data points for the train, dev, and test datasets, respectively.
We use the top-  $k$  hit rate  $(\mathrm{HR}@\mathbf{k})$  to evaluate the performance on the MKR-NQ and MWR tasks. Assume that  $P = \{(p_1,c_1),(p_2,c_2),\ldots ,(p_n,c_n)\}$  denotes the set of predictions for a data point  $x$  where  $p_t$  and  $c_{t}$  refer to the predicted word and confidence score of the  $t$  -th prediction, respectively. Then, the top-  $k$  hit rate for a data point  $x$  is defined as follows:
$$
H R @ k (x) = \frac {\sum_ {i = 1} ^ {k} \mathbb {1} \left(p _ {i} \in \mathcal {W} _ {x}\right)}{k},
$$
where  $\mathcal{W}_x$  is the wrong prediction set of  $x$ . Intuitively, the metric measures the ratio of top- $k$  predicted words that belong to the wrong prediction set.
To reflect the prediction confidence score to the evaluation metric, we additionally define the weighted top-  $k$  hit rate (WHR@  $k$ ) that uses the
confidence score as weights. It is worth to mention that lower metrics mean a better model performance in both cases as the metrics assess how likely the models make inaccurate answers that they must avoid. The weighted metric can be defined as follows:
$$
W H R @ k (x) = \frac {\sum_ {i = 1} ^ {k} c _ {i} \times \mathbb {1} \left(p _ {i} \in \mathcal {W} _ {x}\right)}{\sum_ {i = 1} ^ {k} c _ {i}}.
$$
For the SAR task, we employ accuracy as an evaluation metric, because each data point has its own label, and the label distribution is not skewed.
3 PLMs Lack Information of Negation and Lexical Semantics
We select the following PLMs for the experiments: bidirectional encoder representations from transformers (BERT)-base/large (Devlin et al., 2019), RoBERTa-base/large (Liu et al., 2019b), and ALBERT-base/large (Lan et al., 2019). These PLMs are pre-trained with the MLM training objective. We added the ELECTRA-small/base/large models (Clark et al., 2020) for the SAR task, but it is not used for the MKR-NQ and MWR experiments, as the discriminator of the ELECTRA models are trained with the replaced token prediction (RTP) training objective and have no MLM classifier. No additional training is required for the MKR-NQ and MWR tasks. For the SAR task, we fine-tune each PLM for 10 epochs and apply the early stopping technique. We use the AdamW optimiser (Loshchilov and Hutter, 2019) for training with a learning rate of  $5e^{-6}$  and a batch size of 32.
3.1 Results for MKR-NQ
The results for the MKR-NQ task are summarised in Table 1. In general, the results are consistent with previous works (Ettinger, 2020; Kassner and Schütze, 2020). We observe three important characteristics from the experimental results.
Model	MKR-NQ	MWR
HR@1	HR@3	WHR@3	HR@5	WHR@5	HR@1	HR@3	WHR@3	HR@5	WHR@5
BERT-base	9.57	6.38	8.42	5.00	7.81	35.03	18.83	28.71	13.26	26.03
BERT-large	13.33	7.70	11.17	6.03	10.51	36.66	20.45	29.68	14.60	26.56
RoBERTa-base	11.52	6.85	9.63	5.30	8.91	13.02	8.47	10.50	6.54	9.32
RoBERTa-large	15.72	9.25	13.31	6.86	12.24	27.92	17.07	23.06	12.84	20.82
ALBERT-base	4.24	3.75	4.24	3.26	4.09	26.37	14.95	22.10	10.75	20.32
ALBERT-large	9.22	6.38	7.96	4.94	7.30	50.77	25.05	42.67	17.03	39.09
Table 1: Overall results for the MKR-NQ and MWR experiments. We multiply 100 to each value to improve readability. Note that the lower the values the better.
BERT	RoBERTa	ALBERT
base	large	base	large	base	large
Rsyn	37.79	36.58	17.13	46.01	11.37	76.10
Rant	41.44	41.79	13.57	30.21	33.26	62.65
Table 2: Ratios of instances that PLMs regenerate the word in the input sentence.  $\mathcal{R}_{syn}$  and  $\mathcal{R}_{ant}$  are the ratios of synonym and antonym-asking questions, respectively.
Model	Encoder-fixed	Fine-tune
Aval	Atest	Aval	Atest
BERT-base (108M)	53.1	55.0	84.0	85.6
BERT-large (333M)	54.4	53.5	92.1	92.5
RoBERTa-base (124M)	71.1	70.1	87.2	87.8
RoBERTa-large (355M)	69.7	69.1	93.7	94.2
ALBERT-base (11M)	56.6	58.1	81.5	84.0
ALBERT-large (17M)	54.7	56.6	86.9	88.0
ELECTRA-small (13M)	64.1	63.9	80.2	80.9
ELECTRA-base (109M)	67.9	70.6	93.3	92.9
ELECTRA-large (334M)	69.4	72.7	95.9	95.4
Table 3: Results of the SAR experiment.  $\mathcal{A}_{val}$  and  $\mathcal{A}_{test}$  are the accuracy of the validation and test dataset, respectively. We record the average of five repetitions.
First, large models produce a higher hit rate than their corresponding base-size models in all three PLMs, recording an average of about 1.5 times higher values. This implies that large-size models are more likely to generate wrong predictions for negated queries, even though they perform better than small-size models in many benchmark tests. The results suggest that evaluating a model's performance solely based on the accuracy metric is unwise.
Second, the hit rate decreases as  $k$  increases, which implies that the majority of PLMs' top predictions (e.g.,  $k = 1$  or  $k = 2$ ) are incorrect. Finally, the weighted hit rate is much higher than the vanilla hit rate, suggesting that PLMs generate wrong predictions with high confidence.
3.2 Results for MWR
The results of the MWR task are summarised in Table 1. The three characteristics found in the
MKR-NQ task are also observed in the MWR task. Also, we found the following additional patterns.
PLMs lack knowledge of antonyms. In general, the hit rates are extremely high compared to the MKR-NQ task in all the PLMs. Analysing their predictions, we find that PLMs generate incorrect predictions primarily in antonym-asking queries. Specifically, the average HR@1 of the antonym-asking queries is  $41.9\%$ , while that of the synonym-asking queries is only  $1.4\%$ . A leading cause is that PLMs simply replicate the word presented in the input query. Table 2 shows the ratio of instances where each PLM reproduces the same word in a question. While the values are quite high for both synonym-asking and antonym-asking queries, the problem is more severe in the latter case, because the generated predictions are definitely incorrect. Based on our results, we conclude that PLMs' contextualised representations lack lexical semantic information. Our conclusion is in line with the findings of Liu et al. (2019a) showing that encoder-fixed PLMs are not suitable to deal with tasks that require fine-grained linguistic knowledge.
Issues are more severe with nouns. We observe that the hit rates are higher when a word in a question is a noun. Specifically, the average HR@1 values of nouns, adjectives, and adverbs are  $35.1\%$ ,  $27.4\%$ , and  $11.8\%$ , respectively. Interestingly, PLMs have a high error rate when dealing with nouns even though they are trained with a large written English corpus, where nouns form the greatest portion (at least  $37\%$ ) of all POS tags (Hudson, 1994; Liang and Liu, 2013).
3.3 Results for SAR
As part of the comparison, we fine-tune each PLM on the SAR task, i.e., train the entire set of parameters. The results are summarised in Table 3. We observe a huge gap between the performance of fine-tuned models and that of encoder-fixed models.
In contrast to the fine-tuned models that produce a high accuracy, encoder-fixed models fall short of expectations, even recording almost a random guess performance in BERT models. Also, just as a common belief, large models' performance is greatly improved when fine-tuned. However, the difference between the large and small encoder-fixed models is insignificant, except for the ELECTRA models that exhibit only a marginal improvement. The two phenomenons suggest that PLMs' outstanding performance is predicated on updating many parameters to learn syntactic associations presented in training data (Niven and Kao, 2019; McCoy et al., 2019), but their contextualised representations do not carry abundant lexical meaning information.
4 Intermediate Training on Meaning Matching Task:  $\mathbf{IM}^2$
4.1 Issue of PLMs
Through the previous experiments, we observe that PLMs contain little information about negation and especially lexical semantics. We hypothesise a leading cause lies in the training objective of PLMs: the language modelling (LM) objective, which is a backbone pre-training task of almost all PLMs.
In the LM objective, words are generated based on given contexts. The distributional hypothesis (Harris, 1954), which assumes that semantically related or similar words will appear in similar contexts (Mrkšić et al., 2016), is the underpinning assumption of the LM objective (Sinha et al., 2021a). Under this assumption, a model learns the meaning of texts based on their correlation to others. This is a great benefit, because a model can learn the meaning of texts using only the text form, allowing unsupervised training. Based on this advantage, many unsupervised representations, such as Word2Vec (Mikolov et al., 2013), Glove (Pennington et al., 2014), and current PLMs, have been developed.
However, the problem is that the distributional hypothesis has limitations in reflecting a word's semantic meanings, because words having different or even opposite semantic meanings can appear in similar or the same contexts. For instance, consider the two words "boy" and "girl". We can readily imagine sentences in which the two words appear in the same context, e.g., "the little boy/girl cuddled the teddy bear closely". As a result, a model can learn their common functional meanings, i.e., young human beings, and the vector representa
tions would be very similar if they were trained based on the distributional hypothesis. However, the representation hardly captures their semantic antonymy, e.g., gender. Similarly, negated sentences have almost identical contexts to their original forms. As a result, models cannot effectively learn the semantic meaning of words and negation expressions, provided they leverage only the text forms.
4.2 Meaning-Matching Task
In the light of meaning-text theory, there is a correspondence between linguistic expressions (text) and semantic contents (meaning) (Mel'čuk and Žolkovskij, 1970; Milicevic, 2006). Instead of solely relying on the distributional hypothesis, we propose the new meaning-matching task, which can directly learn the correspondence. Specifically, meaning-matching is a classification that takes a word and a sentence as input and determines whether the sentence defines the word correctly. Through this task, a model can learn both meaning-text correspondences and correlations between a word and other words in a definition, which is rarely found in general corpora.
For training PLMs on our new task, we apply the intermediate-training technique (Phang et al., 2018; Wang et al., 2019a; Liu et al., 2019a; Pruksachatkun et al., 2020; Vu et al., 2020), which first fine-tunes PLMs on an intermediate task, and then fine-tunes the model again on target tasks. It has been shown that training on intermediate tasks that require high-level linguistic knowledge and inference ability could improve performance (Liu et al., 2019a; Pruksachatkun et al., 2020). Furthermore, it is more efficient in time and resources than pretraining models on large corpora (e.g., BERTNOT model (Hosseini et al., 2021)).
Dataset. We collect about 150K free-text definitions that depict the meaning of English words from WordNet (Miller, 1995) and the English Word, Meaning, and Usage Examples dataset. In cases when a word appears in both datasets, we concatenate the word's definitions. Several examples of our data are presented in Table 12 in Appendix A. We use publicly available English datasets for convenience, but our approach is easily adaptable to other languages, since most of them have their own dictionaries.
3https://data.world/idrismunir/english-word-meaning-and-usage-examples/
Model	MKR-NQ	MWR
HR@1	HR@3	WHR@3	HR@5	WHR@5	HR@1	HR@3	WHR@3	HR@5	WHR@5
BERT-large	13.33	7.70	11.17	6.03	10.51	36.66	20.45	29.68	14.60	26.56
BERT-large (IM2)	11.41	7.01	9.86	5.57	9.14	18.92	13.07	15.78	10.30	14.14
RoBERTa-large	15.72	9.25	13.31	6.86	12.24	27.92	17.07	23.06	12.84	20.82
RoBERTa-large (IM2)	6.56	4.97	6.05	3.99	5.67	22.08	12.68	18.94	9.20	17.63
Table 4: Results of BERT-large and RoBERTa-large after applying the IM² approach. We multiply 100 to each value for a better readability. Note that the lower the values the better.
Model	Encoder-fixed	Fine-tune
ΔVal	ΔTest	ΔVal	ΔTest
BERT-base (108M)	+5.5*	+5.1*	+3.9*	+3.0*
BERT-large (333M)	+3.1*	+6.3*	+1.0	+0.2
RoBERTa-base (124M)	+4.5*	+5.9*	+1.3*	+1.6*
RoBERTa-large (355M)	+15.0*	+17.1*	+0.6	+0.5
ALBERT-base (11M)	-2.6	+2.5	+4.7*	+3.3*
ALBERT-large (17M)	+1.3	+1.4	+1.2	+1.6
ELECTRA-small (13M)	-4.1*	-2.7*	+1.1	+1.1
ELECTRA-base (109M)	+3.8*	+3.2*	-0.2	+0.7
ELECTRA-large (334M)	+14.0*	+10.2*	+0.4	+0.5
Figure 2: The performance of the RoBERTa-base (IM²) model with different  $k$  values. We repeat each experiments for five times and record their average.
Training details. It is necessary to generate false word-definition pairs to train PLMs on the meaning-matching task. To achieve this, we use a negative sampling technique. We investigate the proper  $k$  in the range of 3, 5, 10, and 20. For a hyperparameter search, the performance of the RoBERTa-base model on the SAR task is used as a criterion. Figure 2 illustrates the SAR performance of the RoBERTa-base model with different  $k$  values. Intuitively, a large  $k$  value will lead the model to a better performance by investigating more wordmeaning combinations. However, we observe that
Table 5: PLMs' accuracy change in the SAR task when we apply  $\mathrm{IM}^2$ . We record the average across 5 runs. Our models show a statistically significant difference with  $p$ -value  $< 0.05 (*)$  compared to the baseline results in Table 3.
QUERY: demand is an antonym of [MASK]
ROBERTA-LARGE demand	ROBERTA-LARGE (IM²) supply
QUERY: tomorrow is the opposite of [MASK]
BERT-LARGE tomorrow	BERT-LARGE (IM²) today
QUERY: question is an antonym of [MASK]
BERT-LARGE question	BERT-LARGE (IM²) answer
Table 6: Examples of top-1 predictions on MWR queries. Unlike the original PLMs, our models do not reproduce a word in a query and make quite accurate predictions.
the model performs the best when  $k$  is 10, and the performance decreases if  $k$  is too large. We conjecture that a leading cause is that the dataset contains many words with similar meanings, mostly derived from the same stem. As a result, large  $k$  values can increase the possibility of recognising the meaning of such similar words as different.