SlowGuess/NAACL_2022_OCR huggingface dataset:
To avoid the class-imbalance issue in a batch, we duplicate the correct word-definition pairs  $k$  times when we construct the training data. For training, the AdamW optimiser is used with a learning rate of  $5e^{-6}$ . We use  $5\%$  of data points for validation and train the models for 15 epochs with a batch size of 32. The early stopping technique is used to prevent overfitting.
We conduct the same probing tasks after the intermediate training on the meaning-matching task.
5.1 SAR Results
We first focus on the SAR task. After the intermediate training, all models are fine-tuned on the SAR task with the same hyperparameters described in Section 3. The results are summarised in Table 5.
Our models trained with the meaning-match task can be downloaded from the following repositories: ELECTRA-large, BERT-large, RoBERTa-large.
Figure 3: Frobenius norm box plots of PLMs' layer after intermediate training on the meaning-matching task.
Improved lexical semantic information. We generally observe marginal or no significant improvements when fine-tuning the whole parameters, especially for large-size PLMs. However, with fixed encoder, the performance is significantly improved for PLMs with more than 100M parameters, and the improvements are more significant for large PLMs. Our results show that the proposed approach assists PLMs to learn enhanced representations with more abundant lexical semantic information.
Catastrophic forgetting. We find that small PLMs, such as ELECTRA-small and ALBERT models, show no significant increase in performance or are negatively impacted. Because all PLMs achieve a comparable performance on the meaning-matching task, we hypothesise that a leading cause is catastrophic forgetting (Pruksachatkun et al., 2020; Wallat et al., 2020), where the model forgets previous knowledge learned through pretraining to accept new information from the intermediate task. To verify this, we measure the change of parameter values after  $\mathrm{IM}^2$ . Concretely, let  $M_{i}$  and  $M_{i}^{mm}$  denote the parameter of  $i$ -th layer before and after  $\mathrm{IM}^2$ . We calculate the average Frobenius norm for each layer:
$$
\mathcal {F} _ {i} = \frac {1}{| M _ {i} |} | M _ {i} - M _ {i} ^ {m m} | _ {F}.
$$
Figure 3 shows the boxplots of  $\mathcal{F}_i$  for each PLMs. We observe that the parameters of the ELECTRA-small model, which is negatively impacted, are changed considerably compared to other PLMs having parameters more than 100M. The results suggest that the size of PLMs is an
important property to prevent the catastrophic forgetting issue.
5.2 MKR-NQ and MWR Results
Next, we perform the MKR-NQ and MWR tasks after applying the  $\mathrm{IM}^2$  method. Since our models are not trained with the MLM objective, we replace the encoder of original PLMs with that of the models after fine-tuning on the meaning-matching task and reuse the MLM classifier. For the experiments, we use BERT-large and RoBERTa-large, because they are pre-trained based on the MLM objective, and parameters are hardly changed after applying the  $\mathrm{IM}^2$  method. The results are summarised in Table 4.
We observe substantial decreases in the hit rates of incorrect predictions in both PLMs. For the MWR task, we find that the issue of regenerating a word in a given query is greatly relieved after applying the  $\mathrm{IM^2}$  method. Specifically, the percentage of such instances drops from  $40.3\%$  to  $19.6\%$  and from  $33.8\%$  to  $25.2\%$  for BERT-large and RoBERTa-large, respectively. Several examples of the predicted results are presented in Table 6. The results lend support to our claim that the  $\mathrm{IM^2}$  approach is of benefit to learning lexical semantic information and the meaning of negated expressions.
5.3 Fine-Tuning on the GLUE Benchmark
A critical drawback of intermediate training is that the target task performance could be negatively impacted if the intermediate task is not related to the target task (Liu et al., 2019a; Pruksachatkun et al., 2020). To confirm whether the issue occurs, we compare the performance of BERT, RoBERTa, and ELECTRA-large on 7 GLUE benchmark datasets (Wang et al., 2018) with their  $\mathrm{IM}^2$  counterparts. We train the models for 10 epochs for each dataset and apply the early stopping technique where the patience number is set to 3. It is observed that the training is generally finished within 8 epochs for all the models. The batch size per GPU and learning rates used for each dataset are described in Table 8. Datasets with large training set (e.g., MNLI, QNLI, and QQP) were not sensitive to the hyperparameters.
The results are presented in Table 7. We find no significant difference in performance for tasks with large datasets, such as MNLI, QNLI, QQP, and SST2. On the contrary, tasks with small datasets, like MRPC and RTE, are slightly improved. The
Model	COLA	MNLI-m	MNLI-mm	QNLI	RTE	QQP	MRPC	SST2
BERT-large	59.6±1.1	85.5±0.4	85.3±0.5	91.7±0.1	65.5±2.6	89.9±0.2	80.9±2.0	92.3±0.3
BERT-large (IM²)	61.5±1.0	85.7±0.1	85.5±0.1	91.6±0.2	66.8±1.0	90.0±0.1	82.8±1.1	92.4±0.3
RoBERTa-large	62.9±1.9	90.2±0.1	90.0±0.2	94.5±0.1	81.7±1.8	90.9±0.4	87.2±1.1	95.7±0.1
RoBERTa-large (IM²)	64.8±2.1	90.3±0.1	89.9±0.1	94.4±0.1	83.1±1.4	91.0±0.0	88.2±1.5	95.4±0.3
ELECTRA-large	68.4±2.3	90.9±0.1	90.7±0.2	94.5±0.3	86.9±2.2	91.6±0.5	88.9±1.5	96.7±0.1
ELECTRA-large (IM²)	69.1±0.7	90.8±0.1	90.7±0.1	94.3±0.2	87.0±1.3	91.7±0.3	89.5±0.5	96.4±0.4
Table 7: GLUE benchmark validation performance of PLMs before and after intermediate training on the meaning-matching task. Matthew's correlation for the COLA and accuracy for the other tasks are used as an evaluation metric. We report the mean and standard deviation across 5 runs. The best values for each PLM are in bold.
COLA	MNLI	QNLI	RTE	QQP	MRPC	SST2
b-size	16	64	64	8	64	8	64
lr	2e-5	1e-5	1e-5	2e-5	1e-5	1e-5	1e-5
Table 8: Batch size and learning rates used for the GLUE benchmark experiments.
Model	SNLI	MNLI
dev	w/neg	dev	w/neg
BERTNOT	89.0±0.1	46.0±0.4	84.3±2.3	60.9±0.3
BERT-IM2	90.3±0.2	48.00±0.5	83.1±0.3	61.8±0.6
Table 9: Accuracies on the original development dataset (dev) and the NegNLI (w/neg) dataset for SNLI and MNLI tasks. The results of our approach are averaged across 5 runs. The best values are in bold.
result is consistent with Pruksachatkun et al. (2020) and Vu et al. (2020), which showed that smaller tasks benefit much more from the intermediate training. Furthermore, unlike the previous studies that observed a negative transfer with the COLA dataset (Phang et al., 2018; Pruksachatkun et al., 2020), the performance is improved in our approach. The result suggests that meaning-matching is a safe intermediate task that ensures a positive transfer with target downstream tasks.
5.4 Experiments on the NegNLI Dataset
Finally, we conduct experiments on the NegNLI benchmark dataset (Hossain et al., 2020), where negation plays an important role for NLI tasks. As a baseline, we compare the reported performance of BERTNOT (Hosseini et al., 2021), which is a recently proposed remedy to improve PLMs' ability to understand negation. Since Hosseini et al. (2021) used BERT-base as a backbone model, we also apply the  $\mathrm{IM}^2$  method to BERT-base. The results are summarised in Table 9.
For both SNLI and MNLI, we observe that our approach outperforms BERTNOT in the NegNLI datasets, while yielding a comparable performance in the original development datasets. It is interest-
ing that our approach improves the understanding of negation in both MKR-NQ and NegNLI tasks. We conjecture that a leading cause is that the definitions of the meaning-matching dataset contain many negation expressions, which enables a model to learn their proposed meaning (see Table 12). The results suggest that our proposed approach is more efficient than BERTNOT, because the  $\mathrm{IM}^2$  method leverages less time and resources for training.
PLMs are at the core of many success stories in natural language processing (NLP). However, it remains unclear to what extent PLMs understand the syntactic and semantic properties of the human language. A series of probing tasks have been conducted on PLMs and have found them lacking or falling short on some language properties. Among the many findings of these probing tasks, PLMs have been found to be insensitive to the order of sentences when generating representations (Pham et al., 2021; Gupta et al., 2021; Sinha et al., 2021a), struggle to comprehend number-related representations (Wallace et al., 2019; Lin et al., 2020; Nogueira et al., 2021), and display a lack of semantic content understanding (Ravichander et al., 2020; Elazar et al., 2021).
In addition to the above faulty behaviours, Ettinger (2020) and Kassner and Schütze (2020) show that PLMs fail to comprehend negation, which is an important property of language in many natural language understanding (NLU) tasks. Ettinger (2020) check the ability of PLMs to understand the meaning of negation in given contexts. In their work, they check whether models are sensitive in their completions of sentences that either include negation or not. Under normal circumstances, the completions are expected to vary in truth depending on the presence or absence of negation in given sentences. Their results show that PLMs are insensitive to the impacts of negations when completing
sentences. Kassner and Schütze (2020) construct the negated LAMA dataset by inserting negation elements (e.g., "not") in the LAMA cloze questions (Petroni et al., 2019). They use negated and original question pairs to query PLMs and establish that models are equally prone to make the same predictions for both the original and negated questions. In a well-informed setting, it is expected that PLMs should make different predictions for the original and negated questions. This shows that PLMs struggle to comprehend negation.
In light of the highlighted faulty behaviours of PLMs, especially their struggle to comprehend negation, Hosseini et al. (2021) propose a remedy to alleviate the problem. In their remedy, they augment the language modelling objective with an unlikelihood objective (Welleck et al., 2020) based on negated sentences from the training corpus. They use a syntactic augmentation method to generate negated sentences. In this method, the dependency parse of the sentences, POS tags, and morphological information of each word are taken as input, and the negation of sentences is done using sets of dependency tree regular expression patterns, such as Semgrex (Chambers et al., 2007). During training, they replace objects in negated sentences with [MASK] tokens and use likelihood training to make the masked-out tokens unlikely under the PLM distribution. To ensure that negated sentences are factually false, they use the corresponding positive sentences as context for the unlikelihood prediction task.
Previous studies (e.g., Kassner and Schütze (2020)) have mostly limited the scope of the logical negation property only to the negation expressions (e.g., "no" and "not"). However, the core spirit of this property is the opposite meaning, which is not only limited to the negation. Welleck et al. (2020) consider negating sentences using dependency tree regular expression patterns. This widens the scope of negation, as it is not only limited to the negation expressions "no" and "not". However, their approach relies on other components, such as Semgrex, and dependency and POS parsers, which could impact the quality of the data, hence impact the models' performance. In this work, we consider other perturbation methods to generate the opposite-meaning sentences to investigate whether PLMs satisfy the logical negation property, and we propose a remedy, called intermediate-training on meaning-matching  $(\mathrm{IM}^2)$ , which hardly employs
additional linguistic components.
7 Summary and Outlook
In this work, we investigated PLMs' LNP. Compared to previous works that only examine negation expressions, we expanded the boundary of LNP to lexical semantics. We confirmed that PLMs are likely to violate LNP through extensive experiments.
We hypothesise that the distributional hypothesis is an insufficient basis for understanding the semantic meaning of texts. To alleviate the issue, we proposed a novel intermediate task: meaning-matching. Via experiments, we verified that meaning-matching is a stable intermediate task that substantially improves PLMs' understanding of negation and lexical semantic information while guaranteeing a positive transfer with multiple downstream tasks. Also, our approach produces a better performance on the negated NLI datasets compared to the unlikelihood training-based method, which leverages much more time and resources. Our work suggests that it is time to move beyond the distributional hypothesis to develop logically consistent and stable language models.
This work was partially supported by the Alan Turing Institute under the EPSRC grant EP/N510129/1, by the AXA Research Fund, by the EPSRC grant EP/R013667/1, and by the ESRC grant "Unlocking the Potential of AI for English Law". We also acknowledge the use of Oxford's ARC facility, of the EPSRC-funded Tier 2 facilities JADE (EP/P020275/1) and JADE II (EP/T022205/1), and of GPU computing support by Scan Computers International Ltd. Frank Mtumbuka was supported by the Rhodes Trust under a Rhodes Scholarship.
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. International Conference on Learning Representations.
Laura Aina, Raffaella Bernardi, and Raquel Fernandez. 2018. A distributional study of negated adjectives and antonyms. In Proceedings of the 5th Italian Conference on Computational Linguistics (CLiC-it 2018), volume 2253 of CEUR Workshop Proceedings.
Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49-72.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015), pages 632-642.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Oana-Maria Camburu, Brendan Shillingford, Pasquale Minervini, Thomas Lukasiewicz, and Phil Blunsom. 2020. Make up your mind! Adversarial generation of inconsistent natural language explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4157-4165.
Nathanael Chambers, Daniel Cer, Trond Grenager, David Hall, Chloe Kiddon, Bill MacCartney, Marie-Catherine de Marneffe, Daniel Ramage, Eric Yeh, and Christopher D. Manning. 2007. Learning alignments and leveraging natural logic. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 165-170.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pretraining text encoders as discriminators rather than generators. International Conference on Learning Representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9:1012-1031.
Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48.
Ashim Gupta, Giorgi Kvernadze, and Vivek Srikumar 2021. Bert & family eat word salad: Experiments with text understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12946-12954.
Zellig S. Harris. 1954. Distributional structure. Word, 10(2-3):146-162.
Matthew Honnibal and Mark Johnson. 2015. An improved non-monotonic transition system for dependency parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015), pages 1373-1378.
Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020. An analysis of natural language inference benchmarks through the lens of negation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), pages 9106-9118.
Arian Hosseini, Siva Reddy, Dzmitry Bahdanau, R. Devon Hjelm, Alessandro Sordoni, and Aaron Courville 2021. Understanding by understanding not: Modeling negation in language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics. Human Language Technologies, pages 1301-1312.
Richard Hudson. 1994. About  $37\%$  of word-tokens are nouns. Language, 70(2):331-339.
Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811-7818.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut 2019. ALBERT: A lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations.
Junying Liang and Haitao Liu. 2013. Noun distribution in natural languages. *Poznan Studies in Contemporary Linguistics*, 49(4):509-529.
Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xi-ang Ren. 2020. Birds have four legs?! NumerSense: Probing numerical commonsense knowledge of pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), pages 6862-6868
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual
representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1073-1094.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448.
I. A. Mel'čuk and A. K. Žolkovskij. 1970. Towards a functioning 'meaning-text' model of language. Linguistics, 8(57):10-47.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.
Jasmina Milicevic. 2006. A short guide to the meaning-text linguistic theory. Journal of Koralex, 8:187-233.
George A. Miller. 1995. WordNet: A lexical database for English. Communications of the ACM, 38(11):39-41.
Nikola Mrkšić, Diarmuid Žéaghdha, Blaise Thomson, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 142-148.
Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340-2353.
Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658-4664.
Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2021. Investigating the limitations of transformers with simple arithmetic tasks. arXiv preprint arXiv:2102.13019.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP 2014), pages 1532-1543.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP 2019), pages 2463-2473.
Thang Pham, Trung Bui, Long Mai, and Anh Nguyen. 2021. Out of order: How important is the sequential order of words in a sentence in natural language understanding tasks? In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1145-1160, Online. Association for Computational Linguistics.
Jason Phang, Thibault Févry, and Samuel R. Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. CoRR, abs/1811.01088.
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231-5247.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. 2020. On the systematicity of probing contextualized word representations: The case of hypernymy in BERT. In Proceedings of the 9th Joint Conference on Lexical and Computational Semantics, pages 88-102.
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021a. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021), pages 2888-2913.
Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau, and Adina Williams. 2021b. UnNatural Language Inference. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7329-7346.