SlowGuess/NAACL_2022_OCR huggingface dataset:
Ellie Pavlick and Chris Callison-Burch. 2016a. "babies" are "little" and most "problems" are "huge": Compositional entailment in adjective-nouns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2164-2173, Berlin, Germany. Association for Computational Linguistics.
Ellie Pavlick and Chris Callison-Burch. 2016b. So-called non-subsective adjectives. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, pages 114-119, Berlin, Germany. Association for Computational Linguistics.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Emily Pittler, Shane Bergsma, Dekang Lin, and Kenneth Church. 2010. Using web-scale n-grams to improve base NP parsing performance. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 886-894, Beijing, China. Coling 2010 Organizing Committee.
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699-2712, Online. Association for Computational Linguistics.
Vered Shwartz and Chris Waterson. 2018. Olive oil is made of olives, baby oil is made for babies: Interpreting noun compounds using paraphrases in a neural model. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 218-224, New Orleans, Louisiana. Association for Computational Linguistics.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanian Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Alexandra Teodorescu. 2006. Adjective ordering restrictions revisited. In 25th West Coast Conference on Formal Linguistics, pages 399-407. Cascadilla Proceedings Project.
David Vadas and James Curran. 2007. Adding noun phrase structure to the Penn Treebank. In Proceedings of the 45th Annual Meeting of the Association of
Computational Linguistics, pages 240-247, Prague, Czech Republic. Association for Computational Linguistics.
Johan Van Benthem. 1983. Determiners and logic. Linguistics and Philosophy, 6(4):447-478.
Jesse Vig. 2019. A multiscale visualization of attention in the transformer model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 37-42, Florence, Italy. Association for Computational Linguistics.
Sarah Wiegrefe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11-20, Hong Kong, China. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
A Dataset Construction Details
A.1 RNPC Statistics
NPs. RNPC has 1,299 NPs. For an NP in the form of Det  $\mathbf{M}_1\mathbf{M}_2\mathbf{N}$ , the two modifiers  $M_{1}$  and  $M_2$  can each belong to one of three possible semantic categories (intersective, subsective, or privative), resulting in nine possible combinations. We plot the distribution of NPs with different combinations in RNPC in Table 6. Note that the distribution is not balanced because certain categories (e.g., NPs containing privative modifiers) yield many more minority class examples for our three tasks (e.g., non-entailment in SPTE). Thus, considering the final class balance in RNPC tasks, we include more NPs of certain categories.
Training and test sets for finetuning. In the experiment where we finetune models on RNPC, described in Section 6, we split again the dataset for each task into a training set and a new test set, ensuring no overlap of modifiers occurring in the same position. The training set contains 200 examples, which are gradually provided to the model. The test set contains the remaining examples. Table 7 shows the number of examples for each task.
A.2 Crowdsourcing Details
In the construction of RNPC, we hire college students as crowdworkers for instance creation and label verification. Specifically, they are undergraduate and graduate students in an Artificial Intelligence class (CIS 421/521 and MCIT 521 at the University of Pennsylvania), with good English proficiency. Both tasks are given as optional extra credit assignments in the class. Participation is solely voluntary. Before participation, students can preview the tasks, and are given a clear description of how the data will be used at the beginning of the instructions.
During instance creation, we provide detailed instructions on how to write high-quality examples for each task, which can be found in the Supplementary Materials. Annotations are collected via Google Forms. With 100 valid instances (equivalent to 2.5-4.75 hours of work, depending on their proficiency), students can earn  $1\%$  in extra credit of the overall course grade.
During label verification, we host our questions on Amazon Mechanical Turk. We design a HIT type for each RNPC task, which is also included in the Supplementary Materials. With 600 correctly answered questions (equivalent to 3.5-4 hours of
M1/M2	Int.	Sub.	Pri.
Int.	13	37	74
Sub.	138	109	162
Pri.	99	420	250
Table 6: Number of NPs in RNPC with different combinations of modifier category in the  $M_{1}$  and  $M_{2}$  position. Possible categories include intersective, subsective, and privative.
Task	Train	Test
SPTE	200	963
MPTE	200	863
EPC	200	1,279
Table 7: Number of examples in the training and testing split for each RNPC task in the finetuning experiment.
Dataset	Train	Dev	Test
MNLI	392,702	20,000	20,000
SNLI	550,152	10,000	10,000
MPE	8,000	1,000	1,000
ADEPT	12,892	1,611	1,612
Table 8: Number of examples in existing datasets of the same format used for finetuning.
work), students can earn  $1\%$  in extra credit of the overall course grade. We calculate the inter-annotator agreement using Krippendorff's alpha.[17] The agreement is 0.843 for SPTE, 0.575 for MPTE, and 0.933 for EPC.
A.3 Debiasing and Anonymization
The collected data does not contain any information that names or uniquely identifies individual people or offensive content. We ensure this by 1) manually reviewing the set of extracted NPs from corpora, and filtering out any NP that contains any sensitive/offensive information, 2) not requesting any personal information during human annotation, and 3) manually reviewing each RNPC example written by the human participants.
B Existing Benchmarks for Finetuning
We use the following benchmark datasets for finetuning. Each of them has the same format as one of our RNPC tasks. Table 8 shows the number of examples in each dataset.
MNLI. The Multi-Genre Natural Language Inference corpus (Williams et al., 2018) is a dataset of 433k textual entailment examples, labeled as entailment, contradiction, or neutral. It covers a
$^{17}$ https://pypi.org/project/krippendorff
range of genres of spoken and written text. The language in the dataset is English. The corpus is released under the OANC's license, the Creative Commons Share-Alike 3.0 Unported License, and the Creative Commons Attribution 3.0 Unported Licenses, depending on the portion.
SNLI. The Stanford Natural Language Inference corpus (Bowman et al., 2015) is a crowdsourced dataset of textual entailment examples, labeled as entailment, contradiction, or neutral. The sentences are written by humans doing a novel grounded task based on image captioning. The language in the dataset is English. The dataset is released under the Creative Commons Attribution-ShareAlike 4.0 International License.
MPE. Lai et al. (2017) introduce a Multiple Premise Entailment Task dataset. This is a novel textual entailment task that requires inference over multiple premise sentences. Each example consists of four premise sentences (captions from a FLICKR30K image), one hypothesis sentence (a simplified FLICKR30K caption), and one label (entailment, neutral, or contradiction) that indicates the relationship between the set of four premises and the hypothesis. The language in the dataset is English. The license of the dataset is unspecified.
ADEPT. Emami et al. (2021) introduce a dataset of the Adjective-Dependent Plausibility Task (ADEPT). Each example contains a base sentence, and a slightly modified sentence obtained by adding an adjective to a noun in the base sentence. The dataset is created to support explorations into how certain classes of adjectives might influence the plausibility of events depicted in natural language sentences. The textual data come from Wikipedia, the Common Crawl, and ConceptNet. The language of the dataset is English. ADEPT is released under the CC BY-SA 3.0 license. It is intended to be used only for research, exploratory evaluation, and auditing, which our use is consistent with.
C Probing Pretrained LMs
When addressing question (a), we finetune pretrained LMs on existing benchmarks of the same format as each RNPC task, assuming that the finetuning process allows models to do textual inference in the required format. However, it is possible that this assumption does not hold, because LMs can overfit the finetuning data beyond just learning the format. Then even if the target knowledge is
present in pretrained LMs, catastrophic forgetting (Kemker et al., 2018) can happen during finetuning.
C.2 Task Conversion
We complement Section 5 with another experiment, where we directly probe pretrained LMs using a prompting method inspired by the line of work on LMs as knowledge bases (Petroni et al., 2019). Specifically, we convert each RNPC task to a likelihood comparison task:
SPTE. Given the original formulation which has a premise and a hypothesis, we define  $L_{entail}$  as the conditional likelihood that the hypothesis is necessarily true given the premise, assigned by an LM. Contrarily,  $L_{non-entail}$  stands for the conditional likelihood that the hypothesis is NOT necessarily true given the premise.[18] If  $L_{entail} > L_{non-entail}$ , the model is considered to predict entailment, and vice versa.
MPTE. The conversion method is the same as that for SPTE, except that in the conditional likelihood computation, we now consider the concatenation of two premises as the given condition.
EPC. Given the original formulation with two events, Event 1 and Event 2, we define  $L_{1}$  and  $L_{2}$  as the (unconditional) likelihood of Event 1 and Event 2 assigned by an LM, respectively. We then choose a threshold  $\theta$ ,[19] and compare it to the absolute difference between  $L_{1}$  and  $L_{2}$ . If the difference is smaller than  $\theta$ , we consider the model prediction as equally likely. Otherwise, the model prediction is more likely if  $L_{2}$  is higher, and less likely if  $L_{1}$  is higher.
For Causal LMs (e.g., GPT), the likelihood is computed with standard left-to-right language modeling scores. For Masked LMs (e.g., BERT, RoBERTa, BART), the likelihood is computed with pseudo-log-likelihood scores (Salazar et al., 2020).
C.3 Sanity Check
Before evaluating LMs on the converted RNPC, we perform a sanity check to see if our formalization makes sense to LMs, i.e., whether they understand the meaning of necessarily and not necessarily.
18 For example, if the original SPTE example has the premise This is the second green ball and the hypothesis This is the second ball, then  $L_{\text{entail}}$  equals to  $L$  (This is necessarily the second ball | This is the second green ball), and  $L_{\text{non-entail}}$  equals to  $L$  (This isn't necessarily the second ball | This is the second green ball).
In the range [0.1, 0.5, 1, 2, 3, 5], 0.5 is the empirical optimal.
Model	SPTE	MPTE	EPC
gpt2-base	59.4	52.8	33.4
gpt2-medium	62.6	53.6	33.6
gpt2-large	61.4	56.1	32.4
gpt2-x1	61.7	56.9	31.7
gpt3-ada	55.2	55.2	33.2
We write 50 sentence pairs for likelihood comparison, all consisting of simple commonsense knowledge. For example, comparing A human being is necessarily female and A human being isn't necessarily female, the second sentence should be more likely; while for Humans are necessarily mortal and Humans aren't necessarily mortal, the first sentence should be more likely. Such comparisons do not require any knowledge about recursive NPs, and involve only common entities and facts. If models understand necessarily and not necessarily correctly, they should find the task easy.
To our surprise, almost all Masked LMs we test (BERT-base/large, RoBERTa-base/large) fail the sanity check, mostly performing around chance (50 accuracy). However, most Causal LMs (GPT-2-base/medium/large/xl, GPT-3-ada) reasonably perform above chance, with accuracy scores ranging from 70 to 80. We suspect that pseudo-log-likelihood scores are not entirely suitable for our purposes; also, the task is harder than expected due to reporting bias, as the tested knowledge (e.g., not all humans are female) is potentially too obvious to be explicitly stated in the pretraining data.
C.4 Results
We evaluate LMs that pass the sanity check on the converted RNPC, and report their performance in Table 9. Despite the decent performance on the sanity check examples (70-80), the accuracy on RNPC is remarkably lower. Compared to our original results of probing the finetuned models, the optimal performance on SPTE and MPTE slightly improves, while accuracy on EPC decreases. However, the same patterns hold: most models perform around or slightly above chance, with a large difference from human performance. These findings further strengthen our answer to question (a), i.e. LMs do not inherently have the knowledge to interpret recursive NPs.
Table 9: Accuracy of SOTA pretrained models directly evaluated on RNPC tasks.
Model	Acc.	P	R	F1
BERT-base (SNLI)	49.8	49.9	77.0	60.5
BERT-base (MNLI)	51.3	50.7	97.8	66.8
RoBERTa-large (MNLI)	61.1	56.3	99.1	71.9
BART-large (MNLI)	59.3	55.1	97.9	70.7
Table 10: Full results of SOTA models evaluated on SPTE. The finetuning dataset is in brackets.
Model	Acc.	P	R	F1
BERT-base	47.2	48.0	44.0	45.9
BERT-large	41.5	34.2	16.3	22.1
RoBERTa-base	51.1	51.0	100.0	67.5
RoBERTa-large	50.9	50.9	100.0	67.5
GPT3-ada	52.0	51.5	97.0	67.3
GPT3-curie	54.1	52.6	97.4	68.4
Table 11: Full results of SOTA models evaluated on MPTE. The finetuning dataset is MPE for all models.
Model	Acc.	P	R	F1
BERT-base	31.6	29.2	31.6	22.4
BERT-large	32.2	27.7	32.2	23.7
RoBERTa-base	31.0	46.8	31.0	22.3
RoBERTa-large	39.5	54.1	39.5	32.7
GPT3-ada	35.2	40.2	35.2	28.3
GPT3-curie	38.7	69.9	38.7	32.8
Table 12: Full results of SOTA models evaluated on EPC. The finetuning dataset is ADEPT for all models.
In Section 5, we evaluate SOTA LMs on RNPC tasks. In addition to accuracy, we also report precision, recall, and F-1 score here. Tables 10, 11 and 12 show the full results for each task, respectively.
E.1 Models Finetuned on Existing Benchmarks
In Section 5, we evaluate SOTA LMs finetuned on existing benchmarks of the same format on RNPC. We use four different pretrained models, BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019b), BART (Lewis et al., 2020), and GPT3 (Brown et al., 2020), in different sizes. The first three are implemented with HuggingFace Transformers $^{20}$ , and the last is from OpenAI's standard API $^{21}$ .
The pretrained model checkpoints we use include: bert-base-uncased (110M parameters), bert-large-uncased (336M parameters), roberta-base (125M param
$^{20}$ https://github.com/huggingface/ transformers
21https://beta.openai.com/docs/ api-reference
eters), roberta-large (335M parameters), facebook/bart-large (406M parameters), GPT3-ada (350M parameters), and GPT3-curie (6.7B parameters).[22] Their licenses include Apache License 2.0 (BERT and BART), GNU General Public License v2.0 (RoBERTa), and MIT license (GPT3).
Due to the size of MNLI and SNLI, we use existing checkpoints available on the Huggingface Transformers model hub. For all other datasets, we finetune the pretrained models using the SequenceClassification pipeline on Huggingface, or the standard prompt completion finetuning API on OpenAI.[23] The finetuning scripts are adapted from the text-classification example in the HuggingFace Transformers repository.[24] We performed hyperparameter search in the following range:
-batch size:[4,8,16,32]
- learning rate: [1e-5, 1e-6]
- number of epochs: [2, 3, 5]
-max sequence length:[64,128]
The optimal hyperparameter values and finetuned models are available on the HuggingFace model hub.
We run our finetuning experiments on an NVIDIA GeForce RTX 2080 Ti GPU, with half-precision floating point format (FP16). The finetuning takes 2 to 5 hours depending on the task.
E.2 Models Finetuned on RNPC
In Section 6, we address the question of whether LMs can learn the meaning of recursive NPs. We finetune each model from Section E.1 on an increasing number of examples of each RNPC task. The model architectures, the pipelines used, the range of hyperparameter search, and the computing resources used are all the same as in the previous subsection. After being finetuned on 200 examples, the best performing models are RoBERTa-large (MNLI) for SPTE, RoBERTa-base (MPE) for MPTE, and RoBERTA-large (ADEPT) for EPC. The optimal hyperparameter values and finetuned models on the full 200 examples of each RNPC task are available on the HuggingFace model hub.
Figure 6: An illustration of the Edge Probing method. Figure adapted from Tenney et al. (2019).
E.3 The "Edge Probing" Method
In Section 7, we adopt the Edge Probing technique from Tenney et al. (2019) to investigate if the modifier category feature can be learned from our tasks.
To reintroduce the general idea of this method, consider the following setup: we have data  $D = \{(x_{1},y_{1}),(x_{2},y_{2}),\ldots ,(x_{n},y_{n})\}$ , where  $(x_{1},x_{2},\dots,x_{n})$  are the model representations to be probed and  $(y_{1},y_{2},\dots,y_{n})$  are the labels of a linguistic feature we are interested in probing for. The goal is to see if  $(x_{1},x_{2},\dots,x_{n})$  encodes  $(y_{1},y_{2},\dots,y_{n})$ .
In our case, given an NP of the form Det  $\mathbf{M}_1\mathbf{M}_2\mathbf{N}$ $(x_{1},x_{2},\dots,x_{n})$  are the token representations of the best-performing models after being finetuned on each RNPC task, as mentioned in Section E.2, and  $(y_{1},y_{2},\dots,y_{n})$  are the semantic categories of  $\mathbf{M}_1$  and  $\mathbf{M}_2$ .
We freeze the representations  $(x_{1}, x_{2}, \ldots, x_{n})$ , and attach a simple auxiliary classifier (or probing classifier) on top of them. The auxiliary classifier is trained to predict the semantic category  $y_{i}$  corresponding to every modifier token representation  $x_{i}$ . In our case, we choose a linear classifier as the probing classifier. The assumption is that the expressivity of the auxiliary classifier should be limited, or otherwise it might learn the target feature itself. Based on this assumption, the accuracy of predicting  $y$  indicates the extent to which the model representations encode the target feature. The method is illustrated in Figure 6.
Specifically, the linear classifier is an SGDClassifier implemented with scikit-learn.[25] We perform hyperparameter search in the following range:
$^{25}$ https://scikit-learn.org
22All models above are available at https: //huggingface.co/transformers/v4.8.   
2/pretrained_models.html or https: //beta.openai.com 23https://beta.openai.com/docs/   
api-reference/fine-tunes 24https://github.com/huggingface/   
transformers/tree/master/examples/legacy
- loss: [hinge, log, modified_huber, squared_hinge, perceptron]
- warm_start: [True, False]
- penalty: [12, 11, elasticnet]
- alpha: [0.0001, 0.001, 0.01, 0.1]
-n_jobs:[-1]
-max_iter: [500, 1000, 2000]
- random_state: [0]
- eta0: [0.1, 0.01, 0.001, 0.0001]
- learning_rate: [optimal, invscaling, adaptive]
The optimal hyperparameter values for the best-performing models of each RNPC task from Section E.2 are available on the HuggingFace model hub.
E.4 Models Used for Extrinsic Evaluation
In Section 8, one of the models we compare to is GPT3. We exploit three versions of different sizes, GPT3-ada, GPT3-curie, and GPT3-davinci, using the same API described in Section E.1. The models are evaluated in a zero-shot setting, where we only provide the following task description:
Given a "how-to" query, determine if the task is harmful to a teenager. The answer should be Yes or No.
Each input question is then formatted as:
Question: Is “{how-to query}” harmful?
For every how-to query in our test set, we use the concatenation of the task description and the instantiated input question as the prompt, and let the model generate a one-token continuation. The top generated token is always Yes or No, implying that GPT3 has a good understanding of the task format.
F.1 Limitations
Assumptions. One assumption we make in answering question (a) is that LMs finetuned on existing benchmarks can learn the required format without overfitting the specific domains of the finetuning data. Suppose this assumption does not hold, then even if the target knowledge is present in pretrained LMs, they can "forget" it during finetuning. Therefore, the finetuning process does not allow us to elicit the target knowledge from pretrained LMs. To address this issue, we complement the behavioral test probing method with another experiment
to directly probe the pretrained LMs via likelihood scoring. See Section C for details.
Another assumption occurs in our answer to question (d). We assume that a query is harmful if it contains a harmful entity. However, in practice, there can be queries like How to prevent a fire, which does contain a harmful entity (fire) but is precautionary instead of harmful. Our model does not take into account factors like predicates in context, and will therefore identify all such cases as false positives.
Scope of claims. Our first three claims (i.e. answers to question (a)-(c)) are only verified to hold on the RNPC dataset, which 1) is in English and 2) mainly consists of NPs in the news domain. Our last claim (i.e. answer to question (d)) is only verified to hold on the harm detection dataset we collect, which 1) is also in English, 2) consists of how-to queries in the domain of human activities, and 3) is annotated based on a non-exhaustive keyword list of harmful entities.
Moreover, part of our answer to question (b) (i.e. LMs have learned the feature of modifier semantic category from RNPC) is qualitative. The absolute increase in the probing accuracy after finetuning is limited, so it is likely not the entire picture. Quantifying to what extent LMs have learned this feature is an interesting direction for future work.
F.2 Risks
The risks associated with the study are minimal. Harm detection models. Our harm detection models are intended for research purposes only. They are designed for specific types of harmful queries, i.e. those with harmful entities. One should not deploy them directly in real life since they are by no means applicable under all scenarios.
Data collection. Our human participants may experience slight discomfort due to boredom during data collection. To minimize this, we make sure that it is entirely voluntary to participate and discontinue at any time.
F.3 Intended Use