SlowGuess/NAACL_2022_OCR huggingface dataset:
Is “My Favorite New Movie” My Favorite Movie? Probing the Understanding of Recursive Noun Phrases
Qing Lyu<sup>1</sup>
Hua Zheng
Daxin Li3
Li Zhang
Marianna Apidianaki
Chris Callison-Burch
$^{1}$ Department of Computer and Information Science, University of Pennsylvania
$^{2}$ Key Lab of Computational Linguistics (MOE), Peking University
$^{3}$ Department of Linguistics, University of Pennsylvania
{lyuqing,zharry,marapi,ccb}@seas.upenn.edu
zhenghua@pku.edu.cn
daoxinli@sas.upenn.edu
Recursive noun phrases (NPs) have interesting semantic properties. For example, my favorite new movie is not necessarily my favorite movie, whereas my new favorite movie is. This is common sense to humans, yet it is unknown whether language models have such knowledge. We introduce the Recursive Noun Phrase Challenge (RNPC), a dataset of three textual inference tasks involving textual entailment and event plausibility comparison, precisely targeting the understanding of recursive NPs. When evaluated on RNPC, state-of-the-art Transformer models only perform around chance. Still, we show that such knowledge is learnable with appropriate data. We further probe the models for relevant linguistic features that can be learned from our tasks, including modifier semantic category and modifier scope. Finally, models trained on RNPC achieve strong zero-shot performance on an extrinsic Harm Detection evaluation task, showing the usefulness of the understanding of recursive NPs in downstream applications.
Recursion, the self-embedding of a linguistic structure, constitutes a fundamental property of human language. Due to its hierarchical structure, it poses many challenges to human language acquisition. One such challenge occurs in the context of recursive Noun Phrases (NPs), i.e., NPs with multiple prenominal modifiers. For instance, in Figure 1, when asked to point to the second green ball in a series of balls, children sometimes erroneously point to the second and green ball (intersective interpretation), instead of the second among green balls (recursive interpretation) (Matthei, 1982; Hamburger and Crain, 1984; Marcilese et al., 2013).
"Point to the second green ball."
Figure 1: The intersective (incorrect) and the recursive (correct) interpretation of the second green ball.
We investigate whether language models (LMs) make similar errors, since the understanding of recursive NPs is also fundamental in real-world AI applications. For example, a summarization system should know that the former US president cannot be shortened as the president, since they are no longer in power. Also, a self-driving car asked to take the first left-hand exit should not assume that it is always the first exit.
Previous work has studied the syntactic parsing of recursive NPs (Nakov and Hearst, 2005; Pitler et al., 2010), as well as the semantic categorization of modifiers in NPs with only one prenominal modifier (Kamp and Partee, 1995; McCrae et al., 2014). However, neither parsing nor modifier categorization alone can sufficiently capture the meaning of recursive NPs ( $\S 2$ ).
In this paper, using recursive NPs with two modifiers as our test-bed, we address the following questions about LMs' understanding of recursion:
(a) Is the knowledge of how to interpret recursive NPs present in LMs (§5)? We propose
1Our code and data are available at https://github. com/veronica320/Recursive-NPs.
Human Language Technologies, pages 5286 - 5302
Task	ID	Input	Label
Single-Premise
Textual Entailment
(SPTE)	(1a)	Premise: This is my new favorite movie.
Hypothesis: This is my favorite movie.	Entailment
(1b)	Premise: This is my favorite new movie.
Hypothesis: This is my favorite movie.	Non-Entailment
Multi-Premise
Textual Entailment
(MPTE)	(2a)	Premise 1: He is a skillful American violinist.
Premise 2: He is a father.
Hypothesis: He is an American father.	Entailment
(2b)	Premise 1: He is a skillful American violinist.
Premise 2: He is a father.
Hypothesis: He is a skillful father.	Non-Entailment
Event Plausibility
Comparison (EPC)	(3a)	Event 1: The actress is known by everyone.
Event 2: The famous former actress is known by everyone.	(Event 2 is)
More Plausible
(3b)	Event 1: The actress lives in France.
Event 2: The famous former actress lives in France.	(Event 2 is)
Equally Plausible
(3c)	Event 1: The actress stars in many latest movies.
Event 2: The famous former actress stars in many latest movies.	(Event 2 is)
Less Plausible
Table 1: Examples for each task in our dataset. The NPs of interest are underlined. Differences between examples are in bold. See Section 3 for details.
the Recursive Noun Phrase Challenge (RNPC), a challenge set containing three classification tasks: Single-Premise Textual Entailment, Multi-Premise Textual Entailment, and Event Plausibility Comparison ( $\S 3$ ). Table 1 provides examples for each task. Results show that state-of-the-art (SOTA) LMs finetuned on standard benchmarks of the same format (e.g., MNLI (Williams et al., 2018)) all struggle on our dataset, suggesting that the target knowledge is not readily available.
(b) Is such knowledge learnable with appropriate data ( $\S 6$ )? We adopt the challenge set analysis technique proposed by Liu et al. (2019a), which exposes models to a small amount of data and assesses how well they can adapt. All models achieve a noticeable performance improvement with as few as 200 examples, indicating that the target knowledge is potentially learnable.
(c) What can models learn from recursive NPs (§7)? We probe the finetuned models for two well-studied linguistic features in previous work, modifier semantic category and modifier scope. We show that both features can be learned from RNPC, with techniques including edge probing (Tenney et al., 2019) and attention visualization (Vig, 2019).
(d) Is such knowledge useful for downstream tasks ( $\S 8$ )? When evaluated on an extrinsic Harm Detection task, models finetuned on RNPC achieve strong zero-shot performance. This shows that the understanding of recursive NPs can benefit downstream language understanding tasks.
In summary, our work identifies an interesting linguistic phenomenon that is common sense to
humans but challenging for models. It contributes to the characterization of LMs' limitations and capabilities in language understanding.
Noun Phrases (NPs) have been extensively studied in both linguistics and NLP, primarily from the following perspectives.
Syntactic structure. A line of work focuses on the syntactic structure of NPs, which essentially explains the modifier scope (Campbell, 2002) in NPs. One classic task is NP bracketing, i.e., deciding whether an NP is right-branching (e.g., [world [oil prices]]) or left-branching (e.g., [[crude oil] prices]) (Lauer, 1995; Nakov and Hearst, 2005). A harder task is full parsing (Vadas and Curran, 2007; Pitler et al., 2010), i.e., reconstructing the complete dependency tree.
Modifier semantics. Another line of research revolves around the semantics of simple modifier-noun composition, starting with ways to categorize modifiers based on their inference patterns (Kamp and Partee, 1995; Bouillon and Viegas, 1999; Chierchia and McConnell-Ginet, 2000). With  $M$  as the modifier and  $N$  as the noun, a representative taxonomy summarized by McCrae et al. (2014) is:
(1) intersective:  $X$  is a  $MN \Rightarrow X$  is  $M \wedge X$  is a  $N$ , e.g., "an American surgeon" describes someone who is both American and a surgeon;
(2) **subjective:**  $X$  is a  $MN \Rightarrow X$  is a  $N$ , but  $X$  is a  $MN \Rightarrow X$  is  $M$ , e.g., someone who is "a skillful surgeon" is not necessarily skillful in all disciplines;
(3) privative:  $X$  is a  $MN \Rightarrow X$  is a  $N$ , e.g., "a former surgeon" describes someone who is no longer a surgeon.
Despite the variations $^{2}$  and debates $^{3}$  on the taxonomy, we follow these conventional terms in subsequent sections.
With the advances in NLP, more recent works start modeling the semantics of simple modifier-noun constructions with first-order logic (McCrae et al., 2014), linear mapping (Baroni and Zamparelli, 2010), and other explicit compositional operations (Boleda et al., 2012, 2013). In particular, Pavlick and Callison-Burch (2016a,b) propose a novel contextualized inference-based approach. They define the Add-One Entailment task with natural contexts from textual corpora, where the hypothesis differs from the premise by the insertion of one modifier. For example, The **crowd** roared entails *The enthusiastic crowd* roared, though *enthusiastic crowd* denotes a subset of *crowd* without context. However, natural contexts also introduce complications from monotonicity (Van Benthem, 1983). For instance, red apple entails apple, but *He didn't eat any red apple* does not entail *He didn't eat any apple* due to the downward entailment context. In our proposed approach, we handle this issue by controlling for context monotonicity.
Other related work explores which attributes of the head noun are affected by the presence of modifiers. Mullenbach et al. (2019) look at how modifiers project from a noun to its parts (e.g., does a red jeep have red tires?). Emami et al. (2021) test the likelihood change of an event when a modifier is added (e.g., a false key is less likely to open a door than a key). Apidianaki and Garí Soler (2021) study the prototypical properties of nouns (e.g., a strawberry entails a red strawberry). Researchers also examine the interpretation of noun compounds (Shwartz and Waterson, 2018; Hendrickx et al., 2013) (e.g., olive oil is made of olives, while baby oil is made for babies).
Summary. Neither syntactic parsing nor modifier semantics alone can fully capture the meaning of recursive NPs. In terms of syntax, modifier scope cannot always explain NPs due to the influence from modifier semantics. For instance, a [big] [fake
gun]] and a [big [black gun]] have the same structure but different inference patterns, i.e. only the latter is a gun. Meanwhile, modifier category itself does not suffice without taking into account modifier scope. For example, a so-called healthy food and a so-called homeopathy expert start with the same privative modifier (so-called). However, so-called questions truthfulness of the second modifier (healthy) in the former case while that of the noun (expert) in the latter. Therefore, we introduce a dataset containing three novel and challenging textual inference tasks, which rely on the interplay of syntax and semantics in determining the meaning of recursive NPs.
Our dataset contains three tasks. Let us denote a canonical two-modifier recursive NP by Det  $\mathbf{M}_1\mathbf{M}_2\mathbf{N}$  (Determiner, Modifier 1, Modifier 2, Noun). With this notation, the tasks are outlined below. See Table 1 for concrete examples.
Single-Premise Textual Entailment (SPTE) follows the conventional TE task format. Given a premise and a hypothesis, the model decides whether the premise semantically entails the hypothesis. The labels include entailment and non-entailment. An SPTE example can be represented in regular expression as:
Premise:  $P\text{Det}M_1M_2N$
Hypothesis:  $P\text{Det}(M_1|M_2)?N$
Label: entailment|non-entailment
where  $\mathbf{P}$  is a sentence prefix, which can be instantiated as This is/He is/She is, etc., depending on the NP. Intuitively, this task tests whether an NP entails its various components. This holds for most simple NPs (e.g., the second ball entails ball), but recursive NPs offer interesting counterexamples (e.g., (1b) in Table 1).
Multi-Premise Textual Entailment (MPTE) is adapted from the attributive propagation test described in Lalisse (2015). The format differs from SPTE only in that it has two premises instead of one. Given that both are true, the task is to determine whether the hypothesis is also true. The first premise is of the same form as in SPTE. The second premise contains a noun other than N, denoted
4We do not distinguish between neutral and contradiction in order to minimize label ambiguity.
2For example, other studies call category (3) "non-subjective" instead, and further decompose it into "privative"  $(X$  is a  $MN$  contradicts  $X$  is a  $N$ , e.g., fake) and "nonprivative"  $(X$  is a  $MN$  is neutral to  $X$  is a  $N$ , e.g., alleged). Some linguists (for example, Partee (2010)) argue that (3) should be subsumed by (2), since privative modifiers can coerce the noun they modify into a looser interpretation.
Category	Count	Examples: modifier (ATTRIBUTE)
Intersective	296	red (COLOR), female (GENDER), German (NATIONALITY)
Subsective	269	short (HEIGHT), small (SIZE), far (DISTANCE)
Privative	124	former (TIME), vice (AUTHORITY), fake (AUTHENTICITY)
by  $\mathbf{N_2}$ . A regular expression representation is:
Premise 1:  $P$  Det  $M_1M_2N$
Premise 2:  $P$  Det  $N_{2}$
Hypothesis:  $P\operatorname{Det}\left(M_1|M_2\right)N_2$
Label:entailment|non-entailment
This test targets the compositionality of modifiers and nouns. While most of the time a modifier can be freely “detached” and “attached” (e.g., (2a)), sometimes it cannot (e.g., (2b)).
Event Plausibility Comparison (EPC) follows the task formalization by Emami et al. (2021) for single-modifier NPs. Given two events, Event1 and Event2, a model needs to assess the plausibility of Event2 compared to that of Event1. The two events have the same event predicate E, and differ only in the NP. A regular expression representation is:
Event 1: Det  $(M_1|M_2)?$  N E
Event 2: Det  $M_{1}M_{2}NE$
Label: more|equally|less plausible
This task tests the influence of adding modifier(s) on the plausibility of different events about the noun. Not all events are affected in the same way: in (3), stars in many latest movies becomes less plausible, while is known by everyone is more so.
We choose the three tasks defined above because they allow us to study different interesting properties of recursive NPs that conventional parsing tasks do not. For example, SPTE is convenient for comparing the impact of modifier order on the meaning of the NP (e.g., (1a) and (1b)); MPTE precisely reflects the property of subsective modifiers (e.g., skillful); whereas EPC is suitable for NPs with privative modifiers, since the other formats often cause ambiguity in this case.[6]
Table 2: Statistics and examples for each semantic category in our modifier lexicon.
Task	Total	Entail	Non-entail
SPTE	1,163	582	581
Task	Total	Entail	Non-entail
MPTE	1,063	541	522
Task	Total	More	Equal Less
EPC	1,479	508	392 579
Table 3: Number of examples in each RNPC task. Entail/Non-entail stand for Entailment/Non-entailment, and More/Equal/Less stand for More Plausible/Equally Plausible/Less Plausible.
4 Dataset Construction
Our dataset is constructed in four stages: (a) modifier lexicon construction, (b) NP extraction and selection, (c) instance creation and review, and (d) label verification. Among them, (c) and (d) involve crowdsourcing.<sup>7</sup>
Modifier lexicon construction. We first construct a lexicon of modifiers following the taxonomy in Section 2 (McCrae et al., 2014). We include modifiers studied in relevant linguistics literature (Nayak et al., 2014; Lalisse, 2015) and complement the list with modifiers that are missing or have not been addressed before under this lens (for example, modifiers that describe material, such as wooden, can also be viewed as privative). Each entry in the lexicon contains the modifier itself, its category (intersective, subsective, or privative), and its attribute (e.g., green is a COLOR). In total, the lexicon contains 689 modifiers, the largest resource of this kind. See Table 2 for category distribution and examples.
NP extraction and selection. Next, we collect recursive NPs from a variety of resources: linguistics literature (Matthei, 1982; Abdullah and Frost, 2005; Teodorescu, 2006; Morzycki, 2016), text corpora (Penn Treebank (Marcus et al., 1993) and the Annotated Gigaword corpus (Napoles et al., 2012)), and our creation. From text corpora, we extract all NPs with more than two consecutive modifiers in our lexicon, and manually select NPs considering a set of factors: lexical diversity, class balance, whether there is an interaction between the modifiers, etc. Finally, we complement the set with deliberately designed challenging cases of our invention, resulting in 1,299 NPs in total.
a kind of fur (Partee, 2010). Annotators would thus probably disagree on the label if it were an SPTE example.
See more statistics, crowdsourcing setup, and agreement details in Appendix A; see annotation guidelines and HIT design in the Supplementary Materials.
For both premises to hold at the same time, we need an  $\mathbf{N}_2$  that can refer to the same entity as  $\mathbf{N}$ .
For example, fake fur might or might not be considered
Figure 2: Given SOTA models finetuned on existing benchmark(s) of the same format as each RNPC task, we compare their accuracy on these benchmark(s) and on the RNPC task. The dotted line represents the majority baseline, and the solid line stands for human performance. Models for SPTE are finetuned on MNLI and SNLI, while models for the other two tasks are finetuned on MPE and ADEPT, respectively.
Acc. on existing benchmark Acc. on RNPC task
Instance creation and review. We hire college students to write examples for the three tasks based on our collection of NPs. Each student is given a screening test containing five NPs. If  $\geq 75\%$  of their created examples across all tasks are valid, they are qualified to continue. Each instance is then reviewed and/or revised by one of the authors, resulting in 8,260 valid instances.
Label verification. We again hire college students to verify instance labels via Amazon Mechanical Turk. Each task has a screening test of 10 easy instances with an unambiguous answer, and only students with an accuracy of  $\geq 90\%$  can proceed. During the official annotation, a HIT contains 10 questions of a task, including one control question. Each HIT is completed by three people, excluding its creator. Annotations are then filtered based on the accuracy on control questions and the time used. Only examples with  $\geq 2$  people agreeing with the gold label are retained, yielding 4,567 examples. We then down-sample the examples in each task for a relatively balanced ratio among classes, resulting in 3,705 examples. See Table 3 for details.
5 Do LMs understand recursive NPs?
To answer question (a), whether the knowledge of how to interpret recursive NPs is present in pretrained LMs, we use the "behavioral test" probing method (Belinkov et al., 2020). Namely, we evaluate SOTA models finetuned on existing benchmark(s) of the same format as each RNPC task.
The rationale is that LMs should acquire the ability of textual inference in the required format during finetuning, which allows us to elicit their potential knowledge about recursive NPs.
Experimental setup. We consider the following datasets that address similar phenomena as our tasks: (1) MNLI (Williams et al., 2018) and SNLI (Bowman et al., 2015) for our SPTE; (2) MPE (Lai et al., 2017) for our MPTE; and (3) ADEPT (Emami et al., 2021) for our EPC. We choose SOTA and close-to-SOTA models on these benchmarks as probing candidates, including BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019b), BART (Lewis et al., 2020), and GPT3 (Brown et al., 2020).
Results and analysis. We evaluate the finetuned models on each RNPC task. When the finetuning dataset has more classes than our task does, we map the model prediction to one of our classes by summing probability scores.[11] Figure 2 compares the performance of the models on the relevant
${}^{9}$  LMs can also overfit the finetuning dataset and thus "forget" the target knowledge acquired during pretraining. Thus, we also directly probe the pretrained LMs in a complementary "likelihood scoring" experiment, described in Appendix C.
Due to the size of MNLI and SNLI, we only evaluate available checkpoints from the Huggingface Transformers model hub. For the other two benchmarks, all models are trained by us. Also, the largest GPT3-davinci is unavailable for finetuning and thus excluded. See Appendices B and E.1 for dataset, model and hyperparameter details.
For example, for a model trained on MNLI (with three labels), we compare the score of entailment and the summed score of neutral and contradiction. If the former is higher, we predict entailment on SPTE; otherwise non-entailment. Empirically, this strategy results in higher performance than directly mapping the highest-score MNLI label to its corresponding SPTE label.
${}^{8}$  Specifically,undergraduate and graduate students in an Artificial Intelligence class.
Task	ID	Input	Gold Label	Predicted Label
Single-Premise	(1a)	Premise: This is my new favorite movie. Hypothesis: This is my favorite movie.	Entailment	Entailment✔
Textual Entailment	(1b)	Premise: This is my favorite new movie. Hypothesis: This is my favorite movie.	Non-Entailment	EntailmentX
Multi-Premise	(2a)	Premise 1: He is a short American basketball player.	Entailment	Entailment✔
Premise 2: He is a man.
Hypothesis: He is an American man.
Textual Entailment	Premise 1: He is a short American basketball player.
(2b)	Premise 2: He is a man.	Non-Entailment	EntailmentX
Hypothesis: He is a short man.
Event Plausibility Comparison	(3a)	Event 1: An animal can be harmful to people.	Less Plausible	Less Plausible✔
Event 2: A dead dangerous animal can be harmful to people.
(3b)	Event 1: An animal can be harmful to people.	More Plausible	Less PlausibleX
Event 2: A dangerous dead animal can be harmful to people.
Table 4: Minimal-pair examples where the best-performing models make errors for each RNPC task. Differences between each pair are underlined.
benchmarks and our tasks. We also include human performance, calculated by averaging the accuracy of three college student annotators on a random sample of 300 examples for each task.
All models struggle on RNPC with performance around chance, while human accuracy is constantly above 90. On SPTE and MPTE, almost all models have a high false-positive rate. As long as all tokens in the hypothesis (e.g., This is the second ball) appear in the premise (e.g., This is the second green ball), they tend to predict entailment, indicating that they are making the same intersective interpretation errors as children do. On EPC, most models over-predict equally plausible, arguably due to the class imbalance during finetuning. This also shows that our task is not trivially solvable by models that understand non-recursive NPs, which the finetuning dataset comprises.
Next, we closely examine the best-performing models on each task, including RoBERTa-large finetuned on MNLI, GPT3-curie finetuned on MPE, and RoBERTA-large finetuned on ADEPT. On MPTE and EPC, even the best model barely surpasses chance performance. On SPTE, the best accuracy (61.2) is still unimpressive for a binary classification task. To understand where exactly the models fail, we further present a qualitative minimal-pair analysis in Table 4. On SPTE, the two examples differ only in the order of modifiers (new and favorite) in the premise, leading to opposite labels. However, the model predicts entailment for both, suggesting its insensitivity to subtle meaning differences incurred by modifier order changes. On MPTE, the difference between the two examples lies in the modifier in the hypothesis, an Ameri
can man vs. a short man. As basketball players are generally tall, the second hypothesis should not be entailed. Again, the model predicts entailment for both cases, which shows its lack of relevant world knowledge. Finally, on EPC, a dead dangerous animal and a dangerous dead animal have subtly different meanings – the former refers to a dangerous animal that is dead (e.g., a dead lion, which is no longer harmful to people), while the latter refers to a dead animal that has become dangerous (e.g., a dead squirrel carrying viruses, which is indeed harmful). The model fails to distinguish between them, predicting less plausible for both. All the above observations show that the knowledge for interpreting recursive NPs is not present in LM representations.
6 Can LMs Learn the Meaning of Recursive NPs?