SlowGuess/NAACL_2022_OCR huggingface dataset:
John S. Y. Lee, Ho Hung Lim
Department of Linguistics and Translation
City University of Hong Kong Hong Kong SAR, China
jsylee@cityu.edu.hk
hhlim3@cityu.edu.hk
Carol Webster
College of Professional and
The Hong Kong Polytechnic University
carol.webster@cpce-polyu.edu.hk
Commonly found in academic and formal texts, a nominalization uses a verbal noun to describe an event associated with its corresponding verb. Nominalizations can be difficult to interpret because of ambiguous semantic relations between the verbal noun and its arguments. Automatic generation of clausal paraphrases for nominalizations can help disambiguate their meaning. However, previous work has not identified cases where it is awkward or impossible to paraphrase a compound nominalization. This paper investigates unsupervised prediction of paraphrasability, which determines whether the prenominal modifier of a nominalization can be re-written as a noun or adverb in a clausal paraphrase. We adopt the approach of overgenerating candidate paraphrases followed by candidate ranking with a neural language model. In experiments on an English dataset, we show that features from an Abstract Meaning Representation graph lead to statistically significant improvement in both paraphrasability prediction and paraphrase generation.
A nominalization is a noun (e.g., "response") that is morphologically derived from a verb ("respond"), and that designates some aspects of the event referred to by the verb (Quirk et al., 1985). In a compound nominalization, this deverbal noun may have both prenominal and postnominal modifiers. The prenominal modifier can be a noun (e.g., "police response to the rioting") or an adjective ("bodily injury to a friend"), while postnominal modifiers are prepositional phrases ("presidential nomination of Harrison").
Academic and other formal texts utilize nominalization extensively to produce a compact and abstract writing style. The meaning of compound nominalizations can however be difficult to interpret because of ambiguous semantic relations be
tween the deverbal noun and its modifiers. In particular, the prenominal modifier can play multiple semantic roles in the corresponding predicate or clausal paraphrase: as a subject (e.g., "the police response"  $\rightarrow$  "the police responds"); as an object ("bodily injury"  $\rightarrow$  "injure the body"); as an oblique ("presidential nomination"  $\rightarrow$  "nominate as president"; as an adverb ("symbolic admission"  $\rightarrow$  "admit symbolically"; or none of the above ("stellar performance"  $\nrightarrow$  "a star performs").
The paraphrasability of the prenominal modifier — whether it describes an entity, the manner of an action, or neither — therefore has direct impact on NLP tasks that require interpretation of compound nominalizations. This ambiguity affects accuracy in relation extraction, which is important for information retrieval and question answering (Greenwood, 2004; Klein et al., 2020). A machine translation system must also be able to render the deverbal noun and its prenominal modifier properly when there is no equivalent nominalization in the target language. Further, paraphrasability prediction could benefit nominal semantic role labeling, which needs to identify the role played by the prenominal modifier (Lapata, 2002; Padó et al., 2008; Kilicoglu et al., 2010). Finally, it is critical for nominalization paraphrasing. When a clausal paraphrase is not available for the input nominalization, approaches that do not consider paraphrasability may produce an invalid or misleading output (Lee et al., 2021).
This study focuses on English, the dominant language for academic texts. It aims to make two contributions. First, we enlarge an existing dataset to cover three paraphrasability categories for prenominal modifiers in a compound nominalization (paraphrased as noun, as adverb or non-paraphrasable). Second, we extend an algorithm to take paraphrasability into account, and show that features from Abstract Meaning Representation graphs improve performance in both paraphrasabil
Human Language Technologies, pages 3254 - 3263
ity prediction and paraphrase generation.
The rest of the paper is organized as follows. After defining our task (Section 2), we summarize previous research (Section 3). We then describe our dataset (Section 4) and present our approach (Sections 5-6). Then, we discuss experimental results (Section 7-8) and conclude (Section 9).
2 Paraphrasability of nominalizations
Our goal is to paraphrase a nominalization into a "clausal paraphrase", which we define as a clause headed by a verb whose syntactic arguments (e.g., subject, object, and prepositional object) are transformed from the nominal arguments in the input nominalization. We focus on compound nominalizations in which the head noun has a prenominal modifier and a prepositional object, following the syntactic form targeted by the only publicly available dataset for nominalization paraphrasing (Lee et al., 2021). Some example inputs and outputs are shown in Table 1.
A clausal paraphrase would not be possible for a compound nominalization if there is no suitable verb equivalent for its head noun. Large-scale language resources (Meyers et al., 1998) already exist to help determine whether such a verb exists, and the task has been tackled in the context of QA semantic role labeling for nominalization (Klein et al., 2020). Less attention has been paid to another factor, namely, whether the prenominal modifier can be expressed as a subject, object or prepositional object in the paraphrase. We are not aware of previous data-driven research on this task, which is the focus of this paper. We will not consider the prepositional object in the input nominalization, since it can be incorporated into a clausal paraphrase in most cases.
2.2 Task definition
The term "paraphrasability" has been used for the degree of semantic equivalence between syntactic variants of predicate phrases (Fujita and Sato, 2008). We will use this term to refer to the three categories of paraphrasing behavior of prenominal modifiers in compound nominalizations:
Noun The prenominal modifier is a noun, or is an adjective that pertains to a noun, that can serve as the subject, object or prepositional object in a clausal paraphrase. In other words, either the
prenominal modifier itself (e.g., “police”) or its pertainym (“president” for “presidential”) literally refers to the entity that participates in the event denoted by the verbal noun (“police response”, “presidential nomination”).
Adverb The prenominal modifier is an adjective that can appear in the clausal paraphrase in its adverbial form (e.g., "frontal opposition"  $\rightarrow$  "oppose frontally"), but not as pertainym ("frontal opposition"  $\nrightarrow$  "the front opposes").
Nil The prenominal modifier cannot be paraphrased with either method above (e.g., "stellar performance"  $\nrightarrow$  "a star performs"; "brain drain"  $\nrightarrow$  "drain a brain").
As shown in Table 1, the input is a nominalization that consists of a deverbal noun (derived from the verb  $V$ ); its prenominal modifier (bolded); and a prepositional phrase. The output of the Paraphrasability Prediction task is the part-of-speech label of the word to which the prenominal modifier is paraphrased (bolded). The label can be Noun, Adverb, or Nil when it is not paraphrasable.
The output of the Paraphrase Generation task is a clausal paraphrase of the input. It incorporates the verb  $V$ , the prepositional object from the input (marked with  $O$ ); and either a noun (marked with  $M$ ) or an adverb (marked with  $B$ ) corresponding to the prenominal modifier. The gold paraphrase of the N1 type input is defined as null. The only way to render such an input as a clause is with a support verb or light verb (e.g., "stellar performance") → "give a stellar performance"). Since the paraphrase retains the original nominalization, it does not serve our goal of unpacking its meaning.
3.1 Noun literality prediction
There has been extensive research on compositionality analysis on noun compounds (Reddy et al., 2011), adjective-noun combinations and other types of multiword expressions (MWEs) (Biemann and Giesbrecht, 2011; Ramisch et al., 2016; Cordeiro et al., 2019; Jana et al., 2019). Compositionality refers to the extent to which the meaning of the MWE can be expressed in terms of the meaning of its constituents. It therefore has considerable overlap with literality prediction, which would identify, for example, the noun "rat" in "rat race" as non-literal (Reddy et al., 2011).
Input Nominalization	Paraphrase Generation	Paraphrasability Prediction
American influence on global culture police response to the rioting	\( \mathbf{A}\mathbf{m}\mathbf{r}\mathbf{e}\mathbf{a}\mathbf{r}\mathbf{s}^{M}\text{ influences}^{V}\text{ global }\text{ culture}^{O} \) the \( \mathbf{p}\mathbf{o}\mathbf{l}\mathbf{e}\mathbf{m}^{M}\text{ responds}^{V}\text{ to the }\text{rioting}^{O} \)	Noun
climatic effects of air pollutants war preparations of the government	air pollutants \( {}^{O} \) affect \( {}^{V} \) the \( \mathbf{c}\mathbf{l}\mathbf{m}\mathbf{t}\mathbf{e}^{M} \) the \( \mathbf{g}\mathbf{o}\mathbf{n}\mathbf{m}\mathbf{t}\mathbf{e}^{O}\text{ prepares}^{V}\text{ for }\mathbf{w}\mathbf{a}\mathbf{r}^{M} \)
bodily injury to a friend student admissions into universities	\( injure^{V} \) the \( \mathbf{b}\mathbf{o}\mathbf{y}\mathbf{M} \) of a \( \mathbf{f}\mathbf{r}\mathbf{e}\mathbf{o}^{O} \) \( \mathbf{a}\mathbf{d}\mathbf{i}\mathbf{m}\mathbf{t}\mathbf{e}\mathbf{s}\mathbf{M} \) into \( \mathbf{u}\mathbf{s}\mathbf{i}\mathbf{r}\mathbf{s}\mathbf{E}^{O} \)
presidential nomination of Harrison business travel to Greece	\( \mathbf{n}\mathbf{o}\mathbf{m}\mathbf{t}\mathbf{e}\mathbf{V}\mathbf{H}\mathbf{r}\mathbf{o}\mathbf{r}\mathbf{h}\mathbf{o}\mathbf{O}\text{ as } \mathbf{p}\mathbf{e}\mathbf{s}\mathbf{m}\mathbf{t}\mathbf{E}^{M} \) \( \mathbf{t}\mathbf{r}\mathbf{e}\mathbf{V} \) to \( \mathbf{G}\mathbf{e}\mathbf{c}\mathbf{c}\mathbf{e}\mathbf{O} \) for \( \mathbf{b}\mathbf{u}\mathbf{s}\mathbf{e}\mathbf{S}^{M} \)
naval assistance from Italian powers majority decision of the panel	\( \mathbf{n}\mathbf{a}\mathbf{v}\mathbf{e}\mathbf{s}^{M} \) of \( \mathbf{I}\mathbf{t}\mathbf{a}\mathbf{L}\mathbf{I}\mathbf{a}\mathbf{p}\mathbf{o}\mathbf{s}^{O} \) assist \( {}^{V} \) a \( \mathbf{m}\mathbf{a}\mathbf{o}\mathbf{r}\mathbf{y}^{M} \) of the \( \mathbf{p}\mathbf{a}\mathbf{l}\mathbf{O} \) \( \mathbf{d}\mathbf{e}\mathbf{s}\mathbf{V} \)
frontal opposition of the employers dynamic allocation of licenses	The \( \mathbf{e}\mathbf{o}\mathbf{m}\mathbf{y}\mathbf{s}\mathbf{P}\mathbf{o}\mathbf{o}\mathbf{P}\mathbf{a}\mathbf{s}\mathbf{o}\mathbf{O}\text{ assist }^{V} \) a \( \mathbf{m}\mathbf{a}\mathbf{o}\mathbf{r}\mathbf{y}^{M} \) of the \( \mathbf{p}\mathbf{a}\mathbf{l}\mathbf{O} \) \( \mathbf{d}\mathbf{e}\mathbf{s}\mathbf{V} \)	Adverb
stellar performance of the rookies brain drain of scientists	null null	Nil
Table 1: Input and output of the Paraphrasability Prediction and Paraphrase Generation task (Section 2.2)
Our task is closely related to literality prediction since compound nominalizations are a subset of noun-noun compounds; in particular, a prenominal modifier that is "literal" would likely be of paraphrasability type Noun (Section 2.2). We will therefore evaluate the performance of a state-of-the-art noun literality prediction model (Shwartz and Dagan, 2019) in our experiment.
Our task is nonetheless distinct from literality prediction since it focuses on paraphrasability rather than literalness. For example, even when a prenominal modifier is used metaphorically and is non-literal (e.g. "circular argument"), it would be labeled Noun in terms of paraphrasability if it can appear in a clausal paraphrase ("argue in a circle").
3.2 Noun compound interpretation
A noun compound can be disambiguated with a free-form paraphrase (Hendrickx et al., 2013), or with verbs and prepositions linking the two nouns, e.g., "apple pie" is a "pie with apples" (Butnariu et al., 2010; Nakov and Hearst, 2013). Unsupervised approaches have been found to be effective for noun compound interpretation. Paraphrase templates with slots for prepositions and predicates, for example, can be filled using pre-trained masked language models (Ponkiya et al., 2020). We will likewise investigate unsupervised approaches in this work. Even though compound nominalizations are a subset of noun-noun compounds, our task is different since paraphrases in noun compound interpretation do not transform the head noun into
a verb.
3.3 Paraphrasing nominalizations
Research on nominalization interpretation has mostly focused on nominal semantic role labeling, which assigns abstract labels (e.g., agent, patient, manner) to arguments of nominalizations (Lapata, 2002; Nicholson and Baldwin, 2008; Padó et al., 2008; Kilicoglu et al., 2010). Given the systematic correspondences between nominalization and clause structure, there have also been efforts to paraphrase nominalizations as clauses. Algorithms have been proposed for automatic acquisition of paraphrase templates, which can cover nominalization inputs (Shinyama et al., 2002). The paraphrasing task has also been indirectly addressed in a model for question and answer generation from nominalizations (Klein et al., 2020).
The most closely related work to this paper was reported in Lee et al. (2021). Their proposed model first overgenerates paraphrase candidates, and then uses textual entailment to identify the optimal candidate. However, since all nominalizations in their dataset have paraphrases, their algorithm makes no judgment on paraphrasability. We extend their dataset and investigate paraphrasability prediction to fill in this research gap.
The only publicly available dataset of clausal paraphrases, developed by Lee et al. (2021), provides 450 paraphrases for English nominalizations. All
Paraphrasability Label	Adjectival modifier	Nominal modifier
Noun	281	169
Adverb	55	n/a
Nil	91	38
Table 2: Breakdown of our dataset according to paraphrasability label and part-of-speech of the prenominal modifier
instances are of the paraphrasability label Noun (Section 2.2). To facilitate our study, we enlarged this dataset with inputs of the Adverb and Nil paraphrasability labels.
4.1 Data source
In the interest of consistency with the existing dataset, we focus on nominalizations with the same syntactic pattern. Specifically, we collected sentences from English Wikipedia that contain a noun phrase headed by a verbal noun with one prenominal modifier and one postnominal modifier.
As shown in Table 1, the postnominal modifier is a prepositional phrase with prepositional object  $O$ . The prenominal modifier can be a noun or an adjective. To create a challenging dataset, the adjective must have a pertainym, or can itself also serve as noun (e.g., "light"), such that multiple paraphrasability labels are plausible.
Two annotators, a native speaker and a near-native speaker of English, independently classified the nominalizations into one of three paraphrasability labels (Section 2.2). For those labeled as Adverb, the annotator composed a paraphrase with an adverb that is derivationally related to the adjective. For those labeled as Ni1, no paraphrase was required. Examples for each label are provided in Table 1.
A professor of linguistics who is a native speaker of English reviewed the annotation, either keeping both or selecting one of them. A total of 184 non-Noun instances were collected and added to the dataset, resulting in an expanded dataset with 634 paraphrases (Table 2).<sup>1</sup>
5 Paraphrase candidates
Our approach is to first overgenerate paraphrase candidates for each input, and then identify the op-
timal candidate. This section presents the candidate types, and the next section describes the candidate selection algorithm.
Table 3 shows the paraphrase candidates for the input "frontal opposition of the employers". The prenominal modifier, "frontal", is transformed into various parts-of-speech and placed at different positions in the candidates. Each candidate is associated with one of the three paraphrasability types:
5.1 Noun
The Noun type candidates are the five paraphrases defined in Lee et al. (2021). The prenominal modifier is paraphrased as a noun in the subject (MVO, MOV), object (OVM, VMO), and oblique (VOM) positions.
5.2 Adverb
There are four paraphrase candidates for the Adverb type. The prenominal modifier must be an adjective. It is paraphrased as an adverb that pertains to itself, according to WordNet (Fellbaum, 2010); or an adverb that is derivationally related to itself, according to CatVar (Habash and Dorr, 2003). The adverb (B) is placed either before the verb (BVO, OBV) or at the end of the clause (VOB, OVB).
5.3 Nil
There are, by definition, no obvious paraphrase candidates to represent inputs of the Nil type. We implemented the following alternatives:
Identity The nominalization input itself.
Light verb The paraphrase retains the nominalization as the object of a light verb or support verb (Grefenstette and Teufel, 1995). One paraphrase candidate prepends the light verb; e.g., "home run against Arizona"  $\rightarrow$  "hit a home run against Arizona". The other candidate uses the prepositional object  $(O)$  as subject; e.g., "stellar performance of the rookies"  $\rightarrow$  "the rookies give a stellar performance".
Predicative adjective Limited to prenominal modifiers that are adjectives, this paraphrase uses the adjective predicatively to form a clause. The paraphrase is designed, on the one hand, to be acceptable for Nil type inputs, e.g., "stellar performance of the rookies"  $\rightarrow$  "the performance of the rookies is stellar"; and on the other hand, to be unacceptable for
<sup>1</sup>Accessible at https://github.com/NominalizationParaphrase
Paraphrasability Label	Paraphrase Candidate	Word Order
Noun	the front opposes the employers
the employers oppose the front
oppose the front of the employers
oppose the employers at the front
front employers oppose	MVO
OVM
VMO
VOM
MOV
Adverb	oppose the employers frontally
frontally oppose the employers
the employers oppose frontally
the employers frontally oppose	VOB
BVO
OVB
OBV
Nil	frontal opposition of the employers
Opposition of the employers is frontal
have frontal opposition of the employers
the employers face frontal opposition	Identity
Predicative Adjective
Light Verb
Light Verb
Table 3: Automatically generated paraphrase candidates (Section 5) for the input "frontal opposition of the employers", which place the prenominal modifier (M or B), verb (V) and prepositional object (O) in the input nominalization in different word orders
nominal adjectives, which cannot be used as a predicate (Coates, 1971), e.g., "presidential nomination of Harrison"  $\nrightarrow$  "the nomination of Harrison is presidential".
We used a masked language model, BERT (Devlin et al., 2019), to generate the most likely determiners, prepositions and light verbs for the paraphrase candidates above.
6 Approach
As discussed in Section 4.1, the input is a sentence that contains a nominalization, headed by a de-verbal noun that has a prenominal modifier and a prepositional phrase (Table 1). The prenominal modifier may be a noun or an adjective.
We first overgenerate paraphrases to construct a candidate pool (Section 6.1), and then filter the pool by considering paraphrasability (Section 6.2). For the Paraphrase Generation task, the output is the best candidate selected by the textual entailment and language models (Section 6.3). For the Parapharasability Prediction task, the output is the paraphrasability label associated with the selected candidate (Table 3).
6.1 Candidate pool construction
This step constructs a pool of paraphrase candidates. We evaluated the following methods:
All Include all paraphrase candidates in Table 3.
Gold Include only those paraphrase candidates associated with the gold paraphrasability label.
Majority baseline Include only those paraphrase candidates associated with the majority paraphrasability label, which is Noun in our dataset. This baseline replicates the algorithm proposed by Lee et al. (2021), which considers only the MVO, OVM, VMO, VOM, and MOV paraphrases in Table 3.
Word frequency baseline Include the Noun type (Adverb type) paraphrase candidates only when the noun (the adverb) corresponding to the prenominal modifier has high frequency. The frequency threshold is optimized on our dataset based on frequency statistics in the Google Web 1T N-gram Corpus (Brants and Franz, 2006).
6.2 Candidate pool filtering
This step filters the candidate pool constructed by the All model. We evaluated two methods that consider paraphrasability through semantic parsing and literality prediction, respectively.
6.2.1 Filtering with AMR
Abstract Meaning Representation (AMR) abstracts away from the syntactic realization of a sentence and expresses its meaning with a directed acyclic graph, where nodes represent events and concepts, and edges represent relationships between
Figure 1: AMR graph of the sentence "... a court for the constitutional interpretation of law", which is predicted as Noun by the All+AMR model (Section 6.2.1)
the nodes (Banarescu et al., 2013). In an AMR graph, deverbual nouns are annotated as verbs, and adjectives pertaining to nouns are annotated in their nominal form whenever possible.
We use PERIN (Samuel and Straka, 2020) to construct an AMR graph for the input sentence, and align the nodes with the words in the input. We will refer to the node aligned to the deverbal noun as the "deverbal noun node"; and the node aligned to the prenominal modifier as the "prenominal modifier node". In Figure 1, the "constitution" node is the prenominal modifier node (aligned to "constitutional" in the input); and the "interpret-01" node is the deverbal noun node (aligned to "interpretation").
The All+AMR model predicts Noun as paraphrasability label and removes all non-Noun type candidates from the pool if the prenominal modifier node:
- is an argument of the deverbal noun node; or
- is the domain of the deverbal noun node, and is annotated as a noun.
Otherwise, it predicts paraphrasability to be non-Noun and removes all Noun type candidates from the pool.
For example, the model predicts Noun as paraphrasability label for the sentence in Figure 1, since the prenominal modifier node ("constitution") serves as arg0 to the deverbal noun node ("interpret-01"). The model rejects Noun as paraphrasability
Figure 2: AMR graph of the sentence "... the secular celebration of Christmas", which is predicted as non-Noun by the All+AMR model (Section 6.2.1)
label for the sentence in Figure 2. Even though the prenominal modifier node ("secular") is the domain of the deverbal noun node ("celebrate-02"), it is annotated with the original adjective rather than its nominal form.
6.2.2 Filtering with noun literality prediction
Noun literality prediction is closely related to paraphrasability prediction (Section 3.1). We use Lex-Comp (Shwartz and Dagan, 2019), a state-of-the-art model in noun literality prediction, which is trained on datasets from Reddy et al. (2011) and Tratz et al. (2010) using contextualized word embeddings. $^{2}$
Given the prenominal modifier and the deverbalm noun in the input, the All+LexComp model predicts Noun as paraphrasability label if LexComp predicts "literal", and removes all non-Noun type candidates. Otherwise, it predicts Nil and keeps only the Nil type paraphrases. This model does not perform filtering on an input with an adjectival modifier.
6.3 Candidate selection
A textual entailment model (TE), enhanced with re-ranking by language model scores, was found to yield the strongest performance in paraphrase generation for compound nominalizations (Lee et al., 2021). Taking the nominalization input as the
$^{2}$ https://github.com/vered1986/lexcomp
premise and a paraphrase candidate as the hypothesis, the TE model predicts whether the facts in the former imply those in the latter. Among the three candidate paraphrases that yield the highest TE scores, the candidate with the highest language model (LM) score is selected.