SlowGuess/NAACL_2022_OCR huggingface dataset:
We replicate this algorithm and apply it on the filtered candidate pool. We use the AllenNLP textual entailment model<sup>3</sup>, and the log-probability score based on GPT-2 (117M) as the LM score (Salazar et al., 2020).<sup>4</sup>
7 Experimental set-up and metrics
The entire dataset was used for evaluation since our approach is unsupervised. We used SpaCy (Honnibal and Johnson, 2015) for POS tagging to determine the POS of the prenominal modifier.
Paraphrasability prediction. We report precision, recall and  $\mathbf{F}_1$ . Precision is defined as the number of actual Noun instances, out of all instances predicted by the system as Noun. Recall is defined as the proportion of gold Noun instances that are correctly identified by the system as Noun.
Paraphrase generation. We report "paraphrase accuracy" and "word order accuracy" as defined in Lee et al. (2021). For the former, the determiners are removed from all paraphrases. The system is considered correct if the lemmatized form of all words in the predicted paraphrase are identical with those in the gold paraphrase. The latter is defined likewise, except that prepositions are not taken into consideration. It thus essentially measures the system's ability to predict the verb and arguments and to put them into the correct word order. The word orders VOB/BVO and OVB/OBV are considered interchangeable.
Table 4 shows system performance on the paraphrasability prediction (Section 8.1) and its effect on paraphrase generation (Section 8.2).
8.1 Paraphrasability prediction
Given the preponderance of the Noun label in our dataset, the Majority baseline produced a strong performance at 0.673 precision and perfect recall. It outperforms the Word Frequency baseline, which has slightly higher precision (0.686) but lower recall (0.911), both in terms of  $\mathrm{F}_1$  and accuracy.
Using all paraphrase candidates resulted in an improvement in binary classification accuracy (0.711 vs. 0.673) over the Majority baseline, demonstrating the effectiveness of the Adverb and Nil paraphrases (Section 5.2-5.3). In terms of three-way classification, however, it offered no improvement over the Majority baseline (0.671 vs. 0.673). This indicates that while the candidate selection method (Section 6.3) can correctly detect some Noun type candidates as inappropriate, it is less competent in judging between Adverb vs. Nil paraphrases.
The All+LexComp model raised the accuracy by only  $0.4\%$  in comparison to the All model. This result suggests that noun literality prediction is only slightly helpful as a proxy for paraphrasability.
The All+AMR model achieved the highest  $\mathrm{F_1}$  (0.852) by raising both the precision and recall of the All model. The improvement is statistically significant in terms of both binary classification  $(0.782)^5$  and three-way classification  $(0.744)^6$ . These results show that AMR is useful for predicting paraphrasability, which may be due to the more fine-grained semantic information in the AMR graphs that could not be inferred by the LM and TE models in the candidate selection step. The improvement of the All+AMR model over the All+LexComp model is also significant<sup>7</sup>, likely because the semantic features in the AMR graphs are more relevant to paraphrasability than literality.
Table 5 shows the paraphrasability labels predicted by the All+AMR model. While it was able to identify most of the Noun inputs, it did so for only half of the Adverb ones. The most challenging turned out to be the Nil inputs, which the model succeeded in detecting less than one-third of the time.
8.2 Paraphrase generation
Despite its higher accuracy in paraphrasability prediction, the Majority baseline (0.264 paraphrase accuracy) performed worse than the Word Frequency baseline (0.275) in paraphrase generation. This likely reflects the greater challenge in identifying the correct Noun type paraphrases than the Advice and Nil types.
For the other models, performance in para
<sup>3</sup>https://demo.allennlp.org/textual-entailment/roberta-snli
$^{4}$ https://github.com/awslabs/mlm-scoring
${}^{5}p = {0.000556}$  according to McNemar’s Test with continuity correction; the same test is used henceforth
$^6$ At  $p = 0.000172$
${}^{7}p = {0.00169}$  on binary classification and  $p = {0.000667}$  on three-way classification
Metric →
↓ Model	Paraphrasability Prediction	Paraphrase Generation
Noun vs. non-Noun	3-way	Word order Accuracy	Paraphrase Accuracy
P	R	F1	Accuracy	Accuracy
Word Frequency	0.686	0.911	0.783	0.660	0.645	0.384	0.275
Majority	0.673	1.000	0.804	0.673	0.673	0.376	0.264
All	0.743	0.873	0.802	0.711	0.671	0.416	0.318
All+LexComp	0.752	0.859	0.802	0.715	0.675	0.424	0.333
All+AMR	0.784	0.932	0.852	0.782	0.744	0.462	0.347
Gold	1	1	1	1	1	0.704	0.591
Table 4: Performance on paraphrasability prediction and paraphrase generation
Predicted →
↓ Gold	Noun	Adverb	Nil
Noun	345	13	12
Adverb	17	24	8
Nil	78	13	40
Table 5: Contingency table for the paraphrasability prediction of the All+AMR model
phrasability prediction is largely correlated to paraphrase generation. The All model improved over the Majority baseline both in terms of word order (0.416) and paraphrase accuracy (0.318). Consider the input "... the apocalyptic destruction of the town and the cult". The All model correctly declined to paraphrase (null output) on the basis of the high score secured by the light-verb paraphrase "the town and the cult suffer an apocalyptic destruction". In contrast, the Majority baseline produced the inappropriate paraphrase "the apocalypses destroy the town and the cult".
The All+AMR model again offered the best performance, at 0.462 word order accuracy and 0.347 paraphrase accuracy. For the sentence in Figure 1, the All model generated the predicative adjective paraphrase "interpretation of law is constitutional" due to the high LM score, even though the word "constitutional" yields a different meaning in this paraphrase. The All+AMR model was able to reject this paraphrase since the word "constitution" was inferred to play the subject role. Conversely, in Figure 2, the model was able to reject paraphrases involving the noun "secularism", since the AMR parser annotated with the original adjective "secu
lar". The considerable performance gap from the Gold model (paraphrase accuracy 0.591), however, indicates there is still much room for improvement in interpreting nominalizations.
A clausal paraphrase can help disambiguate a nominalization semantically, especially when the prenominal modifier is difficult to interpret. This paper has presented the first study on determining the paraphrasability of the prenominal modifier in a compound nominalization. We have expanded an existing dataset to cover cases when the prenominal modifier can appear as a noun in the paraphrase, as an adverb, or not at all.
Our experiments suggest that overgeneration of paraphrase candidates, followed by ranking with a textual entailment model and language model, can yield competitive results. Further, AMR-based features lead to statistically significant improvement in performance.
A limitation of our study is the restricted syntactic form of the input nominalizations. To facilitate more comprehensive evaluation, future research should consider expanding the dataset further to cover a wider range of nominalizations, and richer variations in their clausal paraphrases.
This project was supported by an HKSAR UGC Teaching Learning Grant (Meeting the Challenge of Teaching and Learning Language in the University: Enhancing Linguistic Competence and Performance in English and Chinese, 2016-19 Triennium).
The improvement in word order accuracy is statistically significant over the All and All+LexComp models at  $p = 0.0211$  and  $p = 0.0482$ , respectively. The improvement in paraphrase accuracy is not significant, however, at  $p = 0.0970$  and  $p = 0.428$  against the All and All+LexComp models, respectively.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proc. 7th Linguistic Annotation Workshop and Interoperability with Discourse.
Chris Biemann and Eugenie Giesbrecht. 2011. Distributional Semantics and Compositionality 2011: Shared Task Description and Results. In Proc. Workshop on Distributional Semantics and Compositionality (DiSCo).
Thorsten Brants and Alex Franz. 2006. The Google Web 1T 5-gram Corpus Version 1.1. In LDC2006T13.
Cristina Butnariu, Su Nam Kim, Preslav Nakov, Diarmuid O Seaghdha, Stan Szpakowicz, and Tony Veale. 2010. SemEval-2010 Task 9: The interpretation of noun compounds using paraphrasing verbs and prepositions. In Proc. NAACL HLT Workshop on Semantic Evaluations: Recent Achievements and Future Directions, page 100-105.
Jennifer Coates. 1971. Denominal Adjectives: A Study in Syntactic Relationships between Modifier and Head. Linguua, 27:160-169.
Silvio Cordeiro, Aline Villavicencio, Marco Idiart, and Carlos Ramisch. 2019. Unsupervised compositionality prediction of nominal compounds. Computational Linguistics, 45(1):1-57.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pretraining of Deep Bidirectional Transformers for Language Understanding. In Proc. North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL-HLT).
Christiane Fellbaum. 2010. Wordnet. In Theory and applications of ontology: computer applications, pages 231-243, Springer, Dordrecht.
Atsushi Fujita and Satoshi Sato. 2008. A Probabilistic Model for Measuring Grammaticality and Similarity of Automatically Generated Paraphrases of Predicate Phrases. In Proc. 22nd International Conference on Computational Linguistics (COLING).
Mark A. Greenwood. 2004. Using Pertainyms to Improve Passage Retrieval for Questions Requesting Information About a Location. In Proc. SIGIR Workshop on Information Retrieval for Question Answering (IR4QA), page 17-22, Sheffield, UK.
Gregory Grefenstette and Simone Teufel. 1995. Corpus-based Method for Automatic Identification of Support Verbs for Nominalizations. In Proc. 7th Conference on European chapter of the Association for Computational Linguistics (EACL).
Nizar Habash and Bonnie Dorr. 2003. A Categorical Variation Database for English. In Proc. North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL).
Iris Hendrickx, Preslav Nakov, Stan Szpakowicz, Zornitsa Kozareva, Diarmuid O Seaghdha, and Tony Veale. 2013. SemEval-2013 task 4: Free paraphrases of noun compounds. In Proc. 7th International Workshop on Semantic Evaluation (SemEval), page 138-143.
Matthew Honnibal and Mark Johnson. 2015. An Improved Non-monotonic Transition System for Dependency Parsing. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP).
Abhik Jana, Dima Puzyrev, Alexander Panchenko, Pawan Goyal, Chris Biemann, and Animesh Mukherjee. 2019. On the Compositionality Prediction of Noun Phrases using Poincaré Embeddings. In Proc. 57th Annual Meeting of the Association for Computational Linguistics (ACL).
Halil Kilicoglu, Marcelo Fiszman, Graciela Roseblat, Sean Marimpietri, and Thomas C. Rindflesch. 2010. Arguments of Nominals in Semantic Interpretation of Biomedical Text. In Proc. Workshop on Biomedical Natural Language Processing.
Ayal Klein, Jonathan Mamou, Valentina Pyatkin, Daniela Stepanov, Hangfeng He, Dan Roth, Luke Zettlemoyer, and Ido Dagan. 2020. QANom: Question-Answer driven SRL for Nominalizations. In Proc. 28th International Conference on Computational Linguistics (COLING).
Maria Lapata. 2002. The Disambiguation of Nominalizations. Computational Linguistics, 28(3):357-388.
John Lee, Ho Hung Lim, and Carol Webster. 2021. Paraphrasing Compound Nominalizations. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP).
Adam Meyers, Catherine Macleod, Roman Yangarber, Ralph Grishman, Leslie Barrett, and Ruth Reeves. 1998. Using NOMLEX to Produce Nominalization Patterns for Information Extraction. In Proc. Computational Treatment of Nominals.
Preslav I. Nakov and Marti A. Hearst. 2013. Semantic Interpretation of Noun Compounds Using Verbal and Other Paraphrases. ACM Transactions on Speech and Language Processing, 10(3).
Jeremy Nicholson and Timothy Baldwin. 2008. Interpreting Compound Nominalisations. In Proc. LREC Workshop Towards a Shared Task for Multiword Expressions (MWE 2008).
Sebastian Padó, Marco Pennacchiotti, and Caroline Sporleder. 2008. Semantic role assignment for event nominalisations by leveraging verbal data. In Proc. 22nd International Conference on Computational Linguistics (COLING).
Girishkumar Ponkiya, Rudra Murthy, Pushpak Bhattacharyya, and Girish Palshikar. 2020. Looking inside noun compounds: Unsupervised prepositional and free paraphrasing. In Findings of the Association for Computational Linguistics: EMNLP 2020, page 4313-4323.
Randolph Quirk, Sidney Greenbaum, Geoffrey Leech, and Jan Svartvik. 1985. A Comprehensive Grammar of the English Language. Longman.
Carlos Ramisch, Silvio Cordeiro, Leonardo Zilio, Marco Idiart, Aline Villavicencio, and Rodrigo Wilkens. 2016. How naked is the naked truth? a multilingual lexicon of nominal compound compositionality. In Proc. Annual Meeting of the Association for Computational Linguistics (ACL).
Siva Reddy, Diana McCarthy, and Suresh Manandhar. 2011. An empirical study on compositionality in compound nouns. In Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 210-218.
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked Language Model Scoring. In Proc. Annual Meeting of the Association for Computational Linguistics (ACL).
David Samuel and Milan Straka. 2020. UFAL at MRP 2020: Permutation-invariant Semantic Parsing in PERIN. In Proc. CoNLL 2020 Shared Task: Cross-Framework Meaning Representation Parsing, page 53-64.
Yusuke Shinyama, Satoshi Sekine, and Kiyoshi Sudo. 2002. Automatic Paraphrase Acquisition from News Articles. In Proc. Second international conference on Human Language Technology Research (HLT).
Vered Shwartz and Ido Dagan. 2019. Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition. Transactions of the Association for Computational Linguistics, 7:403-419.
Stephen Tratz and Eduard Hovy. 2010. A Taxonomy, Dataset, and Classifier for Automatic Noun Compound Interpretation. In Proc. 48th Annual Meeting of the Association for Computational Linguistics (ACL), page 678-687.