Asymmetries in Extraction From Nominal Copular Sentences: a Challenging Case Study for NLP Tools
Authors: Paolo Lorusso, Matteo Greco, Cristiano Chesi, Andrea Moro (IUSS, 2019)
Topic: Syntax and Dependency Treebanks

Abstract
In this paper we discuss two types of nominal copular sentences AUTHOR and we demonstrate how the peculiarities of these two configurations are hardly considered by standard NLP tools that are currently publicly available. Here we show that example-based MT tools (e.g. Google Translate) as well as other NLP tools (UDpipe, LinguA, Stanford Parser, and Google Cloud AI API) fail in capturing the critical distinctions between the two structures in the end producing both wrong analyses and, possibly as a consequence of a non-coherent (or missing) structural analysis, incorrect translations in the case of MT tools. To support the proposed analysis, we present also an empirical study showing that native speakers are indeed sensitive to the critical distinctions. This poses a sharp challenge for NLP tools that aim at being cognitively plausible or at least descriptively adequate AUTHOR.

Introduction
The main hypothesis of this paper is that sentence comprehension cannot be achieved independently from a coherent structural analysis. To support this claim, we first present a precise structural analysis that is critical for recovering the relevant dependencies within specific constructions, then we will show that the crucial structural properties captured by the theoretical framework are in fact correctly perceived by native speakers, but not revealed by some widely used Natural Language Processing (NLP) tools. This leads to poor performance in tasks like Machine Translation (MT). This argument seems to us especially relevant in those structural configurations in which a nonlocal dependency must be established: in parsing, for instance, interpreting correctly a whdependency requires that the dependent (the whphrase) and the dependee (the head selecting the whphrase as its argument/modifier) are identified, and the nature of the dependence disambiguated (e.g. argument vs. modifier). In (1) we exemplify the special case of a non-local dependency between a whPP and a DP it depends on (a co-indexed underscore signals the possible extraction sites, hence the dependent constituent; the diacritic “*” prefixes, as usual, illegal sites): (1) [Di quale segnale]i [i telescopi *_ i] hanno Of which signal the telescopes have scoperto *_i[un’interferenza _ i]? discoveredan interference? ‘[which signal]i did the telescopes discover an interference of _ i?’ The second DP un’interferenza (an interference) (the internal argument) is the dependee of the whphrase and neither the subject DP nor the predicate can host this whdependency instead. According to Google Translate (as of 12th July 2019), this second option seems indeed a viable one: (2) What signal did the telescopes find an interference? The translation is ill formed being the internal argument of find filled both by the whphrase and the DP an interference (which cannot take a whDP as its own argument due to the absence of a relevant preposition). In this work we focus on a similar non-local dependency involving two kinds of copular sentences: Inverse (3.a) and Canonical (3.b). Using these constructions, we will test the availability of whPP sub-extraction from both the first and the second DP as exemplified in (4). (3) a. le foto del muro sono la causa della rivolta the pictures of the wall are the cause of the riot b. la causa della rivolta sono le foto del muro the cause of the riot are the pictures of-the wall ‘the cause of the riot is the pictures of the wall’ (4) a. [Di quale rivolta]i le foto del muro sono of which riot the pictures of_the wall are la causa _ i ? the cause b. [Di quale muro]i le foto _ i sono of which riot the pictures of the wall are la causa della rivolta? the cause of_the riot In the first part of this paper (§2), we will briefly present an analysis for these constructions, then we will demonstrate that native speakers are selectively sensitive both to the copular structural configuration (Canonical vs. Inverse) and to the extraction site (subject vs. predicate) (§3). In §4 we will test the insensibility of some freely available NLP tools (Google Translate, the Natural Language service of Google Cloud AI API, UDpipe, Stanford Parser and LinguA) to the syntactic oppositions previously discussed.

Conclusion
In this paper we demonstrated that nominal copular sentences constitute a clear challenge for the computational analysis since the same string of elements [DP V DP] can have in principle two different syntactic representations (hence two different meanings), depending on which kind of copular sentence is realized (Canonical or Inverse). In this paper, we spotted various glitches in the automatic analyses which in the end led either to significant failures (Google Translate) or to rough structural hypotheses that bluntly ignore the relevant contrasts here discussed. Our empirical study, testing both online and offline the whPP sub-extraction possibilities from both subject and predicate DPs, shows that native speakers are sensitive with respect to the different structural roles; in addition, they perceive as expected the underlying structural representation of Canonical vs. Inverse copular construction. None of the NLP tools we tested succeeded in providing a full set of coherent analyses, with the exception of the Stanford Parser for English that at least succeeded in analyzing correctly the canonical copular sentences. This analysis was however insufficient in the case of inverse constructions and in case of sub-extraction, confirming that non-local dependencies are critical configurations native speakers are able to parse but machine do not, yet.