Venue: NAACL 2019

# Content Differences in Syntactic and Semantic Representations

Daniel Hershcovich\({}^{1,2}\)  Omri Abend\({}^{2}\)  Ari Rappoport\({}^{2}\)

\({}^{1}\)The Edmond and Lily Safra Center for Brain Sciences

\({}^{2}\)School of Computer Science and Engineering

Hebrew University of Jerusalem

{danielh,oabend,arir}@cs.huji.ac.il

###### Abstract

Syntactic analysis plays an important role in semantic parsing, but the nature of this role remains a topic of ongoing debate. The debate has been constrained by the scarcity of empirical comparative studies between syntactic and semantic schemes, which hinders the development of parsing methods informed by the details of target schemes and constructions. We target this gap, and take Universal Dependencies (UD) and UCCA as a test case. After abstracting away from differences of convention or formalism, we find that most content divergences can be ascribed to: (1) UCCA's distinction between a Scene and a non-Scene; (2) UCCA's distinction between primary relations, secondary ones and participants; (3) different treatment of multi-word expressions, and (4) different treatment of inter-clause linkage. We further discuss the long tail of cases where the two schemes take markedly different approaches. Finally, we show that the proposed comparison methodology can be used for fine-grained evaluation of UCCA parsing, highlighting both challenges and potential sources for improvement. The substantial differences between the schemes suggest that semantic parsers are likely to benefit downstream text understanding applications beyond their syntactic counterparts.

## 1 Introduction

Semantic representations hold promise due to their ability to transparently reflect distinctions relevant for text understanding applications. For example, syntactic representations are usually sensitive to distinctions based on POS (part of speech), such as between compounds and possessives. Semantic schemes are less likely to make this distinction since a possessive can often be paraphrased as a compound and vice versa (e.g., "US president"/president of the US"), but may distinguish different senses of possesses (e.g., "some of the presidents" and "inaquoration of the presidents").

Nevertheless, little empirical study has been done on what distinguishes semantic schemes from syntactic ones, which are still in many cases the backbone of text understanding systems. Such studies are essential for (1) determining whether and to what extent semantic methods should be adopted for text understanding applications; (2) defining better inductive biases for semantic parsers, and allowing better use of information encoded in syntax; (3) pointing at semantic distinctions unlikely to be resolved by syntax.

The importance of such an empirical study is emphasized by the ongoing discussion as to what role syntax should play in semantic parsing, if any [8, 9, 50]. See SS8.

This paper aims to address this gap, focusing on _content_ differences. As a test case, we compare relatively similar schemes (SS2): the syntactic Universal Dependencies (UD; Nivre et al., 2016), and the semantic Universal Conceptual Cognitive Annotation (UCCA; Abend and Rappoport, 2013).

We UCCA-annotate the entire web reviews section of the UD EWT corpus (SS3), and develop a converter to assimilate UD and UCCA, which use formally different graphs (SS4). We then align their nodes, and identify which UCCA categories match which UD relations, and which are unmatched.

Most content differences are due to (SS5):

1. UCCA's distinction between words and phrases that evoke Scenes (events) and ones that do not. For example, eventive and non-eventive nouns are treated differently in UCCA, but similarly in UD.
2. UCCA's distinction between primary relations, secondary relations and Participants, in contrast to UD's core/non-core distinction

3. Different treatment of multi-word expressions (MWEs), where UCCA has a stronger tendency to explicitly mark them.
4. UCCA's conflation of several syntactic realizations of inter-clause linkage, and disambiguation of other cases that UD treats similarly.

We show that the differences between the schemes are substantial, and suggest that UCCA parsing in particular and semantic parsing in general are likely to benefit downstream text understanding applications. For example, only 72.9% of UCCA Participants are UD syntactic arguments, i.e., many semantic participants cannot be recovered from UD.1 Our findings are relevant to other semantic representations, given their significant overlap in content Abend and Rappoport (2017).

Footnote 1: This excludes cases of shared argumenthood, which are partially covered by _enhanced UD_. See §4.1.

A methodology for comparing syntactic and semantic treebanks can also support fine-grained error analysis of semantic parsers, as illustrated by Szubert et al. (2018) for AMR Banarescu et al. (2013). To demonstrate the utility of our comparison methodology, we perform fine-grained error analysis on UCCA parsing, according to UD relations (SS6). Results highlight challenges for current parsing technology, and expose cases where UCCA parsers may benefit from modeling syntactic structure more directly.2

Footnote 2: Our conversion and analysis code is public available at [https://github.com/danielhers/synsem](https://github.com/danielhers/synsem).

## 2 Representations

The conceptual and formal similarity between UD and UCCA can be traced back to their shared design principles: both are designed to be applicable across languages and domains, to enable rapid annotation and to support text understanding applications. This section provides a brief introduction to each of the schemes, whereas the next sections discuss their content in further detail.3

Footnote 3: See Supplementary Material for a definition of each category in both schemes, and their abbreviations.

Uccais a semantic annotation scheme rooted in typological and cognitive linguistic theory. It aims to represent the main semantic phenomena in text, abstracting away from syntactic forms. Shown to be preserved remarkably well across translations Sulem et al. (2015), it has been applied to improve text simplification Sulem et al. (2018), and text-to-text generation evaluation Birch et al. (2016); Choshen and Abend (2018); Sulem et al. (2018).

Formally, UCCA structures are directed acyclic graphs (DAGs) whose nodes (or _units_) correspond either to words, or to elements viewed as a single entity according to some semantic or cognitive consideration. Edges are labeled, indicating the role of a child in the relation the parent represents. Figure 1 shows a legend of UCCA abbreviations. A _Scene_ is UCCA's notion of an event or a frame, and is a description of a movement, an action or a state which persists in time. Every Scene contains one primary relation, which can be either a Process or a State. Scenes may contain any number of Participants, a category which also includes abstract participants and locations. They may also contain temporal relations (Time), and secondary relations (Adverbials), which cover semantic distinctions such as manner, modality and aspect.4

Footnote 4: Despite the similar terminology, UCCA Adverbials are not necessarily adverbs syntactically.

Scenes may be _linked_ to one another in several ways. First, a Scene can provide information about some entity, in which case it is marked as an Elaborator. This often occurs in the case of particles or relative clauses. For example, "child who went to school" is an Elaborator Scene in "The child who went to school is John". A Scene may also be a Participant in another Scene. For example, "John went to school" in the sentence: "He said John went to school". In other cases, Scenes are annotated as Parallel Scenes (H), which are flat structures and may include a Linker (L), as in: "When\({}_{L}\) [he arrives]\({}_{H}\), [he will call them]\({}_{H}\)".

Non-Scene units are headed by units of the category Center, denoting the type of entity or thing described by the whole unit. Elements in non-Scene units include Quantifiers (such as "_dozens_ of people") and Connectors (mostly coordinating conjunctions). Other modifiers to the Center are marked as Elaborators.

UCCA distinguishes _primary_ edges, corresponding to explicit relations, from _remote_ edges,

\begin{table}
\begin{tabular}{l l|c c} Participant & A & Linker & L \\ Center & C & Connector & N \\ Advrebial & D & Process & P \\ Elaborator & E & Quantifier & Q \\ Function & F & Relator & R \\ Ground & G & State & S \\ Parallel Scene & H & Time & T \\  & & & \\ \end{tabular}
\end{table}
Table 1: Legend of UCCA categories (edge labels).

 

which allow for a unit to participate in several super-ordinate relations. See example in Figure 1. Primary edges form a tree, whereas remote edges (dashed) enable reentrancy, forming a DAG.

UDis a syntactic dependency scheme used in many languages, aiming for cross-linguistically consistent and coarse-grained treebank annotation. Formally, UD uses bi-lexical trees, with edge labels representing syntactic relations.

One aspect of UD similar to UCCA is its preference of lexical (rather than functional) heads. For example, in auxiliary verb constructions (e.g., "is eating"), UD marks the lexical verb (_eating_) as the head, while other dependency schemes may select the auxiliary _is_ instead. While the approaches are largely inter-translatable Schwartz et al. (2012), lexical head schemes are more similar in form to semantic schemes, such as UCCA and semantic dependencies Oepen et al. (2016).

Being a dependency representation, UD is structurally underspecified in an important way: it is not possible in UD to mark the distinction between an element modifying the head of the phrase and the same element modifying the whole phrase de Marneffe and Nivre (2019).

An example UD tree is given in Figure 2. UD relations will be written in typewriter font.

## 3 Shared Gold-standard Corpus

We annotate 723 English passages (3,813 sentences; 52,721 tokens), comprising the web reviews section of the English Web Treebank EWT; Bies et al. (2012). Text is annotated by two UCCA annotators according to v2.0 of the UCCA guidelines5 and cross-reviewed. As these sentences are included in the UD English_EWT treebank, this is a _shared_ gold-standard UCCA and UD annotated corpus.6 We use the standard train/development/test split, shown in Table 2.

Footnote 5: [http://bit.ly/ucca_guidelines_v2](http://bit.ly/ucca_guidelines_v2)

## 4 Comparison Methodology

To facilitate comparison between UCCA and UD, we first assimilate the graphs by abstracting away from formalism differences, obtaining a similar graph format for both schemes. We then match pairs of nodes in the converted UD and UCCA trees if they share all terminals in their yields.

UD annotates bi-lexical dependency trees, while UCCA graphs contain non-terminal nodes. In SS4.1, we outline the unified DAG converter by Hershcovich et al. (2018a,b),7 which we use to reach a common format. In SS4.2, we describe a number of extensions to the converter, which abstract away from further non-content differences.

Footnote 7: Our data is available at [https://github.com/UniversalConceptualCognitiveAnnnotation/UCCA_English-EWT](https://github.com/UniversalConceptualCognitiveAnnnotation/UCCA_English-EWT).

Footnote 7: [https://github.com/huji-nlp/semstr](https://github.com/huji-nlp/semstr)

### Basic Conversion

Figure 3 presents the same tree from Figure 2 after conversion. The converter adds one pre-terminal per token, and attaches them according to the original dependency tree: traversing it from the root, for each head it creates a non-terminal parent with the edge label _head_, and adds the dependents as children of the created non-terminal. Relation subtypes are stripped, leaving only universal relations. For example, the language-specific definite article label det:def is replaced by the universal det.

\begin{table}
\begin{tabular}{l|c c c}  & **Train** & **Dev** & **Test** \\ \hline \# Passages & 347 & 192 & 184 \\ \# Sentences & 2,723 & 554 & 535 \\ \# Tokens & 44,804 & 5,394 & 5,381 \\ \end{tabular}
\end{table}
Table 2: Data split for the shared gold-standard corpus.

Figure 1: UCCA graph. Dashed: remote edge.

Figure 3: Converted UD tree. Non-terminals and _head_ edges are introduced by the unified DAG converter.

 

Reentrancies.Remote edges in UCCA enable reentrancy, forming a DAG together with primary edges. UD allows reentrancy when including _enhanced dependencies_Schuster and Manning (2016),8 which form (bi-lexical) graphs, representing phenomena such as predicate ellipsis (e.g., gapping), and shared arguments due to coordination, control, raising and relative clauses.

Footnote 8: [https://universaldependencies.org/u/overview/enhanced-syntax.html](https://universaldependencies.org/u/overview/enhanced-syntax.html)

UCCA is more inclusive in its use of remote edges, and accounts for the entire class of implicit arguments termed _Constrational Null Instantiation_ in FrameNet Ruppenhofer et al. (2016). For example, in "The Pentagon is bypassing official US intelligence channels [...] in order to create strife" (from EWT), remote edges mark _Pentagon_ as a shared argument of _bypassing_ and _create_. Another example is "if you call for an appointment [...] so you can then make one", where a remote edge in UCCA indicates that _one_ refers to _appointment_. Neither is covered by enhanced UD.

In order to facilitate comparison, we remove remote edges and enhanced dependencies in the conversion process. We thus compare basic UD and UCCA trees, deferring a comparison of UCCA and enhanced UD to future work.

### Extensions to the Converter

We extend the unified DAG converter to remove further non-content differences.

Unanalyzable units.An unanalyzable phrase is represented in UCCA as a single unit covering multiple terminals. In multi-word expressions (MWEs) in UD, each word after the first is attached to the previous word, with the flat, fixed or goeswith relations (depending on whether the expression is grammaticalized, or split by error). We remove edges of these relations and join the corresponding pre-terminals to one unit.

Promotion of conjunctions.The basic conversion generally preserves terminal yields: the set of terminals spanned by a non-terminal is the same as the original dependency yield of its head terminal (e.g., in Figure 3, the yield of the non-terminal headed by _graduation_ is "After graduation", the same as that of "graduation" in Figure 2).

Since UD attaches subordinating and coordinating conjunctions to the subsequent conjunct, this results in them being positioned in the same conjunct they relate (e.g., _After_ will be included in the first conjunct in "After arriving home, John went to sleep"; _and_ will be included in the second conjunct in "John and Mary"). In contrast, UCCA places conjunctions as siblings to their conjuncts (e.g., "[After] [arriving home], [John went to sleep]" and "[John] [and] [Mary]").

To abstract away from these convention differences, we place coordinating and subordinating conjunctions (i.e., cc-labeled units, and marked-labeled units with an advcl head such as _when_, _if_, _after_) as siblings of their conjuncts.

## 5 Analysis of Divergences

Using the shared format, we turn to analyzing the content differences between UCCA and UD.9

Footnote 9: See [http://bit.ly/uccaud](http://bit.ly/uccaud) for a detailed explanation of each example in this section.

### Confusion Matrix

Table 3 presents the confusion matrix of categories between the converted UD and UCCA, calculated over all sentences in the training and development sets of the shared EWT reviews corpus. We leave the test set out of this evaluation to avoid contamination for future parsing experiments.

In case of multiple UCCA units with the same terminal yield (i.e., units with a single non-remote child), we take the top category only, to avoid double-counting. Excluding punctuation, this results in 60,434 yields in UCCA and 58,992 in UD. Of these, 52,280 are common, meaning that a UCCA "parser" developed this way would get a very high F1 score of 87.6%, if it is provided with the gold UCCA label for every converted edge.

Some yields still have more than one UCCA category associated with them, due to edges with multiple categories (\({\bf A}|{\bf P}\) and \({\bf A}|{\bf S}\)). For presentation reasons, 0.15% of the UCCA units in the data are not presented here, as they belong to rare (< 0.1%) multiple-category combinations.

Only 82.6% of UD's syntactic arguments (ccomp, csubj, iobj, nsubj, obj, obll and xcomp) are UCCA Participants, and only 72.9% of the Participants are syntactic arguments--a difference stemming from the Scene/non-Scene (SS5.2) and argument/adjunct (SS5.3) distinctions. Moreover, if we identify predicates as words having at least one argument and Scenes as units with at least one Participant, then only 92.1% of UD's predicates correspond to Scenes (many are secondary relations within one scene), and only 80% 

of Scenes correspond to predicates (e.g., eventing nouns, which are not syntactic predicates).

Examining the _head_ row in Table 3 allows us to contrast the schemes' notions of a head. _head-labeled units have at least one dependent in UD, or are single-clause sentences (technically, they are non-terminals added by the converter). Of them, 75.7% correspond to Processes, States, Parallel Scenes or Centers, which are UCCA's notions of semantic heads, and 11.6% are left unmatched, mostly due to MWEs analyzed in UD but not in UCCA (SS5.4). Another source of unmatched units is inter-Scene linkage, which tends to be flatter in UCCA (SS5.5). The rest are mostly due to head swap (e.g., "_all_ of Dallas", where _all_ is a Quantifier of _Dallas_ in UCCA, but the head in UD).

In the following subsections, we review the main content differences between the schemes, as reflected in the confusion matrix, and categorize them according to the UD relations involved.

### Scenes vs. Non-Scenes

UCCA distinguishes between Scenes and non-Scenes. This distinction crosses UD categories, as a Scene can be evoked by a verb, an eventive or stative noun (_negotiation_, _fatigue_), an adjective or even a preposition ("this is _for_ John").

Core syntactic arguments.Subjects and objects are usually Participants (e.g., "_wine_ was excellent"). However, when describing a Scene, the subject may be a Process/State (e.g., "but _service_ is very poor"). Some wh-pronouns are the subjects or objects of a relative clause, but are Linkers or Relators, depending on whether they link Scenes or non-Scenes, respectively. For example, "who" in "overall, Joe is a happy camer _who_ has found a great spot" is an nsubj, but a Linker. Other arguments are Adverbials or Time (see SS5.3), and some do not match any UCCA unit, especially when they are parts of MWEs (see SS5.4).

Adjectural modifiersare Adverbials when modifying Scenes ("_romantic_ dinner"), States when describing non-Scenes ("_beautiful_ hotel") or when semantically predicative ("such a _convenient_ location"), or Elaborators where defining inherent properties of non-Scenes ("_medical_ school").

Nominal and clausal modifiers.Most are Participants or Elaborators, depending on whether they modify a Scene (e.g., "discount _on services_" and "our decision _to buy when we did_" are Participants, but "_my car's_ gears and brakes" and "Some of the younger kids _that work there_" are Elaborators). Unmatched acl are often free relative clauses (e.g., in "the prices were worth what I _got_", _what_ is the obj of _worth_ but a Participant of I _got_).

Case markers.While mostly Relators modifying non-Scenes (e.g., "the team _at_ Bradley

 

Chevron"), some case markers are Linkers linking Scenes together (e.g., "very informative website _with_ a lot of good work"). Others are Elaborators (e.g., "_over_ a year") or States when used as the main relation in verbless or copula clauses (e.g., "it is right _on_ Wisconsin Ave").

Coordination.Coordinating conjunctions (cc)are Connectors where they coordinate non-Scenes (e.g., "Mercedes _and_ Dan") or Linkers where they coordinate Scenes (e.g., "outdated _but_ not bad"). Similarly, conjuncts and list elements (conj, list) may be Parallel Scenes (H), or Centers when they are non-Scenes.10

Footnote 10: While in UD the conjunction cc is attached to the following conjunct, in UCCA coordination is a flat structure. This is a convention difference that we normalize (§4.2).

Determiners.Articles are Functions, but determiners modifying non-Scenes are Elaborators (e.g., "I will never recommend this gym to _any_ woman"). Where modifying Scenes (mostly negation) they are marked as Adverbials. For example, "_no_ feathers in stock", "_what_ a mistake", and "the rear window had _some_ leakage" are all Adverbials.

### Primary and Secondary Relations

UD distinguishes core arguments, adverb modifiers, and obliques (in English UD, the latter mostly correspond to prepositional dependents of verbs). UCCA distinguishes Participants, including locations and abstract entities, from secondary relations (Adverbials), which cover manner, aspect and modality. Adverbials can be verbs (e.g., _begin_, _fail_), prepositional phrases (_with disrespect_), as well as models, adjectives and adverbs.

Adverbs and obliques.Most UD adverb modifiers are Adverbials (e.g., "I _sometimes_ go"), but they may be Participants, mostly in the case of semantic arguments describing location (e.g., _here_). Obliques may be Participants (e.g., "wait _for Nick_"), Time (e.g., "for _over 7 years_") or Adverbials--mostly manner adjuncts (_by far_).

Clausal argumentsare Participant Scenes (e.g., "it was great _that they did not charge a service fee_", "did not really know _what I wanted_" or "I asked them _to change it_"). However, when serving as complements to a secondary verb, they will not match any unit in UCCA, as it places secondary verbs on the same level as their primary relation. For example, _to pay_ is an xcomp in "they have to pay", while the UCCA structure is flat: _have to_ is an Adverbali and _pay_ is a Process. Single-worded clausal arguments may correspond to a Process/State, as in "this seems _great_".

Auxiliary verbsare Functions (e.g., "_do_ not forget"), or Adverbials when they are modals (e.g., "you _can_ graduate"). Semi-modals in UD are treated as clausal heads, which take a clausal complement. For example, in "able to do well", UD treats _able_ as the head, which takes _do well_ as an xcomp. UCCA, on the other hand, treats it as an Adverbial, creating a mismatch for xcomp.

### Multi-Word Expressions

UD and UCCA treat MWEs differently. In UD they include names, compounds and grammaticalized fixed expressions. UCCA treats names and grammaticalized MWEs as unanalyzable units, but also a range of semantically opaque constructions (e.g., light verbs and idioms). On the other hand, compounds are not necessarily unanalyticable in UCCA, especially if compositional.

Compounds.English compounds are mostly nominal, and are a very heterogeneous category. Most compounds correspond to Elaborators (e.g., "_industry_ standard"), or Elaporator Scenes (e.g., "_out-of-place_ flat-screen TV"), and many are unanalyzable expressions (e.g., "_mark_ up"). Where the head noun evokes a Scene, the dependent is often a Participant (e.g., "_food_ craving"), but can also be an Adverbial (e.g., "_first time_ buyers") depending on its semantic category. Other compounds in UD are phrasal verbs (e.g., "figure _out_", "cleaned _up_"), which UCCA treats as unanalytica (leading to unmatched units).

Core arguments.A significant number of subjects and objects are left unmatched as they form parts of MWEs marked in UCCA as unanalyzable. UD annotates MWEs involving a verb and its argument(s) just like any other clause, and therefore lacks this semantic content. Examples include light verbs (e.g., "give _a try_"), idioms ("bittes _the dust_"), and figures of speech (e.g., "when _it_ comes to", "offer _a taste_ (of)"), all are UCCA units.

Complex propositions.Some complex prepositions (e.g., _according to_ or _on top of_), not encoded as MWEs in UD, are unanalyzable in UCCA.

### Linkage

Head selection.UCCA tends to flatten linkage, where UD, as a dependency scheme, selects a head 

and dependent per relation. This yields scope ambiguities for coordination, an inherently flat structure. For instance, "unique gifts and cards" is ambiguous in UD as to whether _unique_ applies only to _gifts_ or to the whole phrase--both annotated as in Figure 3(a). UCCA, allowing non-terminal nodes, disambiguates this case (Figure 3(b)).

Clausal dependents.UD categorizes clause linkage into coordination, subordination, argumenthood (complementation), and parataxis. UCCA distinguishes argumenthood but conflates the others into the Parallel Scene category. For example, "We called few companies before _we decided to hire them_" and "Check out The Willow Lounge, _you'll be happy_" are Parallel Scenes.

Note that while in UD, mark (e.g., _before_) is attached to the dependent adverbial clause, a UCCA Linker lies outside the linked Scenes. To reduce unmatched advcl instances, this convention difference is fixed by the converter (SS4.2). Many remaining unmatched units are due to conjunctions we could not reliably raise. For instance, the marker _to_ introducing an xcomp is ambiguous between Linker (purposive _to_) and Function (infinitive marker). Similarly, wh-pronouns may be Linkers ("he was willing to budge a little on the price _which_ means a lot to me"), but have other uses in questions and free relative clauses. Other mismatches result from the long tail of differences in how UD and UCCA construe linkage. Consider the sentence in Figure 5. While _moment_ is an oblique argument of _know_ in UD, _From the moment_ is analyzed as a Linker in UCCA.

### Other Differences

AppositionsIn UD always follow the modified noun, but named entities in them are UCCA Centers, regardless of position (e.g., in "its sister store Peking Garden", the UD head _its sister store_ is an Elaborator, while _Peking Garden_ is the Center).

Copulas.UCCA distinguishes copular constructions expressing identity (e.g., "This _is_ the original Ham's restaurant") where the copula is annotated as State, and cases of attribution (e.g., "Mercedes and Dan _are_ very thorough") or location (e.g., "Excellent chefs _are_ in the kitchen"), where the copula is a Function.

Discourse markers and interjections.Units relating a Scene to the speech event or to the speaker's opinion are Ground (e.g., "_no_, Warwick in New Jersey" and "_Please_ visit my website"). On the other hand, discourse elements that relate one Scene to another are Linkers (e.g., _anyway_).

Vocativesare both Ground and Participants if they participate in the Scene _and_ are the party addressed. For example, _Mark_ in "Thanks _Mark_" is both the person addressed and the one thanked.11

Footnote 11: The \({\bf A}|{\bf G}\) column is omitted from Table 3 as this category combination occurs in only 0.02% of edges in the corpus.

Expletives and subjects.Epletives are generally Functions, but some instances of _it_ and _that_ are analyzed as nsubj in UD and as Function in UCCA (e.g., "_if_'s like driving a new car").

Excluded relations.We exclude the following UD labels, as they are irrelevant to our evaluation: root (always matches the entire sentence); punct (punctuation is ignored in UCCA evaluation); dep (unspecified dependency), orphan (used for gapping, which is represented using remote edges in UCCA--see SS4.1); fixed,flat and goeswith (correspond to parts of unanalyzable units in UCCA, and so do not represent units on their own--see SS4.2); reparamdum and dislocated (too rare in EWT).

## 6 Fine-Grained UCCA Parsing Evaluation

In SS5 we used our comparison methodology, consisting of the conversion to a shared format and matching units by terminal yield, to compare goldstandard UD and UCCA. In this section we ap

Figure 4: Coordination in UD and UCCA.

Figure 5: Clause linkage in UD and UCCA.

 

ply the same methodology to parser outputs, using gold-standard UD for fine-grained evaluation.

### Experimental Setup

Data.In addition to the UCCA EWT data (SS3), we use the reviews section of the UD v2.3 English_EWT treebank Nivre et al. (2018),12 annotated over the exact same sentences. We additionally use UDPipe v1.2 Straka et al. (2016); Straka and Strakova (2017), trained on English_EWT,13 for feature extraction. We apply the extended converter to UD as before (SS4.2).

Footnote 12: [https://hdl.handle.net/11234/1-2895](https://hdl.handle.net/11234/1-2895)

Parser.We train TUPA v1.3 Hershcovich et al. (2017, 2018) on the UCCA EWT data, with the standard train/development/test split. TUPA uses POS tags and syntactic dependencies as features. We experiment both with using gold UD for feature extraction, and with using UDPipe outputs.

Evaluation by gold-standard UD.UCCA evaluation is generally carried out by considering a predicted unit as correct if there is a gold unit that matches it in terminal yield and labels. Precision, Recall and F-score (F1) are computed accordingly. For the fine-grained analysis, we split the gold-standard, predicted and matched UCCA units according to the labels of the UD relations whose dependents have the same terminal yield (if any).

### Results

Table 4 presents TUPA's scores on the UCCA EWT development and test sets. Surprisingly, using UDPipe for feature extraction results in better scores than gold syntactic tags and dependencies.

Table 5 shows fine-grained evaluation by UD relations. TUPAC does best on auxiliaries and determiners, despite the heterogeneity of corresponding UCCA categories (see Table 3), possibly by making lexical distinctions (e.g., modals and auxiliary verbs are both UD auxiliaries, but are annotated as Adverbials and Functions, respectively).

Copulas and coordinating conjunctions pose a more difficult distinction, since the same lexical items may have different categories depending on the context: State/Function for copulas, due to the distinction between identity and attribution, and Connector/Linker for conjunctions, due to the distinction between Scenes and non-Scenes. However, the reviews domain imposes a strong prior for both Function and Linker, respectively), which TTPA learns successfully.

Inter-clause linkage (conj, advcl, xcomp, cocomp, parataxis, acl and csubj) is a common source of error for TUPA. Although the match between UCCA and UD is not perfect in these cases, it is overall better than Tupta's unlabeled performance, despite using gold-standard syntactic features. Our results thus suggest that encoding syntax more directly, perhaps using syntactic scaffolding Swayamdipta et al. (2018) or guided attention Strubell et al. (2018), may assist in predicting unit boundaries. However, TUPAC often succeeds at making distinctions that are not even encoded in UD. For example, it does reasonably well (71%) on distinguishing between noun modifiers of Scene-evoking nouns Participants and modifiers of other nouns Elaborators, surpassing a majority baseline based on the UD relation (51%). Lexical resources that distinguish eventive and relational nouns from concrete nouns may allow improving it even further. In the similar case of compounds, lexical resources for light verbs and idioms may increase performance.

## 7 Discussion

NLP tasks often require semantic distinctions that are difficult to extract from syntactic representations. Consider the example "after graduation, John moved to Paris" again. While _graduation_ evokes a Scene (Figure 1), in UD it is an oblique modifier of _moved_, just like _Paris_ is (Figure 2). The Scene/non-Scene distinction (SS5.2) would assist structural text simplification systems in paraphrasing this sentence to two sentences, each one containing one Scene Sulem et al. (2018).

Another example is machine translation--translating the same sentence into Hebrew, which does not have a word for _graduation_, would re

\begin{table}
\begin{tabular}{c c c c|c c c}  & \multicolumn{3}{c|}{**Primary**} & \multicolumn{3}{c}{**Remote**} \\
**Features** & **LP** & **LR** & **LF** & **LP** & **LR** & **LF** \\ \hline Development & & & & & \\ Gold UD & 72.1 & 71.2 & 71.7 & 61.2 & 38.1 & 47.0 \\ UDPipe & 73.0 & 72.1 & 72.5 & 53.7 & 40.8 & 46.4 \\ \hline Test & & & & & & \\ Gold UD & 72.2 & 71.2 & 71.7 & 60.9 & 36.8 & 45.9 \\ UDPipe & 72.4 & 71.7 & 72.1 & 60.3 & 38.5 & 47.0 \\ \end{tabular}
\end{table}
Table 4: Labeled precision, recall and F1 (in %) for primary and remote edges output by TUPAC on the UCCA EWT development (top) and test (bottom) sets, using either gold-standard UD or UDPipe for TUPAC features.

 

quire a clause to convey the same meaning. The mapping would therefore be more direct using a semantic representation, and we would benefit from breaking the utterance into two Scenes.

## 8 Related Work

The use of syntactic parsing as a proxy for semantic structure has a long tradition in NLP. Indeed, semantic parsers have leveraged syntax for output space pruning Xue and Palmer (2004), syntactic features Gildea and Jurafsky (2002); Hershscovich et al. (2017), joint modeling Surdeanu et al. (2008); Hajic et al. (2009), and multi-task learning Swayamdipta et al. (2016, 2018); Hershscovich et al. (2018). Empirical comparison between syntactic and semantic schemes, however, is still scarce. Rudinger and Van Durme (2014) mapped Stanford Dependencies (precursor to UD) to Hobbsian Logical Form, identifying semantic gaps in the former. PredPatt White et al. (2016), a framework for extracting predicate-argument structures from UD, was evaluated by Zhang et al. (2017) on a large set of converted PropBank annotations. Szubert et al. (2018) proposed a method for aligning AMR and UD subgraphs, finding that 97% of AMR edges are evoked by one or more words or syntactic relations. Damonte et al. (2017) refined AMR evaluation by UD labels, similar to our fine-grained evaluation of UCCA parsing.

Some syntactic representation approaches, notably CCG Steedman (2000), directly reflect the underlying semantics, and have been used to transduce semantic forms using rule-based systems Basile et al. (2012). A related line of work tackles the transduction of syntactic structures into semantic ones. Reddy et al. (2016) proposed a rule-based method for converting UD to logical forms. Stanovsky et al. (2016) converted Stanford dependency trees into proposition structures ProPS, abstracting away from some syntactic detail.

## 9 Conclusion

We evaluated the similarities and divergences in the content encoded by UD and UCCA. We annotated the reviews section of the English Web Treebank with UCCA, and used an automated methodology to evaluate how well the two schemes align, abstracting away from differences of mere convention. We provided a detailed picture of the content differences between the schemes. Notably, we quantified the differences between the notions of syntactic and semantic heads and arguments, finding substantial divergence between them. Our findings highlight the potential utility of using semantic parsers for text understanding applications (over their syntactic counterparts), but also expose challenges semantic parsers must address, and potential approaches for addressing them.

#### Acknowledgments

This work was supported by the Israel Science Foundation (grant No. 929/17), and by the HUJI Cyber Security Research Center in conjunction with the Israel National Cyber Bureau in the Prime Minister's Office. We thank Jakob Prange, Nathan Schneider and the anonymous reviewers for their helpful comments.

\begin{table}
\begin{tabular}{c c|r r 



## References

* Abend and Rappoport (2013) Omri Abend and Ari Rappoport. 2013. Universal Conceptual Cognitive Annotation (UCCA). In _Proc. of ACL_, pages 228-238.
* Abend and Rappoport (2017) Omri Abend and Ari Rappoport. 2017. The state of the art in semantic representation. In _Proc. of ACL_, pages 77-89.
* Nivre et al. (2018) Joakim Nivre et al. 2018. Universal dependencies 2.3. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
* Banarescu et al. (2013) Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In _Proc. of the Linguistic Annotation Workshop_.
* Basile et al. (2012) Valerio Basile, Johan Bos, Kilian Evang, and Noortye Venhuizen. 2012. Developing a large semantically annotated corpus. In _Proc. of LREC_, pages 3196-3200.
* Bies et al. (2012) Ann Bies, Justin Mott, Colin Warner, and Seth Kulick. 2012. English web treebank. _Linguistic Data Consortium, Philadelphia, PA_.
* Birch et al. (2016) Alexandra Birch, Omri Abend, Ondrej Bojar, and Barry Haddow. 2016. HUME: Human UCCA-based evaluation of machine translation. In _Proc. of EMNLP_, pages 1264-1274.
* Cai et al. (2018) Jiaxun Cai, Shexia He, Zuchao Li, and Hai Zhao. 2018. A full end-to-end semantic role labeler, syntactic-agnostic over syntactic-aware? In _Proc. of COLING_, pages 2753-2765.
* Choshen and Abend (2018) Leshem Choshen and Omri Abend. 2018. Referenceless measure of faithfulness for grammatical error correction. In _Proc. of NAACL-HLT_.
* Damonte et al. (2017) Marco Damonte, Shay B. Cohen, and Giorgio Satta. 2017. An incremental parser for Abstract Meaning Representation. In _Proc. of EACL_.
* Gildea and Jurafsky (2002) Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. _Computational Linguistics_, 28(3).
* Hajic et al. (2009) Jan Hajic, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antonia Marti, Lluis Marquez, Adam Meyers, Joakim Nivre, Sebastian Pado, Jan Stepanek, Pavel Stranak, Mihai Surdeanu, Niamwen Xue, and Yi Zhang. 2009. The CoNLL-2009 shared task: Syntactic and semantic dependencies in multiple languages. In _Proc. of CoNLL_, pages 1-18.
* He et al. (2018) Shexia He, Zuchao Li, Hai Zhao, and Hongxiao Bai. 2018. Syntax for semantic role labeling, to be, or not to be. In _Proc. of ACL_, pages 2061-2071.
* Hershcovich et al. (2017) Daniel Hershcovich, Omri Abend, and Ari Rappoport. 2017. A transition-based directed acyclic graph parser for UCCA. In _Proc. of ACL_, pages 1127-1138.
* Hershcovich et al. (2018a) Daniel Hershcovich, Omri Abend, and Ari Rappoport. 2018a. Multitask parsing across semantic representations. In _Proc. of ACL_, pages 373-385.
* Hershcovich et al. (2018b) Daniel Hershcovich, Omri Abend, and Ari Rappoport. 2018b. Universal dependency parsing with a general transition-based DAG parser. In _Proc. of CoNLL UD Shared Task_, pages 103-112.
* de Marneffe and Nivre (2019) Marie-Catherine de Marneffe and Joakim Nivre. 2019. Dependency grammar. _Annual Review of Linguistics_, 5(1):197-218.
* Nivre et al. (2016) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In _Proc. of LREC_.
* Oppen et al. (2016) Stephan Oppen, Marco Kuhlmann, Yusuke Miyao, Daniel Zeman, Silvie Cinkova, Dan Flickinger, Jan Hajic, Angelina Ivanova, and Zdenka Uresova. 2016. Towards comparability of linguistic graph banks for semantic parsing. In _Proc. of LREC_.
* Reddy et al. (2016) Siva Reddy, Oscar Tackstrom, Michael Collins, Tom Kwiatkowski, Dipjanjan Das, Mark Steedman, and Mirella Lapata. 2016. Transforming dependency structures to logical forms for semantic parsing. _TACL_, 4:127-141.
* Ribeyre et al. (2015) Corentin Ribeyre, Eric Villemonte de la Clergerie, and Djame Seddah. 2015. Because syntax does matter: Improving predicate-argument structures parsing with syntactic features. In _Proc. of NAACL-HLT_, pages 64-74.
* Rudinger and Van Durme (2014) Rachel Rudinger and Benjamin Van Durme. 2014. Is the Stanford dependency representation semantic? In _Proc. of EVENTS_, pages 54-58.
* Ruppenhofer et al. (2016) Josef Ruppenhofer, Michael Ellsworth, Miriam R. L Petruck, Christopher R. Johnson, Collin F. Baker, and Jan Scheffczyk. 2016. _FrameNet II: Extended Theory and Practice_.
* Schuster and Manning (2016) Sebastian Schuster and Christopher D. Manning. 2016. Enhanced English Universal Dependencies: An improved representation for natural language understanding tasks. In _Proc. of LREC_. ELRA.
* Schwartz et al. (2012) Roy Schwartz, Omri Abend, and Ari Rappoport. 2012. Learnability-based syntactic annotation design. In _Proc. of COLING_, pages 2405-2422.
* Stanovsky et al. (2016) Gabriel Stanovsky, Jessica Fricler, Ido Dagan, and Yoav Goldberg. 2016. Getting more out of syntax with PropS. _ArXiv preprint arXiv:1603.01648_.
* Steedman (2000) Mark Steedman. 2000. _The Syntactic Process_. MIT Press, Cambridge, MA

Milan Straka, Jan Hajic, and Jana Strakova. 2016. UDP: trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, POS tagging and parsing. In _Proc. of LREC_, Portoro, Slovenia. European Language Resources Association.
* Straka and Strakova (2017) Milan Straka and Jana Strakova. 2017. Tokenizing, POS tagging, lemmatizing and parsing UD 2.0 with UDPipe. In _Proc. of CoNLL UD Shared Task_, pages 88-99, Vancouver, Canada.
* Strubell et al. (2018) Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In _Proc. of EMNLP_, pages 5027-5038.
* Sulem et al. (2015) Elior Sulem, Omri Abend, and Ari Rappoport. 2015. Conceptual annotations preserve structure across translations: A French-English case study. In _Proc. of S2MT_, pages 11-22.
* Sulem et al. (2018a) Elior Sulem, Omri Abend, and Ari Rappoport. 2018a. Semantic structural annotation for text simplification. In _Proc. of NAACL_.
* Sulem et al. (2018b) Elior Sulem, Omri Abend, and Ari Rappoport. 2018b. Simple and effective text simplification using semantic and neural methods. In _Proc. of ACL_.
* Surdeanu et al. (2008) Mihai Surdeanu, Richard Johansson, Adam Meyers, Lluis Marquez, and Joakim Nivre. 2008. The CoNLL 2008 shared task on joint parsing of syntactic and semantic dependencies. In _Proc. of CoNLL_, pages 159-177.
* Swayamdipta et al. (2016) Swabha Swayamdipta, Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2016. Greedy, joint syntactic-semantic parsing with stack LSTMs. In _Proc. of CoNLL_, pages 187-197.
* Swayamdipta et al. (2018) Swabha Swayamdipta, Sam Thomson, Kenton Lee, Luke Zettlemoyer, Chris Dyer, and Noah A. Smith. 2018. Syntactic scaffolds for semantic structures. In _Proc. of EMNLP_, pages 3772-3782.
* Szubert et al. (2018) Ida Szubert, Adam Lopez, and Nathan Schneider. 2018. A structured syntax-semantics interface for English-AMR alignment. In _Proc. of NAACL-HLT_, pages 1169-1180.
* White et al. (2016) Aaron Steven White, Drew Reisinger, Keisuke Sakaguchi, Tim Vieira, Sheng Zhang, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2016. Universal decompositional semantics on universal dependencies. In _Proc. of EMNLP_, pages 1713-1723.
* Xue and Palmer (2004) Nianwen Xue and Martha Palmer. 2004. Calibrating features for semantic role labeling. In _Proc. of EMNLP_.
* Zhang et al. (2017) Sheng Zhang, Rachel Rudinger, and Benjamin Van Durme. 2017. An evaluation of PredPatt and open IE via stage 1 semantic role labeling. In _Proc. of IWCS_