Neutralizing Linguistically Problematic Annotations in Unsupervised Dependency Parsing Evaluation

Dependency parsing is a central task in the field of Natural Language Processing (NLP). The task involves the automatic labeling of natural language sentences with dependency structures, such that each word is labeled as the dependent of another word in the sentence (its syntactic head). This formalism is important, both in the linguistic aspect (Mel’čuk, 1988) and in empirical aspects, as it useful in higher level NLP applications such as Statistical Machine Translation (Ding and Palmer, 2004; Xu et al., 2009). Dependency parsing tools (dependency parsers) usually use machine learning techniques. In traditional learning algorithms, automatic tools receive labeled training data, and attempt at inferring a generalization of this data in order to annotate new (unlabeled) data (supervised learning). Recently, there has been a rise in the popularity of unsupervised algorithms, where automatic tools are trained on plain text only (unlabeled data) and induce a labeling for a given test sentence. We focus on the unsupervised setting in this work. Unsupervised learning is a challenging task, as it requires the automatic tools to establish the inter-connections in the data that are used to build the annotation. However, in some cases, there is more than one reasonable way to annotate the data. In such cases, the annotation does not reflect profound data correlations, but may be the result of some arbitrary decision which the manual annotator is forced to make. These cases affect unsupervised tools drastically. As the data does not always clearly point out towards one specific annotation, unsupervised tools have no way of guessing the “correct” annotation (as they do not see any labeled data during the training phase). As a result, making the wrong (plausible) choice could result in a serious performance decrease. In this work we consider an example of such a phenomenon, and focus on problematic dependency structures, for which the dependency annotation is not under linguistic consensus. Examples include coordination structures (e.g., “John and Mary”) and infinitive verbs (e.g., “to eat”), which both have more than one linguistically plausible dependency annotation (Kübler et al., 2009). Although there is more than one way to annotate the data, the gold standards against which dependency parsers are evaluated are forced to choose one of these annotations. We start by experimenting with three leading unsupervised parsers (Klein and Manning, 2004; Cohen and Smith, 2009; Spitkovsky et al., 2010a), and show that for each of them, a small set of parameters can be found whose modification yields a significant improvement in the standard evaluation measures. These parameters correspond to local cases where no linguistic consensus exists as to the correct annotation, and in which the parsers learned a wrong annotation (that is, a plausible annotation which is not the same as the gold standard). As a result, forcing the parsers to choose the “right” annotation, by modifying the parameters controling the annotations of these structures, yields a substantial improvement. Nevertheless, both versions of the parsers (the one with the original parameters and the one with the modified ones) tag every language structure the same, except for problematic structures, where both take a linguistically plausible approach. As a result, this performance improvement does not represent true quality