Venue: NAACL 2018

# Reference-less Measure of Faithfulness for Grammatical Error Correction

Leshem Choshen1 and Omri Abend2

1School of Computer Science and Engineering, 2Department of Cognitive Sciences

The Hebrew University of Jerusalem

leshem.choshen@mail.huji.ac.il, oabend@cs.huji.ac.il

###### Abstract

We propose USim, a semantic measure for Grammatical Error Correction (GEC) that measures the semantic faithfulness of the output to the source, thereby complementing existing reference-less measures (RLMs) for measuring the output's grammaticality. USim operates by comparing the semantic symbolic structure of the source and the correction, without relying on manually-curated references. Our experiments establish the validity of USim, by showing that (1) semantic annotation can be consistently applied to ungrammatical text; (2) valid corrections obtain a high USim similarity score to the source; and (3) invalid corrections obtain a lower score.1

Footnote 1: Our code is available in [https://github.com/borgr/USim](https://github.com/borgr/USim).

## 1 Introduction

Evaluation in Monolingual Translation, and particularly in Grammatical Error Correction (GEC) is a challenging research field, much due to the difficulty in integrating different types of rewriting operations into a single measure, and the vast number of valid outputs Tetreault and Chodorow (2008); Madnani et al. (2011); Chodorow et al. (2012); Bryant and Ng (2015). These difficulties have recently motivated a number of proposals for new, improved reference-based measures (RBMs) Dahlmeier and Ng (2012); Felice and Briscoe (2015); Napoles et al. (2015).

Nevertheless, the size and heterogeneity of the space of valid outputs per sentence often prohibits obtaining a reference set that covers this space well, thereby limiting the applicability of RBMs Bryant and Ng (2015). To address this we propose a semantic RLM, USim, that operates by measuring the graph distance between the semantic representations of the source and the output. Reliable RLMs are appealing both in not relying on references, which are costly to collect, and in avoiding the biases incurred by selecting references that necessarily cannot exhaust the vast space of valid corrections.

Our proposal complements the RLM proposed by Napoles et al. Napoles et al. (2016), which uses grammatical error detection techniques to assess the grammaticality of the output, and the work of Asano et al. Asano et al. (2017), who advocate the use of RLMs for fluency, grammaticality and meaning preservation, but state that a meaning preservation measure for GEC is currently lacking. A similar decomposition of output quality to its adequacy (similar to faithfulness) and fluency (related to grammaticality), has been used in machine translation (MT) evaluation (e.g., Banches et al., 2015).

As a test case, we use the UCCA semantic scheme Abend and Rappoport (2013), motivated by its recent use in semantic evaluation of MT Birch et al. (2016) and text simplification Sulem et al. (2018) systems. Nevertheless, USim can be easily adapted to other semantic schemes, such as AMR Banarescu et al. (2013). USim is conceptually related to RLMs developed for MT Reeder (2006); Albrecht and Hwa (2007); Specia et al. (2009, 2010). Notably, XMEANT Lo et al. (2014) compares the source to the output in terms of their semantic role labeling structures. Our use of UCCA is motivated by its wider coverage of predicate types, as opposed to MEANT's focus on verbal predicates, and UCCA's preservation of structure across translations Sulem et al. (2015). See Birch et al. (2016) for further discussion.

We conduct experiments to confirm USim's validity. Specifically, we show that (1) UCCA can be consistently and automatically applied to learner language (LL) (SS4.2), (2) USim is not prone to unduly penalize valid corrections (SS4.2), and (3) USim assigns a lower score to corrections of poor quality (SS4.5). Our experiments also indicate that UCCA parsing technology is already sufficiently mature for an automatic variant of USim to provide reliable results (SS4.3).

## 2 Background

**LL Annotation.** While most linguistic theories propose that each learner makes consistent use of syntax Huebner (1985); Tarone (1983), this use may not conform to the syntax of the learned language, or of any other known language. This entails difficulties in defining syntactic annotation for LL, as the annotated syntax differs between learners.



Syntactic schemes for LL annotate syntactically erroneous sentences in different ways. Berzak et al. (2016) and Ragheb and Dickinson (2012) annotate according to the syntax used by the learner, even if this use is not grammatical. Such annotation may be unreliable for measuring faithfulness, as GEC systems aim to alter these erroneous syntactic structures. Nagata and Sakaguchi (2016) take the opposite approach, and remain faithful to the syntax intended by the learner. This has also been the tradition in works on parser robustness (Bigert et al., 2005; Foster, 2004). However, such approach is prone to inconsistencies due to the variety of different syntactic structures that can be used to express a similar meaning.

In this paper, we use semantic annotation to structurally represent LL. Semantic structures are faithful to the intended meaning, and not to the formal realization, and thus face fewer conflicts where the syntactic structure used diverges from the one intended. We are not aware of any previous attempts to semantically annotate LL text.

The UCCA Scheme.UCCA is a semantic annotation scheme that builds on typological and cognitive linguistic theories. The scheme's aims are to provide a coarse-grained, cross-linguistically applicable representation. Importantly, UCCA's categories directly reflect semantic, rather than distributional distinctions. For instance, UCCA is not sensitive to POS distinctions: a Scene's main relation can be a verb but also an adjective ("He is **thin**") or a noun ("John's **decision**"). Indeed, Sulem et al. (2015) have found that UCCA structures are preserved remarkably well across English-French translations.

UCCA structures are directed acyclic graphs, where the words correspond to (a subset of) their leaves. The nodes of the graphs, called _units_, are either leaves or several elements jointly viewed as a single entity according to some semantic or cognitive consideration. The edges bear one or more categories, indicating the role of the sub-unit in the relation that the parent represents.

UCCA views the text as a collection of _Scenes_ and relations between them. A Scene describes a movement, an action or a state which is persistent in time. Every Scene contains one main relation, zero or more _Participants_, interpreted in a broad sense to include locations, destinations and complement clauses, and _Adverbials_, such as manner or aspectual modifiers.

## 3 Semantic Faithfulness Measures

We start by defining a simplified measure, used for inter-annotator agreement (IAA). The measure compares two UCCA annotations over the same set of tokens. We then proceed to define USim, which compares two UCCA structures over alignable but different sets of tokens.

Iaa Measure.We define a similarity measure over UCCA annotations \(G_{1}\) and \(G_{2}\) that share their set of leaves (tokens) \(W\). For a node \(v\) in \(G_{1}\) or \(G_{2}\), define its yield \(yield(v)\subseteq W\) as its set of leaf descendants. Define a pair of edges \((v_{1},u_{1})\in G_{1}\) and \((v_{2},u_{2})\in G_{2}\) to be matching if \(yield(u_{1})=yield(u_{2})\) and they have the same label. Labeled precision and recall are defined by dividing the number of matching edges in \(G_{1}\) and \(G_{2}\) by \(|E_{1}|\) and \(|E_{2}|\) respectively. _DAG F-score_ is their harmonic mean. The measure collapses to the common parsing \(F\)-score if \(G_{1},G_{2}\) are trees.

The Usim Measure.Computing a faithfulness measure is slightly more involved, as the source sentence graph \(G_{s}\) and its correction \(G_{c}\) do not share the same set of leaves. We assume a (possibly partial, possibly many-to-1) alignment between \(G_{s}\) and \(G_{c}\), \(A\subset V_{s}\times V_{c}\).

An edge \((v_{1},v_{2})\in E_{c}\) is said to match an edge \((u_{1},u_{2})\in E_{s}\) if they have the same label and \((v_{2},u_{2})\in A\). Recall (Precision) is defined as the ratio of edges in \(E_{s}\) (\(E_{c}\)) that have a match in \(E_{c}\) (\(E_{s}\)) respectively, and \(F\)-score is their harmonic mean. We note that this measure collapses to the _DAG F-score_ if \(A\) includes all pairs of nodes in \(E_{s}\) and \(E_{c}\) that have the same yield. See Figure 1.

In order to define the alignment between \(V_{s}\) and \(V_{c}\), we begin by aligning the leaves (tokens) in \(V_{s}\) and \(V_{c}\). Alignment is cast as a weighted bipartite graph matching problem. Edge weights are assigned to be the edit distances between the tokens. We note that aligning words in GEC (and other monolingual translation tasks) is much simpler than in MT, as most of the words are unchanged, deleted fully, added, or changed slightly. Denote the resulting leaf alignment with \(A_{l}\subset Leaves_{s}\times Leaves_{c}\). We extend \(A_{l}\) to define the node alignment \(A\), aligning each non-leaf \(v\in V_{s}\) to the node \(u\in V_{c}\) that maximizes

\[w\left(v,u\right)=\frac{|A_{t}\cap\left(yield\left(u\right)\times yield\left( v\right)\right)|}{|yield\left(u\right)|}.\]

We exclude from \(A\) zero-weighted pairs. USim is defined to be the \(F\)-score resulting from \(A\). As the alignment may differ when aligning nodes from \(V_{c}\) to

Figure 1: UCCA structures of a learner language (top) and correction (bottom) including word alignments (dashed). On the edges are labels and numbers aligned to (top) or indexes (bottom). Precision is \(\tfrac{7}{9}\) Recall is \(\tfrac{7}{7}\).

 

and the other way around, we report USim in both directions.

USim is somewhat more relaxed than _DAG \(F\)-score_, as, unlike DAG \(F\)-score, it also aligns nodes whose yields are not in perfect alignment with one another. This relaxation is necessary, given that corrections often add or remove nodes, thus eliminating the possibility of a perfect alignment. In order to obtain comparable IAA scores, we report IAA using USim as well.

For completeness, we replicate the protocol used by Sulem et al. (2015) for comparing the UCCA annotations of standard English-French translations, which we call Distributional Similarity (DistSim). For a given UCCA label \(l\), \(c_{i}(l)\) is the number of \(l\)-labeled UCCA edges in the i-th source sentence, and \(d_{i}(l)\) is the number of \(l\)-labeled UCCA edges in its corresponding correction. We define DistSim(I) between these sentences to be \(\frac{1}{N}\sum_{i=1}^{N}|c_{i}(l)-d_{i}(l)|\), where \(N\) is the total number of sentence pairs.

## 4 Experiments

We conduct four types of experiments to validate USim, showing that: (1) semantic annotation can be consistently applied to LL through inter-annotator agreement (IAA) experiments; (2) a valid corrector scores high on USim; (3) an automatic UCCA parser can reliably replace human annotation for USim; (4) USim is sensitive to changes in meaning.

### Experimental Setup.

We train two UCCA annotators, the first author and a paid in-house annotator by annotating both LL and standard English passages, until a high enough agreement is reached (6 training hours). Training passages are excluded from the evaluation. We use UCCA's annotation guidelines2 without any adaptations.

Footnote 2: [http://www.cs.huji.ac.il/~oabend/ucca.html](http://www.cs.huji.ac.il/~oabend/ucca.html)

We experiment on 7 essays and their corrections, each comprising about 500 tokens (see supplementary material 1). In order to measure IAA, we assigned 4 of these essays to both annotators. In order to measure the faithfulness score for a valid correction, we annotate both the source and the manually corrected versions of 6 essays, 3 of which were annotated by both annotators.

### The Faithfulness of Valid Corrections.

We obtain an IAA _DAG \(F\)-score_ of 0.845 (Precision 0.834, Recall 0.857), which is comparable to the IAA reported for English Wikipedia texts by Abend and Rappoport (2013). As another point of comparison, we doubly annotate 3 corrected NUCLE Dahlmier et al. (2013) passages, obtaining a similar IAA. These results suggest that UCCA annotating LL does not degrade IAA: it can be applied as consistently to LL as to standard English.

Table 1 (left-hand side) presents the USim scores obtained by comparing the NUCLE references and the source, or equivalently the score of a valid correction. To control for differences between the annotators, we explore both a setting where both sides are annotated by the same annotator, and a setting where they are annotated by different ones. As an upper bound on the score of a valid corrector (using different annotators), we also report the USim IAA on source sentences.

Our results indicate that a valid correction obtains a score comparable to the IAA, which indicates that USim is indeed insensitive to the surface divergence between a source sentence and its valid corrections. Finally, we compute the DistSim measure between the source and reference sentences (Table 1, right-hand side), obtaining similar results to those obtained by Sulem et al. (2015). It suggests that on a coarse grained level, UCCA structures are as robust to grammatical error corrections as they are to translation from English to French, which was shown to be very robust, specifically more robust than syntactic representation Sulem et al. (2015).

### Automatic USim.

We experiment with an automatic variant of USim, where UCCA structures are parsed automatically. We use the TUPA parser Hershcovich et al. (2017) to generate UCCA structures, instead of the human annotators. Otherwise the setup is as above. TUPA is used with its biLSTM model, trained on the UCCA English Wikipedia corpus.

We obtain a USim score of 0.7 between the parses of the reference correction and the source, which is comparable to the parser's reported performance (0.73 in-domain, 0.68 out-of-domain), despite not performing any domain adaptation to LL. That is, the UCCA parses of the source and the correction are roughly as similar to each other as they are to their gold standard parse. This supports the hypothesis that semantic pars

\begin{table}
\begin{tabular}{p{49.8pt}|p{28.5pt}|p{28.5pt}|p{28.5pt}||p{28.5pt}|p{28.5pt}}  & \multicolumn{3}{c||}{USim} & \multicolumn{3}{c}{DistSim} \\ \cline{2-7}  & s\(\rightarrow\)r & r\(\rightarrow\)s & Avg & A+D & Scene \\ \hline Different & 0.85 & 0.83 & 0.84 & 0.96 & 0.93 \\ Same & 0.92 & 0.91 & 0.92 & 0.97 & 0.96 \\ \hline \hline IAA & 0.85 & 0.81 & 0.83 & - & - \\ \hline SAR15 & - & - & - & 0.95 & 0.96 \\ \end{tabular}
\end{table}
Table 1: The faithfulness of valid corrections. The left-hand side presents USim, where s\(\rightarrow\)r is the setting where alignment is computed from the source to the reference, r\(\rightarrow\)s is the other way around, and Avg is their average. The right-hand side presents DistSim for the UCCA categories Participants and Adverbials together (A+D), and for Scenes (Scene). Rows indicate whether the same annotator annotated the source and reference or not. For comparison, the IAA row is the IAA computed using USim. Results show that the valid corrector’s faithfulness is comparable with IAA. SAR15 are reported by Sulem et al. on English-French translations; similarity is comparable to ours.

 

ing technology is sufficiently mature to be applicable to USim. Results also suggest an improvement in parsing performance may further improve these scores.

### Sensitivity to Error Types

To provide another perspective on automated USim's behaviour, we examined the measure's sensitivity to different error types, using MAEGE [14]. For each NUCLE sentence and set of edits (replacements of sub-strings that contain an error by corrected ones. Such edit for the example in fig. 1 might be "\(gva^{*}\rightarrow\)"\(gave^{*}\), with type \(spelling\)), we sample an order in which edits are applied. We select the source randomly to be one of the resulting sentences. We then compare the difference in USim before and after applying each edit, and average these differences by the applied edit type. We denote the average difference in USim due to correction of errors of type \(t\) with \(\Delta_{t}\). The hypothesis is that \(\Delta_{t}\) should be close to 0 for all \(t\), as edits are manual and are thus assumed to be faithful. We focus on edit types with high \(|\Delta_{t}|\) to better understand where USim fails. See table 2 in the supplementary material for complete results.

We find that among the 5 most penalized error types by USim are "unclear meaning" and corrections of type "other", that fit no specific type; these corrections may actually change the meaning of the original sentence. In the most penalized and most rewarded changes we see "Dangling Modifier", "Pronoun Reference" and "Word Tone" errors, the first usually changes a word into a more complex structure and the latter two the opposite. Such changes alter the lower levels of the UCCA structure (near the leaves); a similarity measure that focuses on the top of the DAG, or one that performs a better lexical semantic abstraction, may address this sensitivity. Corrections of incorrect word order are also highly rewarded (high \(\Delta_{t}\)), probably due to parser performance (the UCCA structures themselves are not affected by word order). Training the parser with LL annotated data may address this sensitivity.

Among the most rewarded changes we also see errors of replacing rare or misconstructed words with proper English words (Acronym and Mechanical errors). We assume this is due to parser performance, as TUPA only extracts features over complete words, and has no character-level encoding at this point. Thus, all miscononstructed words fall into an out-of-vocabulary category and can only be labeled by the context.

Lastly, adding a missing verb is shown to be highly rewarded. Under the UCCA guidelines, a missing verb should be annotated as an implicit unit, but as TUPA does not generate implicit units, it is not surprising that when corrections transforms an implicit unit into an explicit word, the parser's output changes (and hence USim). Future improvements to TUPA may address this.

### Sensitivity to Unfaithfulness.

We have shown that UCCA is insensitive to differences between a source sentence and its valid correction. We now present an evaluation of the sensitivity of USim to proposed corrections that diverge semantically from the source. A semantic measure is, by its definition, sensitive to variation in the semantic dimensions which it encodes. In UCCA's case, these distinctions focus on predicate-argument structures, the inter-relations between them, and the semantic heads of complex arguments. These distinctions are widely regarded as fundamental in the NLP and linguistic literature.

In order to empirically validate this claim, we present an experiment which shows that corrections of a fairly low quality indeed receive a much lower USim faithfulness score. Current state-of-the-art systems rarely alter the source sentences enough to yield semantically unfaithful outputs [13]. Consequently, their human rankings are not determined by their semantic faithfulness, rendering them unuseful for validating USim. We instead experiment with 5 partially trained correctors, trained and evaluated on the JFLEG corpus [17] by Sakaguchi et al. (2017).

USim is computed automatically for each system's output on 754 source sentences. Low faithfulness results are expected, as these outputs include major changes, sometimes deleting full phrases from the output or changing every other word. Indeed, automatic USim obtains scores of 0.32-0.39 for 4 of the systems, and 0.19 for the system that obtains the lowest GLEU [16] score. For completeness, we run USim on the 4 references provided by JFLEG for each source and obtain scores of 0.72-0.78, suggesting the domain change is not the reason for the low USim score.

Taken together, these results indicate that USim, even in its automatic variant, is sensitive to semantic changes. Consider the example:

\begin{tabular}{l l} Source & the good student must know how to understand and work hard to get the idee. \\ Reference & A good student must be able to understand and work hard to get the idea. \\ Corrector & The good student must know how to understand and work hard to get on. \\ \end{tabular}

USim assigns the reference 0.71 and only 0.33 to the corrector. Moreover, although the reference makes more word changes than the proposed correction, it still obtains a higher USim score.

## 5 Conclusion

We propose a measure of semantic faithfulness of a correction to the source, thereby avoiding the pitfalls of reference-based evaluation. We believe that using RLMs in conjunction with RBMs in the training and development of GEC systems will better address the 

challenge of over-conservatism, and the high costs of acquiring many references.

Future work will conduct user studies to assess the relative importance of different evaluation criteria. Specifically, we will explore to what extent users are tolerant to invalid changes to the sentence's structure, i.e., violation of conservatism, relative to their tolerance to invalid changes to the sentence's meaning, i.e., violation of faithfulness. A better understanding of how these interact may lead to improved semantic evaluation that will alleviate the need for a high number of references.

## Acknowledgments

This work was partially supported by the Israel Science Foundation (grant No. 929/17), as well as by the HUJI Cyber Security Research Center in conjunction with the Israel National Cyber Bureau in the Prime Minister's Office. We thank Joel Tetreault and Courtney Napoles for helpful feedback and help with obtaining the partially trained GEC system outputs.

## References

* Abend and Rappoport [2013] Omri Abend and Ari Rappoport. 2013. Universal conceptual cognitive annotation (ucca). In _ACL (1)_, pages 228-238.
* Albrecht and Hwa [2007] Joshua Albrecht and Rebecca Hwa. 2007. A reexamination of machine learning approaches for sentence-level mt evaluation. In _Proc. of ACL_, volume 45, pages 880-887.
* Asano et al. [2017] Hiroki Asano, Tomoya Mizumoto, and Kentaro Inui. 2017. Reference-based metrics can be replaced with reference-less metrics in evaluating grammatical error correction systems. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, volume 2, pages 343-348.
* Banarescu et al. [2013] Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_, pages 178-186, Sofia, Bulgaria. Association for Computational Linguistics.
* Banchs et al. [2015] Rafael E Banchs, Luis F D'Haro, and Haizhou Li. 2015. Adequacy-fluency metrics: Evaluating mt in the continuous space model framework. _IEEE Transactions on Audio, Speech, and Language Processing_, 23(3):472-482.
* Berzak et al. [2016] Yevgeni Berzak, Jessica Kenney, Carolyn Spadine, Jing Xian Wang, Lucia Lam, Keiko Sophie Mori, Sebastian Garza, and Boris Katz. 2016. Universal dependencies for learner english. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers)_, pages 737-746. Association for Computational Linguistics.
* Bigert et al. [2005] Johnny Bigert, Jonas Sjobergh, Ola Knutsson, and Magnus Sahlgren. 2005. Unsupervised evaluation of parser robustness. In _CICLing_, pages 142-154. Springer.
* Birch et al. [2016] Alexandra Birch, Omri Abend, Ondrej Bojar, and Barry Haddow. 2016. Hume: Human ucca-based evaluation of machine translation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1264-1274.
* Bryant and Ng [2015] Christopher Bryant and Huwee Tou Ng. 2015. How far are we from fully automatic high quality grammatical error correction? In _ACL (1)_, pages 697-707.
* Chodorow et al. [2012] Martin Chodorow, Markus Dickinson, Ross Israel, and Joel R Tetreault. 2012. Problems in evaluating grammatical error detection systems. In _COLING_, pages 611-628. Citeseer.
* Choshen and Abend [2018a] Leshem Choshen and Omri Abend. 2018a. Automatic metric validation for grammatical error correction. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
* Choshen and Abend [2018b] Leshem Choshen and Omri Abend. 2018b. Inherent biases in reference-based evaluation for grammatical error correction and text simplification. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
* Dahlmeier and Ng [2012] Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 568-572. Association for Computational Linguistics.
* Dahlmeier et al. [2013] Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. 2013. Building a large annotated corpus of learner english: The nus corpus of learner english. In _Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications_, pages 22-31.
* Felice and Briscoe [2015] Mariano Felice and Ted Briscoe. 2015. Towards a standard evaluation method for grammatical error detection and correction. In _HLT-NAACL_, pages 578-587.
* Foster [2004] Jennifer Foster. 2004. Parsing ungrammatical input: an evaluation procedure. In _LBEC_. Citeseer.
* Hershcovich et al. [2017] Daniel Hershcovich, Omri Abend, and Ari Rappoport. 2017. A transition-based directed acyclic graph parser for uteca. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 1127-1138.
* Huebner [1985] Thorn Huebner. 1985. System and variability in interlanguage syntax. _Language Learning_, 35(2):141 

Chi-kiu Lo, Meriem Beloucif, Markus Saers, and Dekai Wu. 2014. Xmeant: Better semantic mt evaluation without reference translations. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 765-771, Baltimore, Maryland. Association for Computational Linguistics.
* Madnani et al. (2011) Nitin Madnani, Joel Tetreault, Martin Chodorow, and Alla Rozovskaya. 2011. They can help: Using crowdsourcing to improve the evaluation of grammatical error detection systems. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2_, pages 508-513. Association for Computational Linguistics.
* Nagata and Sakaguchi (2016) Ryo Nagata and Keisuke Sakaguchi. 2016. Phrase structure annotation and parsing for learner english. In _Proc. of ACL_, pages 1837-1847.
* Napoles et al. (2015) Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing_, volume 2, pages 588-593.
* Napoles et al. (2016) Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2016. There's no comparison: Referenceless evaluation metrics in grammatical error correction. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2109-2115. Association for Computational Linguistics.
* Napoles et al. (2017) Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. Jfleg: A fluency corpus and benchmark for grammatical error correction. _arXiv preprint arXiv:1702.04066_.
* Ragheb and Dickinson (2012) Marwa Ragheb and Markus Dickinson. 2012. Defining syntax for learner language annotation. In _COLING (Posters)_, pages 965-974.
* Reeder (2006) Florence Reeder. 2006. Measuring mt adequacy using latent semantic analysis. In _Proceedings of the 7th Conference of the Association for Machine Translation of the Americas. Cambridge, Massachusetts_, pages 176-184.
* Sakaguchi et al. (2017) Keisuke Sakaguchi, Matt Post, and Benjamin Van Durme. 2017. Grammatical error correction with neural reinforcement learning. _arXiv preprint arXiv:1707.00299_.
* Specia et al. (2010) Lucia Specia, Dhwaj Raj, and Marco Turchi. 2010. Machine translation evaluation versus quality estimation. _Machine translation_, 24(1):39-50.
* Specia et al. (2009) Lucia Specia, Marco Turchi, Nicola Cancedda, Marc Dymetman, and Nello Cristianini. 2009. Estimating the sentence-level quality of machine translation systems. In _13th Conference of the European Association for Machine Translation_, pages 28-37.
* Sulem et al. (2015) Elior Sulem, Omri Abend, and Ari Rappoport. 2015. Conceptual annotations preserve structure across translations: A french-english case study. _Proceedings of S2MT 2015_, page 11.
* Sulem et al. (2018) Elior Sulem, Omri Abend, and Ari Rappoport. 2018. Semantic structural annotation for text simplification. In _Proc. of NAACL_. To appear.
* Tarone (1983) Elaine Tarone. 1983. On the variability of interlanguage systems. _Applied linguistics_, 4(2):142-164.
* Tetreault and Chodorow (2008) Joel R Tetreault and Martin Chodorow. 2008. Native judgments of non-native usage: Experiments in preposition error detection. In _Proceedings of the Workshop on Human Judgements in Computational Linguistics_, pages 24-32. Association for Computational Linguistics.