Venue: ACL 2017

# Error-repair Dependency Parsing for Ungrammatical Texts

Keisuke Sakaguchi\({}^{\dagger}\)

Matt Post\({}^{\ddagger}\)

Benjamin Van Durme\({}^{\dagger\ddagger}\)

\({}^{\dagger}\)Center for Language and Speech Processing, Johns Hopkins University

\({}^{\ddagger}\)Human Language Technology Center of Excellence, Johns Hopkins University

{keisuke,post,vandurme}@cs.jhu.edu

###### Abstract

We propose a new dependency parsing scheme which jointly parses a sentence and repairs grammatical errors by extending the non-directional transition-based formalism of Goldberg and Elhadad (2010) with three additional actions: Substitute, Delete, Insert. Because these actions may cause an infinite loop in derivation, we also introduce simple constraints that ensure the parser termination. We evaluate our model with respect to dependency accuracy and grammaticality improvements for ungrammatical sentences, demonstrating the robustness and applicability of our scheme.

+
Footnote †: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Short Papers), pages 189–195 Vancouver, Canada, July 30 - August 4, 2017. ©2017 Association for Computational Linguistics [https://doi.org/10.18653/iv/1P17-2030](https://doi.org/10.18653/iv/1P17-2030)

## 1 Introduction

Robustness has always been a desirable property for natural language parsers: humans generate a variety of noisy outputs, such as ungrammatical webpages, speech disfluencies, and the text in language learner's essays. Such _non-canonical_ text contains grammatical errors such as substitutions, insertions, and deletions. For example, a non-native speaker of English might write "*I look in forward hear from you"_, where _in_ is inserted, _to_ is deleted, and _hearing_ is substituted incorrectly.

We propose a novel dependency parsing scheme that jointly parses and repairs ungrammatical sentences with these sorts of errors. The parser is based on the _non-directional easy-first_ (EF) parser introduced by Goldberg and Elhadad (2010) (GE herein), which iteratively adds the most probable arc until the parse tree is completed. These actions are called AttachEfft and AttachRight depending on the direction of the arc. We extend the EF parsing scheme to be robust for ungrammatical inputs by correcting grammatical errors with three new actions: Substitute, Insert, and Delete. These new actions do not add an arc between tokens but instead they edit a single token. As a result, the parser is able to jointly parse and correct grammatical errors in the input sentence. We call this new scheme _Error-Repair Non-Directional Easy-First parsing_ (EREF). Since the new actions may greatly increase the search space (e.g., infinite substitutions), we also introduce simple constraints to avoid such issues.

We first describe the technical details of EREF (SS2) and then evaluate our EREF parser with respect to dependency accuracy (robustness) and grammaticality improvements (SS3). Finally, we

Figure 1: Illustrative example of partial derivation under error-repair easy-first non-directional dependency parsing. Solid arrows represent ATTACHIGHT and ATACHLET in Goldberg and Elhadad (2010). Dotted arcs correspond to actions for each step. Following the notation by GE: arcs are directed from a child to its parent.



position this effort at the intersection of noisy text parsing and grammatical error correction (SS4).

## 2 Model

Non-directional Easy-first ParsingLet us begin with a brief review of a non-directional easy-first (EF) parsing scheme proposed by GE, which is the foundation of our proposed scheme described in the following sections.

The EF parser has a list of partial structures \(p_{1}\), ..., \(p_{k}\) (called _pending_) initialized with sentence tokens \(w_{1}\), ..., \(w_{n}\), and it keeps updating _pending_ through derivations. Unlike left-to-right (e.g., shift-reduce) parsing algorithms Yamada and Matsumoto (2003); Nivre (2004), EF iteratively selects the best pair of adjoining tokens and chooses the direction of attachment: AttachLeft or AttachRight. Once the action is committed, the corresponding dependency arc is added and the child token is removed from _pending_. The first two derivations in Figure 1 depict AttachRight and AttachLeft. Pseudocode is shown in Algorithm 1 (lines 1, 3-12).

``` Input: ungrammatical sentence= \(w_{1}\) ... \(w_{n}\) Output: a set of dependency arcs (Arcs),  repaired sentence (\(\hat{w}_{1}\) ... \(\hat{w}_{m}\)) Acts = { AttACHLeft, ATTACHRight } ActSER = { Delete, Insert, Substitute } Arcs = { } endang = p\({}_{1}\)...\(p_{n}\gets w_{1}\)...\(w_{n}\) repaired = \(\hat{w}_{1}\)...\(\hat{w}_{n}\gets w_{1}\)...\(w_{n}\) whilelen (pending) > 1do best \(\leftarrow\operatorname*{argmax}\limits_{act\in\mathit{Acts\cup\mathit{ActSER}}} score\text{ (act $(i)$)}\) s.t. \(1\leq i\leq len(\textit{pending})\cap\textsc{isLegal}(\textit{act},\textit{ pending})\) ifbest \(\in\)Actsthen (parent, child) \(\leftarrow\) edgeFor(best) Ares.add((parent, child)) pending.remove(child) elseifbest = Substitute then c = bestCandidate(best, repaired) pending.replace(\(p_{i}\), \(c\)) repaired.replace(\(\hat{w}_{p_{i}\),idx}\), \(c\) elseifbest = Deletethen pending.remove(\(p_{i}\)) repaired.remove(\(\hat{w}_{p_{i}\),idx}) Arcs.updateIndex() elseifbest = Insertthen c = bestCandidate(best, repaired) pending.insert(\(i,c\)) repaired.insert(\(p_{i}\).idx, \(c\)) Arcs.updateIndex()

**Algorithm 1**Error-repair non-directional parsing

The parser is trained using the structured perceptron Collins (2002) to choose actions to take given a set of features expanded from templates. The cost of actions is computed at every step by checking the _validity_: whether a new arc is included in the gold parse and whether the child already has all its children. See GE for further description of feature templates and structured perceptron training. Since it is possible that there are multiple _valid_ sequence of actions and it is important to examine a large search space, the parser is allowed to explore (possibly incorrect) actions with a certain probability, termed _learning with exploration_ by Goldberg and Nivre (2013).

Error-repair variant of EFError-repair non-directional easy-first parsing scheme (EREF) is a variant of EF. We add three new actions: Substitute, Delete, Insert as \(\mathit{ActSER}\). We do not deal with a swapping action Nivre (2009) to deal with word reordering errors, since the errors are even less frequent than other error types Leacock et al. (2014). Substitute replaces a token to a grammatically more probable token, Delete removes an unnecessary token, and Insert inserts a new token at a designated index. These actions are shown in Figure 1 and Algorithm 1 (lines 13-25). Because the length of _pending_ decreases as an attachment occurs, the parser also keeps the token indices in _repaired_ (line 5), which holds all tokens in a sentence throughout the parsing process. Furthermore, the parser updates token indices in _pending_ and _repaired_ when Insert or Delete occurs. Technically, when a token at \(i\) is deleted/inserted, the parser decrements/increments the indices that are \(k>=i\) (before executing the action) in _pending_, _repaired_, and parents and children in a (partial) dependency tree (_Arcs_).

To find the best candidate for Substitute and Insert efficiently, we restrict candidates to the same part-of-speech or pre-defined candidate list. We select the best candidate by comparing each n-gram language model score with the same surrounding context.

Similar to EF, while training the parser, the cost 

for \(Acts_{\text{ER}}\) is based on _validity_. The _validity_ of the new actions is computed by taking the edit distance (\(d\)) between the _Gold_ tokens (\(w^{*}_{1}\) ... \(w^{*}_{r}\)) and the sentence state that the parser stores in _repaired_ (\(\hat{w}_{1}\) ... \(\hat{w}_{m}\)). When the edit distance after taking an action (\(d\) after) is smaller than before (\(d\) before), we regard the action as _valid_ (Algorithm 2).

One serious concern of EREF is that the new actions may cause an infinite loop during parsing (e.g., infinite Substitute, or an alternative Delete and Insert sequence.). To avoid this, we introduce two constraints: (1) _edit flag_ and (2) _edit limit_. _Edit flag_ is assigned for each token as a property, and a parser is not allowed to execute \(Acts_{\text{ER}}\) on a token if its flag is on. The flag is turned on when a parser executes \(Acts_{\text{ER}}\) on a token whose flag is off. In INERT action, the flag of the inserted token is activated, while the subsequent token (which gave rise to the Insert) is not. _Edit limit_ is set to be the number of tokens in a sentence, and the parser is not allowed to perform \(Acts_{\text{ER}}\) when the total number of execution of \(Acts_{\text{ER}}\) exceeds the limit. These two constraints prevent the parser from falling into an infinite loop as well as parsing in the same order of time complexity as GE. We also add the following constraints to avoid unreasonable derivations: (i) a word with a dependent cannot be deleted and (ii) any child words cannot be substituted. All the constraints are implemented in the \(isLegal()\) function in Algorithm 1 (line 8). We note that these constraints not only prevent undesirable derivations but also leads to an efficiency in exploring the search space during training.

```
Function isValid(act, repaired, Gold) \(d\)before = editDistance(repaired, Gold) \(repaired~{}+~{}=~{}repaired.\text{apply}(act)\) \(d\)after = editDistance(required~{}+\), Gold) if\(d\)before > \(d\)afterthenreturntrue; else returnfalse; end fore
```

**Algorithm 2**Check validity during training

## 3 Experiment

Data and EvaluationWe evaluate EREF with respect to dependency parsing accuracy (Exp1) and grammaticality improvement (Exp2).1

Footnote 1: Code for the experiments is available at [http://github.com/keisks/error-repair-parsing](http://github.com/keisks/error-repair-parsing)

In the first experiment, as in GE, we train and evaluate our parser on the English dataset from the Penn Treebank Marcus et al. (1993) with the Penn2Malt conversion program (Sections 2-21 for training, 22 for tuning, and 23 for test). We use the PTB for the dependency experiment, since there are no ungrammatical text corpora that has dependency annotation on the _corrected_ texts by human.

We choose the following most frequent error types that are used in CoNLL 2013 shared task Ng et al. (2013):

1. Determiner (substitution, deletion, insertion)
2. Preposition (substitution, deletion, insertion)
3. Noun number (singular vs. plural)
4. Verb form (tense and aspect)
5. Subject verb agreement

Regarding the candidate sets for Insert and Substitute actions, following Rozovskaya and Roth (2014), we focus on the most common candidates for each error type, setting the determiner candidates to be {_a, an, the, \(\phi\)_ (as deletion)}, preposition candidates to be {_on, about, from, for, of, to, at, in, with, by, \(\phi\)_}, and verb forms to be {VB(P|Z|G|D|N)}. We build a 5-gram language model on English Gigaword with the KenLM Toolkit Heafield (2011) for EREF to select the best candidate.

We manually inject grammatical errors into PTB with certain error-rates similarly to the GenERRate toolkit by Foster and Andersen (2009), which is designed to create synthetic errors into sentences to improve grammatical error detection.

We train and tune EREF models with different token-level error injection rates from 5% (E05) to 20% (E20), because language learner corpora have generally around 5% to 15% of token level errors depending on learners' proficiency Leacock et al. (2014). Since the error injection is stochastic, we train each model with 10 runs and take an average of parser performance on the test set.

As a baseline, we use the original parser as described by GE, which is equivalent to EREF with training on an error-free corpus (E00). Since the EF baseline does not allow error correction during parsing, we pre-process the test data with a grammatical error correction system similar to Rozovskaya and Roth (2014), where a combination of classifiers for each error type corrects grammatical errors.

For evaluation, we jointly parse and correct grammatical errors in the test set with different 

error injection rates (from 0% to 20%). It is important to note that the number of tokens between the parser output and the oracle may be different because of error injection into the test set and \(Acts_{\text{ER}}\) during parsing. To handle this mismatch, we evaluate the dependency accuracy with alignment Favre et al. (2010) in the spirit of SParseval Roark et al. (2006), where tokens between a hypothesis and oracle are aligned prior to calculating the dependency accuracy.

In the second experiment, we use the Treebank of Learner English (TLE) Berzak et al. (2016) to see the grammaticality improvement in a real scenario. TLE contains 5,124 sentences and 2.69 (std 1.9) token errors per sentence. The average sentence length is 19.06 (std 9.47). TLE also provides dependency labels and POS tags on the _raw (ungrammatical)_ sentences. It is important to note that TLE has dependency annotation only for the original _ungrammatical_ sentences, and therefore we do not compute the accuracy of dependency parse in this experiment. Since the corpus size is small, we train EREF (E05 to E20) on 100k sentences from Annotated Gigaword Napoles et al. (2012) and used TLE as a test set. Spelling errors are ignored because EREF can use the POS information. Grammaticality is evaluated by a regression model Heilman et al. (2014), which scores grammaticality on the ordinal scale (from 1 to 4).

ResultsTable 1 shows the result of unlabeled dependency accuracy (UAS).3 As previously presented Foster (2007); Cahill (2015), our experiment also shows that parser performance deteriorated as the error rate in the test corpus increased. On the error-free test set (0%), the baseline (EF pipeline) outperforms other EREF models; the accuracy is lower when the parser is trained on noisier data. The difference among the models becomes small when the test set has 10% error injection rate. As the rate increases further, the trend of parser accuracy reverses. When the test set has 15% or higher noise, the E20 is the most accurate parser. This trend is presented by the slope of deterioration \(\nabla=\frac{\mathrm{accuracy}_{20\%}-\mathrm{accuracy}_{0\%}}{20}\) in Table 1; a parser trained on noisier training data shows smaller decline and more robustness.4 This indicates that the EREF is more robust than the vanilla EF on ungrammatical texts by jointly Parsing and correcting errors.

Footnote 3: Technically, it is possible to train the model with learning labels simultaneously (LAS), but there is a trade-off between the preprocessed baseline.

Table 2 demonstrates the result of grammaticality improvement (1-4 scale) on the TLE corpus, and Table 3 shows successful and failure corrections by EREF. Minimally trained models (E05 and E10) show little improvement in grammaticality because the models are too conservative to make edits. The models with higher error-injection rates (E15 and E20) achieve 0.1 to 0.3 improvements that are statistically significant. There is still room to improve regarding the amount of corrections. This is probably because TLE contains a variety of errors (e.g., collocation, punctuation) in addition to the five error types we focus. To deal with other error types, we can extend EREF by adding more actions, although it increases the search space.

From a practical perspective, the level of ungrammaticality should be realized ahead of time. This is an issue to be addressed in future research.

## Appendix B

\begin{table}
\begin{tabular}{l|c c c c c} \hline  & E05 & E10 & E15 & E20 \\ \hline \# edited sents (out of 5,124) & 175 & 391 & 583 & 861 \\ \hline grammaticality (source) & 2.92 & 2.95 & 2.95 & 2.89 \\ grammaticality (this work) & 2.96 & 2.99 & **3.27** & **2.98** \\ \hline \end{tabular}
\end{table}
Table 2: Grammaticality scores by 1-4 scale regression model Heilman et al. (2014). The first row shows the number of sentences that are made (at least one) change. Bold numbers show statistically significant improvements.

\begin{table}
\begin{tabular}{l l} \hline Successful cases \\ \hline I ’m looking forward to [-see-] +“seesing+” you next summer \\ I’ve never [-approped+]s in deal \\ There is not (++) possibility to travel \\ \hline \hline Failure cases \\ \hline I ’ve [-assisted-] + “assisting+” your new musical show \\ I am writing to complain [-about-] +“with+” somethings \\ I hope you liked +the+” everything I said \\ \hline \end{tabular}
\end{table}
Table 3: Successful and failure examples by EREF. The edits are represented by [-deletion-] and +“insertion+”. Adjacent pairs of deletion and insertion are considered as substitution.

 



## 4 Related Work

Our work lies at the intersection of parsing non-canonical texts and grammatical error correction.

Joint dependency parsing and disfluency detection has been pursued Rasooli and Tetreault (2013, 2014); Honnibal and Johnson (2014); Wu et al. (2015); Yoshikawa et al. (2016), where a parser jointly parses and detects disfluency (e.g., reparandum and interregnum) for a given speech utterance. Our work could be considered an extension via adding Substitute and Insert actions, although we depend on easy-first non-directional parsing framework instead of a left-to-right strategy. Importantly, the Delete action is easier to handle than the Substitute and Insert actions, because they bring us challenging issues about a process of candidate word generation and avoiding an infinite loop in derivation. We have addressed these issues as explained in SS2.

In terms of the literature from grammatical error correction, this work is closely related to Dahlmeier and Ng (2012), where they show an error correction decoder with the easy-first strategy. The decoder iteratively corrects the most probable ungrammatical token by applying different classifiers for each error type. The EREF parser also depends on the easy-first strategy to find ungrammatical index to be deleted, inserted, or substituted, but it parses and corrects errors jointly whereas the decoder is designed as a grammatical error correction framework rather than a parser.

There is a line of work for parsing ungrammatical sentences (e.g., web forum) by adapting an _existing_ parsing scheme on domain specific annotations Petrov and McDonald (2012); Cahill (2015); Berzak et al. (2016); Nagata and Sakaguchi (2016). Although we share an interest with respect to dealing with ungrammatical sentences, EREF focuses on the parsing scheme for repairing grammatical errors instead of adapting a parser with a domain specific annotation scheme.

More broadly, our work can also be regarded as one of the joint parsing and text normalization tasks such as joint spelling correction and POS tagging Sakaguchi et al. (2012), word segmentation and POS tagging Kaji and Klisuregawa (2014); Qian et al. (2015).

## 5 Conclusions

We have presented an _error-repair_ variant of the _non-directional easy-first_ dependency parser. We have introduced three new actions, Substitute, Insert, and Delete into the parser so that it jointly parses and corrects grammatical errors in a sentence. To address the issue of parsing incompletion due to the new actions, we have proposed simple constraints that keep track of editing history for each token and the total number of edits during derivation. The experimental result has demonstrated robustness of EREF parsers against EF and grammaticality improvement. Our work is positioned at the intersection of noisy text parsing and grammatical error correction. The EREF is a flexible formalism not only for grammatical error correction but other tasks with jointly editing and parsing a given sentence.

## Acknowledgments

This work was supported in part by the JHU Human Language Technology Center of Excellence (HLTCOE), and DARPA LORELEI. The U.S. Government is authorized to reproduce and distribute repprints for Governmental purposes. The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of DARPA or the U.S. Government.

## References

* Berrzak et al. (2016) Yevgeni Bergzak, Jessica Kenney, Carolyn Spadine, Jing Xian Wang, Lucia Lam, Keiko Sophie Mori, Sebastian Garza, and Boris Katz. 2016. Universal dependencies for learner english. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, pages 737-746.
* Cahill (2015) Aoife Cahill. 2015. Parsing learner text: to shoehorn or not to shoehorn. In _Proceedings of The 9th Linguistic Annotation Workshop_. Association for Computational Linguistics, Denver, Colorado, USA, pages 144-147.
* Collins (2002) Michael Collins. 2002. Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In _Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, pages 1-8.
* Dahlmeier and Ng (2012) Daniel Dahlmeier and Hwee Tou Ng. 2012. A beamsearch decoder for grammatical error correction. In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_. Association for Computational Linguistics, Jeju Island, Korea, pages 568-578

* Favre et al. (2010) Benoit Favre, Bernd Bohnet, and Dilek Hakkani-Tur. 2010. Evaluation of semantic role labeling and dependency parsing of automatic speech recognition output. In _2010 IEEE International Conference on Acoustics, Speech and Signal Processing_. pages 5342-5345.
* Foster (2007) Jennifer Foster. 2007. Treebanks gone bad. _International Journal of Document Analysis and Recognition (IJDAR)_ 10(3):129-145.
* Foster and Andersen (2009) Jennifer Foster and Oistein Andersen. 2009. Generate: Generating errors for use in grammatical error detection. In _Proceedings of the Fourth Workshop on Innovative Use of NLP for Building Educational Applications_. Association for Computational Linguistics, Boulder, Colorado, pages 82-90.
* Goldberg and Elhadad (2010) Yoav Goldberg and Michael Elhadad. 2010. An efficient algorithm for easy-first non-directional dependency parsing. In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_. Association for Computational Linguistics, Los Angeles, California, pages 742-750.
* Goldberg and Nivre (2013) Yoav Goldberg and Joakim Nivre. 2013. Training deterministic parsers with non-deterministic oracles. _Transactions of the Association for Computational Linguistics_ 1:403-414.
* Heafield (2011) Kenneth Heafield. 2011. KenLM: faster and smaller language model queries. In _Proceedings of the EMNLP 2011 Sixth Workshop on Statistical Machine Translation_. Edinburgh, Scotland, United Kingdom, pages 187-197.
* Heilman et al. (2014) Michael Heilman, Aoife Cahill, Nitin Madnani, Melissa Lopez, Matthew Mulholland, and Joel Tetreatult. 2014. Predicting grammaticality on an ordinal scale. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. Association for Computational Linguistics, Baltimore, Maryland, pages 174-180.
* Honnibal and Johnson (2014) Matthew Honnibal and Mark Johnson. 2014. Joint incremental disfluency detection and dependency parsing. _Transactions of the Association for Computational Linguistics_ 2:131-142.
* Kaji and Kitsuregawa (2014) Nobuhiro Kaji and Masaru Kitsuregawa. 2014. Accurate word segmentation and pos tagging for japanese microblogs: Corpus annotation and joint modeling with lexical normalization. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Doha, Qatar, pages 99-109.
* Leacock et al. (2014) Claudia Leacock, Martin Chodorow, Michael Gamon, and Joel Tetreeault. 2014. Automated grammatical error detection for language learners. _Synthesis lectures on human language technologies_ 7(1):1-170.
* Marcus et al. (1993) Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. _Computational linguistics_ 19(2):313-330.
* Nagata and Sakaguchi (2016) Ryo Nagata and Keisuke Sakaguchi. 2016. Phrase structure annotation and parsing for learner english. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, pages 1837-1847.
* Napoles et al. (2012) Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In _Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction_. Association for Computational Linguistics, pages 95-100.
* Ng et al. (2013) Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadwinoto, and Joel Tetreault. 2013. The conll-2013 shared task on grammatical error correction. In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task_. Association for Computational Linguistics, Sofia, Bulgaria, pages 1-12.
* Nivre (2004) Joakim Nivre. 2004. Incrementality in deterministic dependency parsing. In _Proceedings of the Workshop on Incremental Parsing: Bringing Engineering and Cognition Together_. Association for Computational Linguistics, Stroudsburg, PA, USA, IncrementParsing '04, pages 50-57.
* Nivre (2009) Joakim Nivre. 2009. Non-projective dependency parsing in expected linear time. In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_. Association for Computational Linguistics, Suntec, Singapore, pages 351-359.
* Petrov and McDonald (2012) Slav Petrov and Ryan McDonald. 2012. Overview of the 2012 shared task on parsing the web. _Notes of the First Workshop on Syntactic Analysis of Non-Canonical Language (SANCL)_ .
* Qian et al. (2015) Tao Qian, Yue Zhang, Meishan Zhang, Yafeng Ren, and Donghong Ji. 2015. A transition-based model for joint segmentation, pos-tagging and normalization. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Lisbon, Portugal, pages 1837-1846.
* Rasooli and Tetreault (2013) Mohammad Sadegh Rasooli and Joel Tetreault. 2013. Joint parsing and disfluency detection in linear time. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Seattle, Washington, USA, pages 124-129.
* Rasooli and Tetreault (2014) Mohammad Sadegh Rasooli and Joel Tetreault. 2014. Non-monotonic parsing of fluent umm i mean disfluent sentences. In _Proceedings of the 14th Conference of the European Chapter of the Association for 

Computational Linguistics, volume 2: Short Papers_. Association for Computational Linguistics, Gothenburg, Sweden, pages 48-53.
* Roark et al. (2006) Brian Roark, Mary Harper, Eugene Charniak, Bonnie Dorr, Mark Johnson, Jeremy Kahn, Yang Liu, Mari Ostendorf, John Hale, Anna Krasnyanskaya, Matthew Lease, Izhak Shafran, Matthew Snover, Robin Stewart, and Lisa Yung. 2006. Sparseval Evaluation metrics for parsing speech. In _Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06)_. European Language Resources Association (ELRA).
* Rozovskaya and Roth (2014) Alla Rozovskaya and Dan Roth. 2014. Building a state-of-the-art grammatical error correction system. _Transactions of the Association for Computational Linguistics_ 2:414-434.
* Sakaguchi et al. (2012) Keisuke Sakaguchi, Tomoya Mizumoto, Mamoru Komachi, and Yuji Matsumoto. 2012. Joint English spelling error correction and POS tagging for language learners writing. In _Proceedings of COLING 2012_. The COLING 2012 Organizing Committee, Mumbai, India, pages 2357-2374.
* Wu et al. (2015) Shuangzhi Wu, Dongdong Zhang, Ming Zhou, and Tiejun Zhao. 2015. Efficient disfluency detection with transition-based parsing. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Beijing, China, pages 495-503.
* Yamada and Matsumoto (2003) Hiroyasu Yamada and Yuji Matsumoto. 2003. Statistical dependency analysis with support vector machines. In _In Proceedings of IWPT_. pages 195-206.
* Yoshikawa et al. (2016) Masashi Yoshikawa, Hiroyuki Shindo, and Yuji Matsumoto. 2016. Joint transition-based dependency parsing and disfluency detection for automatic speech recognition texts. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, pages 1036-1041.