Segment-Predict-Explain for Automatic Writing Feedback

The internet was envisioned as “a universal place of information” where humans can freely share knowledge [Berners-Lee 1997]. Increasingly, users rely on websites centered around specific communities or topics such as Q&A sites (e.g., Quora), forums (e.g., Reddit, Couchsurfing), reviews (e.g., Yelp, Amazon, AirBnB), and topic-oriented chat services (e.g., Mest [Unauthored 2016]). These services let users both share free-form text (e.g., review a product), and find what others have written (e.g., how do others like this product? ); the value of such services relies on their ability to present high quality user-generated content to other users. Unfortunately, unlike more structured forms of input such as dates or emails for which validation is straightforward, it is challenging to measure and ensure the quality of free-form text content. In fact, the desired characteristics may differ depending on the community: while being balanced and unbiased is important in product reviews, a forum for political discourse may encourage strong stances. Although there are numerous techniques to improve the quality of the text that is served, they are still limited in terms of scalability and effectiveness; there is a need for a low-cost scalable system that can be easily customized to new domains. Post-Submission approaches seek to identify high quality content after it has been written and submitted; by removing poor content [Spirin and Han 2012], finding high quality content algorithmically [Agichtein et al. 2008; Wang et al. 2013] or via user voting [Tang et al. 2013]. However, this assumes that the corpus of contributions is large enough that high quality content can be found for every entity, which is unlikely for less popular or niche websites and topics. Alternatively, Pre-Submission approaches seek to improve quality prior to submission. Indirect approaches include content guides [Yelp 2016b; Amazon 2016; Wikipedia 2016] and incentivization [Yelp 2016a; Bosu et al. 2013], but only a subset of users read through such guidelines or are active enough to benefit from incentives. Direct approaches that provide rapid feedback during the writing process can improve content quality [Kulik and Kulik 1988], for instance, by asking students to write feedback for their peers’ assignments [Kulkarni et al. 2015]. However, even when crowdsourced, such approaches can take on the order 20 minutes [Kulkarni et al. 2015]; nearly as long as the median web-browsing session (30 minutes) [Mozilla 2016]. In contrast, automated feedback systems are instantaneous, but traditionally limited to syntactic, rather than content quality, issues [Madnani and Cahill 2014; Microsoft 2016; Google 2016]. Recent automated systems have highlighted the power of using machine learning features to generate feedback. These techniques identify features of the user’s text that indicate low-quality, and translate those features into general feedback text for the entire document [Krause 2015b; Krause et al. 2016; Biran and McKeown 2014]. They have shown that simple identificaton procedures—looking for outlier features whose value differs from typical values in high quality text, or using features weights in linear models—can generate helpful feedback. Although promising, there are a number of potential directions to improve automated feedback. Psychology studies emphasize the value of giving feedback for specific text segments rather than for an entire document [Nelson and Schunn 2007], and writing quality models show the predictive benefits of accounting for multi-feature interactions [Weimer et al. 2007; Ghose and Ipeirotis 2011]. To the best of our knowledge, we are the first to integrate these findings into an automated writing feedback system and evaluate their effects. To this end, we have built Dialectic, which is a customizable automatic feedback generation system. It employs a Segment, Predict, Explain pattern that first splits the user’s text into coherent segments, predicts the quality of each segment, and generates explanations at the document and segment-level for how low-quality text could be improved. The system automates common tasks such as text segmentation, model training, and deployment, and comes with a