Venue: EMNLP 2017

# Identifying Semantic Edit Intentions from Revisions in Wikipedia

 Diyi Yang\({}^{\clubsuit}\), Aaron Halfaker\({}^{\diamondsuit}\), Robert Kraut\({}^{\spadesuit}\), Eduard Hovy\({}^{\clubsuit}\)

\({}^{\clubsuit}\) Language Technologies Institute, Carnegie Mellon University {diyi,hovy}@cmu.edu

\({}^{\diamondsuit}\) Wikimedia Foundation ahalffaker@wikimedia.org

\({}^{\clubsuit}\) Human-Computer Interaction Institute, Carnegie Mellon University robert.kraut@cmu.edu

###### Abstract

Most studies on human editing focus merely on syntactic revision operations, failing to capture the intentions behind revision changes, which are essential for facilitating the single and collaborative writing process. In this work, we develop in collaboration with Wikipedia editors a 13-category taxonomy of the semantic intention behind edits in Wikipedia articles. Using labeled article edits, we build a computational classifier of intentions that achieved a micro-averaged F1 score of 0.621. We use this model to investigate edit intention effectiveness: how different types of edits predict the retention of newcomers and changes in the quality of articles, two key concerns for Wikipedia today. Our analysis shows that the types of edits that users make in their first session predict their subsequent survival as Wikipedia editors, and articles in different stages need different types of edits.

## 1 Introduction

Many online text production communities, including Wikipedia, maintain a history of revisions made by millions of participants. As Wikipedia statistics as of January 2017 show, English Wikipedia has 5.3 million articles with an average of 162.89 revisions per article, with revisions growing at a rate of about 2 revisions per second. This provides an amazing corpus for studying the types and effectiveness of revisions. Specifically, differences between revisions contain valuable information for modeling document quality or extracting users' expertise, and can additionally support various natural language processing (NLP) tasks such as sentence compression Yamangil and Nelken (2008), lexical simplification Yatskar et al. (2010), information retrieval Aji et al. (2010), textual entailment recognition Zanzotto and Pennacchiotti (2010), language bias detection Recasens et al. (2013), spelling errors and paraphrases Zesch (2012); Max and Wisniewski (2010).

To avoid building different approaches to extract the information needed by different NLP tasks Ferschke et al. (2013), a unified framework to recognize edits from revisions is needed. Prior research on revision editing primarily develop syntactic edit action categories, from which they try to understand the effects of edits on meaning Faigley and Witte (1981); Yang et al. (2016). For instance, Daxenberger and Gurevych (2012) categorized edits based on whether edits affect the text meaning, resulting in syntactic edit categories such as file deletion, reference modification, etc. However, simply understanding the syntactic revision operation types does not provide the information we seek: _why_ do editors do what they do? _how effective_ are their actions? For example, syntactic edit type taxonomies cannot tell the difference between simplifying a paragraph and maliciously damaging that paragraph, since both involve deleting a sentence.

In this work, we focus explicitly on revision intention. We introduce a fine-grained taxonomy of the reasons why an author in Wikipedia made an edit. Example edit intentions include copy editing, elaboration, verification, and simplification. Compared to taxonomies that either focus on low-level syntactic operations Faigley and Witte (1981) or that mix syntactic and semantic classes Daxenberger and Gurevych (2013), a clean higher-level semantic categorization enables us to easily identify textual meaning changes, and to connect revisions to "what happens in the mind of the revising author during the revision"

(Fitzgerald, 1987; Daxenberger, 2016). In order to capture the meaning behind edits, we worked with 13 Wikipedians to build a taxonomy that captured the meaning of an revision, which we term _edit intention_, and hand-labeled a corpus of 7,177 revisions with their edit intentions. We then developed an automated method to identify these edit intentions from differences between revisions of Wikipedia articles. To explore the utility of this taxonomy, we applied this model to better understand two important issues for Wikipedia: new editor retention and article quality. Specifically, we examined whether edit intentions in newcomers' first editing sessions predict their retention, and examined how edits with different intentions lead to changes in article quality. These analyses showed that specific types of editing work were positively correlated with newcomer survival and articles in different stages of development benefited differently from different types of edits.

## 2 Related Work

Wikipedia revision histories have been used for a wide range of NLP tasks (Yamangil and Nelken, 2008; Aji et al., 2010; Zanzotto and Pennacchiotti, 2010; Ganter and Strube, 2009; Nelken and Yamangil, 2008). For instance, Yatskar et al. (2010) used Wikipedia comments associated with revisions to collect relevant edits for sentence simplification. Max and Wisniewski (2010) constructed a corpus of rewritings that can be used for spelling errors and paraphrases (Zesch, 2012). Similarly, Zanzotto and Pennacchiotti (2010) used edits as training data for textual entailment recognition, and Recasens et al. (2013) analyzed real instances of human edits designed to remove bias from Wikipedia articles. Most of these work employed manually defined rules or filters to collect relevant edits to the NLP task at hand.

Towards analyzing revisions and developing unified revision taxonomies (Bronner and Monz, 2012; Liu and Ram, 2011), Fong and Biuk-Aghai (2010) built machine learning models to distinguish between factual and fluency edits in revision histories. Faigley and Witte (1981) made a distinction between changes that affect meaning, called _text-base changes_ and changes which do not affect meaning, called _surface changes_. The two categories are further divided into formal changes, meaning-preserving changes, microstructure changes and macro-structure changes. This taxonomy was later extended by Jones (2008) to take into account edit categories such as significant deletion, style, image insertion, revert, etc. Pfeil et al. (2006) proposed a 13-category taxonomy based on the data and performed manual annotation to compare cultural differences in the writing process in different versions of Wikipedia. Daxenberger and Gurevych (2013) introduced a finer-grained edit taxonomy, and performed multi-label classification to extract edit categories based on unparsed source text (Daxenberger and Gurevych, 2012). However, most taxonomies of edit categories contain only syntactic actions or a mixture of syntactic and semantic actions, failing to capturing the intention of revisions.

In terms of revision intentions, Zhang and Litman (2016) incorporated both argumentative writing features and surface changes from Faigley and Witte (1981) and constructed eight categories of revision _purposes_, such as claims/ideas, warrant/reasoning/backing, rebuttal/reservation, organization, clarify, etc. Tan and Lee (2014) used revisions to understand statement strength in academic writings. There are multiple works on the detection of specific subsets of revision intentions in Wikipedia, such as vandalism detection where the goal is to classify revisions as vandalized or non-vandalized (Harpalani et al., 2011; Adler et al., 2011) and language bias/neutral point of view detection (Recasens et al., 2013). Instead of recognizing a specific type of revision intention each time, our work aims at designing a systematic and comprehensive edit intention taxonomy to capture intentions behind textual changes.

Prior work also used edit types and intentions to better understand the process of collaborative writing, such as article quality improvement (Kittur and Kraut, 2008). For example, Liu and Ram (2011) found that Wikipedia article quality correlates with different types of contributors; similarly Yang et al. (2016) pointed out articles in different quality stages need different types of editors. However, there are few studies examining the specific types of edits that are predictive of article quality. Recent research shows that the number of active contributors in Wikipedia has been steadily declining since 2007, and Halfaker et al. (2012) suggested that the semi-automated rejection of new editors' contributions is a key cause, but they did not explore whether or not specific types of newcomers' work got rejected at different rates 

and how that affects retention. In this paper, we take advantage of this new taxonomy to explore correlations between edit intentions, newcomers' retention, and article quality.

## 3 Semantic Taxonomy of Edit Intentions

A _revision_ is created whenever an editor saves changes to a Wikipedia page. As one revision could contain multiple local changes, each revision can be labeled with one or more edit intentions, representing the purposes of why an editor made that change. Different from prior research Daxenberger2016,Yang2016, we do not distinguish between revisions and edits. Although an edit is a coherent local change and might belong to any edit categories, it cannot be used to represent the intentions of editors during the revision. For example, it might be difficult to recognize _Refactoring_ if only one single edit is present. Since relocation or reorganization might involve several changes in the article, looking at one might lose the whole picture and lead to information loss. Moreover, edit types simply extracted from an edit is inadequate in outlining the correct intentions, for instance, adding a sentence could be _Clarification_, _Elaboration_, or _Vandalism_.

### Taxonomy of Edit Intentions

Our semantic taxonomy of edit intentions builds on prior literature on collaborative writing FaigleyWite91,Fitzgerald187, research on document revision analyses BronnerMonz2012, studies on edit categories Daxenberger2012,FongBiuk-Aghai2010, and work on purpose/intention classification ZhangLitman2016. In order to

\begin{table}
\begin{tabular}{l|p{54.1pt}|c|c|c} \hline
* \\ \hline \hline Corpus Size & & 4,977 & 4,977 & 7,177 \\ \hline \end{tabular}
\end{table}
Table 1: A taxonomy of edit intentions in Wikipedia revisions, Cronbach’s \(\alpha\) agreement and the distributions of edit intention before and after corpus expansion. The percentage in each row represents what percentage of revisions are labeled with this edit intention. The percentages do not sum up to 100% because one revision could belong to multiple categories. The _After_ corpus is used for all our analyses.
 

ensure that our taxonomy captured the _intentions_ that Wikipedians would find meaningful, we set up discussions with a group of 12 interested editors on a Wikipedia project talk page, and iteratively refined our taxonomy based on their feedback. Our discussion with Wikipedia editors is in this page1. We also analyzed which intentions get more confused with which and used that to guide the refinement.

Footnote 1: [https://en.wikipedia.org/wiki/Wikipedia_talk:Labels/Edit_types/](https://en.wikipedia.org/wiki/Wikipedia_talk:Labels/Edit_types/) Taxonomy

We define a top level layer for the revision intention taxonomy: intentions that are common in general revisions: **General Revision Intentions**, and intentions that are specific in Wikipedia: **Wikipedia Specific Intentions**. This categorization leads to 13 distinct semantic intentions, and Table 1 provides detailed descriptions. Specifically, general revision intentions include: _Clarification_, _Copy Editing_, _Elaboration_, _Fact Update_, _Point of View_, _Refactoring_, _Simplification_ and _verification_, and can be applicable to other contexts. _Counter Vandalism_, _Disambiguation_, _Process_, _Vandalism_, and _Wikification_ are edit intentions related to Wikipeida. We also propose an _Other_ category, intended for edits that cannot be labeled using the above taxonomy.

As the first work to model intentions of revisions, our taxonomy distills and extends existing edit type taxonomies. For instance, our intentions of "elaboration" and "verification" are extensions of "evidence" type proposed by Zhang and Litman (2016), and a syntactic category of "information deletion" in Daxenberger and Gurevych (2013) could be an instance of our "vandalism" or "simplification" depending on the context.

### Corpus Construction

To construct a reliable, hand-coded dataset to serve as ground truth for automatic recognition of edit intentions, we employed four undergraduate students who had basic Wikipedia editing experience to label edits using our intention taxonomy, based on written annotation guidelines2 vetted by Wikipedia editors and provided examples3. Moreover, to expose annotators to more working knowledge of Wikipedia, we provided three one-hour training sessions where annotators were asked to label a small set of revisions (around 50 each time) and to discuss their disagreements until consensus.

Footnote 2: [http://www.cs.cmu.edu/~diyiy/data/edit_intention_annotation_doc.pdf](http://www.cs.cmu.edu/~diyiy/data/edit_intention_annotation_doc.pdf)

Footnote 3: [https://en.wikipedia.org/wiki/Wikipedia:Labels/Edit_types/Examples](https://en.wikipedia.org/wiki/Wikipedia:Labels/Edit_types/Examples)

We randomly sampled 5,000 revisions from Jan, 2016 to June 2016 from the recent changes table4 in the Wikipedia database. For each revision, we displayed the content difference5 before and after the change to annotators, via a labeling interface that we developed. Because an editor could make several different types of edits within a single revision, we asked four RAs to label each revision with one or more of the possible semantic intentions. We collected four valid annotations for 4,977 revisions. We used Cronbach's \(\alpha\), a measure of internal consistency, to evaluate agreement among the annotators. The overall agreement \(\alpha\) score was 0.782, indicating substantial agreement between different annotators; The rule of thumb 1993 suggests that Cronbach's alpha scores larger than 0.7 are considered as acceptable. The inter-annotator agreement per semantic intention is described in column \(\alpha\) in Table 1.

Footnote 4: [https://www.mediawiki.org/wiki/ManualReCentchanges_table](https://www.mediawiki.org/wiki/ManualReCentchanges_table)

Footnote 5: en.wikipedia.org/wiki/?diff=712140761

### Corpus Expansion

As shown in column _Before_ in Table 1, some types of edit intentions, such as _disambiguation_ and _clarification_, were very rare in the random-sample corpus. To address this under-representation problem, we used the text of editors' comments to expand the corpus by retrieving 200 more revisions for each edit intention except Vandalism and Counter-Vandalism, resulting in 2,200 revisions6. More precisely, as a common practice Zanzotto and Pennacchiotti (2010); Recasens et al. (2013), we utilized regular expressions to match the text from the comments, which editors often wrote when saving their revisions, to the edit intentions. For example, editors might be signalling that they were intending to fix problems of _Point of View_ when their comments contained keywords such as "npov" or "neutral". Even though the comments sometimes signal the editors' intents, they are not infallible, editors may fail to complete the comment field, may only label one of the multiple edit intentions for a single revision, or write comments that are inaccurate, irrelevant, or incomplete. Thus the first author annotated the 2,200 revisions from the expanded corpus and 

merged it with the randomly sampled corpus. The frequency of the edit intentions before and after the expansion is in Table 1. We used the majority voting to resolve the disagreement. That is, if at least 3 out of 4 annotators picked an intention for a revision, it will be selected as the ground-truth. The final corpus contains 5,777 revisions, and can be downloaded from here7.

Footnote 6: [https://github.com/journals/cai/2018](https://github.com/journals/cai/2018) 

which evaluates how many times the top ranked predicted intention is not in the set of true labels of the instance.

Results show that the Binary Relevance (BR) and MLKNN classifiers, which used all our constructed features, outperformed Random and Majority baselines. Moreover, the BR and MLKNN methods show relatively similar best performances. Although multiple studies have utilized revisions' comments as "groundtruth" to collect desired edits, the CMT method, which includes only comment features, is less accurate than either the BR or MLKNN models. Note that predicting 14-category semantic intentions is more challenging compared to classifying low-level syntactic actions, such as inserting an image [20].

## 5 Intentions, Survival and Quality

The automated measurement of edit intentions provides a general framework to analyze revisions and can facilitate a wide range of applications, such as collecting specific types of revisions [36, 87, 9] and outlining the evolution of author roles [10, 11]. In this section, we demonstrate two examples of how this intention taxonomy can be applied to better understand the success of online collaboration communities [36], specifically the process of these sites to retain new contributors and create innovative products. To this end, we first investigate what newcomers are intended for in their first sessions and whether their edit intentions can account for their survival in Wikipedia. We then examine how edits carrying on different intentions at distinct times in an article's history influence changes in its quality.

### How Edit Intentions Affect Survival

To explore newcomers' intentions during their first experience editing articles, we focus on users' first edit sessions in Wikipedia. Here, **Edit Session** is defined as a sequence of edits performed by a registered user with less than one hour's time gap between two adjacent edits [13]. We then compare edit intentions of newcomers who survive - **Survivors**, and newcomers who do not - **Non-survivors**. Here, newcomers are defined as surviving if they performed an edit at

\begin{table}
\begin{tabular}{c|l|c|c|c|c|c|c|c} \hline  & **Metric** & **Random** & **Majority** & **CMT** & **BR-** & **BR** & **MLKNN** & **RAKEL** \\ \hline \hline \multirow{4}{*}{**Example**} & Exact Match & 0.052 & 0.284 & 0.352 & 0.391 & 0.426 & **0.452** & 0.292 \\ \cline{2-9}  & Accuracy & 0.052 & 0.283 & 0.428 & 0.498 & 0.540 & **0.542** & 0.338 \\ \cline{2-9}  & Precision & 0.084 & 0.417 & 0.479 & 0.626 & 0.586 & **0.599** & 0.381 \\ \cline{2-9}  & Recall & 0.052 & 0.285 & 0.458 & 0.562 & **0.611** & 0.578 & 0.344 \\ \cline{2-9}  & F1 Score & 0.052 & 0.285 & 0.455 & 0.536 & **0.580** & 0.574 & 0.354 \\ \hline \hline \multirow{2}{*}{**Label**} & Macro F1 & 0.060 & 0.042 & 0.310 & 0.487 & **0.597** & 0.576 & 0.385 \\ \cline{2-9}  & Micro F1 & 0.074 & 0.370 & 0.528 & 0.583 & **0.621** & 0.613 & 0.441 \\ \hline \hline
**Ranking** & One Error & 0.920 & 0.583 & 0.415 & 0.400 & 0.358 & **0.320** & 0.434 \\ \hline \end{tabular}
\end{table}
Table 2: Performance comparison for predicting edit intentions from revisions. Best results are bold.

Figure 1: The relative frequency of each edit intention, and its F1 score provided by the **BR** model.

 

least two months after their first edit session.

#### 5.1.1 Intention Comparison

Among 100,000 randomly sampled Wikipedia users, 21,096 made revisions in the Main/Article namespace during their first editing session. Among these 4,407 were survivors (i.e., made an edit two months after registering) and 16,689 were non-survivors. We applied our edit intention model to 53,248 revisions in users' first sessions, and compared the percentages of different types of edit intentions between survivors and non-survivors, as shown in _Intention Dist_ column in Table 3. We also performed 1-way ANOVA to test whether survivors and non-survivors have the same mean for each edit intention. We observed that, survivors tend to do more copy-editing (\(\Delta_{+}\)=2.3%) and more wikification (\(\Delta_{+}\)=6.5%), while non-survivors seem to perform more simplification and vandalism, which might provide signals for detecting vandals.

#### 5.1.2 Revert Analysis

To explore the relationship between rejection of contributions and newcomer retention, we also visualized the revert ratios of different types of edit intentions for survivors and non-survivors in their first session. Here, **Revert** refers to whether an edit from the author was reverted or completely removed by another user, and we detect reverts using MediaWiki Reverts library11. We then measured the revert ratio for each edit intention by calculating the percentage of revisions belonging to a specific edit intention, among all reverted revisions in users' first sessions. As shown in the _Revert Ratio_ column in Table 3, in general, non-survivors get reverted more compared to survivors, across all edit intentions. Interestingly, non-survivors compared to survivors get reverted more when performing _Wikification_, _verification_ and _Refactoring_, suggesting that sophisticated types of work might not be suitable for beginners.

Footnote 11: [http://pythonhosted.org/mwreverts/](http://pythonhosted.org/mwreverts/)

#### 5.1.3 Newcomer Survival

As a further exploration of the relationship between edit intentions and newcomer survival, we performed a logistic regression using edits in survivors' and non-survivors' first sessions. To handle this imbalanced data (i.e., many more negative examples than positive examples in training), we performed majority-class under-sampling to make this dataset balanced. Similar to Halfaker et al. (2012), we controlled the number of revisions completed during the first session (a proxy for an editor's initial investment), and the number of revisions reverted in their first sessions. We

\begin{table}
\begin{tabular}{r|c|c} \hline Edit Intention & Survival & Quality Changes \\ \hline \hline clarification & 0.029 & 0.001 \\ \hline copy editing & 0.033 & 0.011\({}^{\dagger}\) \\ \hline counter vandalism & 0.004 & \(-\)0.020\({}^{\dagger}\) \\ \hline disambiguation & \(-\)0.003 & \(-\)0.006\({}^{\dagger}\) \\ \hline elaboration & \(-\)0.024 & 0.061\({}^{\dagger}\) \\ \hline fact update & \(-\)0.001 & 0.002 \\ \hline point of view & 0.041 & \(-\)0.003 \\ \hline process & 0.051\({}^{\dagger}\) & \(-\)0.024\({}^{\dagger}\) \\ \hline refactoring & \(-\)0.013 & 0.011\({}^{\dagger}\) \\ \hline simplification & \(-\)0.002 & \(-\)0.008\({}^{\dagger}\) \\ \hline vandalism & \(-\)0.211\({}^{\dagger}\) & \(-\)0.005\({}^{\dagger}\) \\ \hline verification & 0.047 & 0.068\({}^{\dagger}\) \\ \hline wikification & 0.099\({}^{\dagger}\) & \(-\)0.010\({}^{\dagger}\) \\ \hline \end{tabular}
\end{table}
Table 4: Regression coefficients of different edit intentions for predicting Newcomer **Survival** and Article **Quality Changes**. \({}^{\ddagger}\) means the coefficient is statistically significant (p<0.05)

\begin{table}
\begin{tabular}{r|c|c|c|c} \hline  & \multicolumn{2}{c|}{_Intention Dist_} & \multicolumn{2}{c}{_Revert Ratio_} \\ \cline{2-5}  & **NS** & **SS** & **NS** & **SS** \\ \hline \hline clarification & **0.2\%** & **0.4\%** & 0.1\% & 0.1\% \\ \hline copy editing & **12.1\%** & **14.4\%** & **6.9\%** & **3.8\%** \\ \hline counter vandalism & 0.1\% & 0.0\% & 0.1\% & 0.0\% \\ \hline disambiguation & 0.0\% & 0.0\% & 0.0\% & 0.0\% \\ \hline elaboration & 27.7\% & 26.5\% & **16.5\%** & **6.9\%** \\ \hline fact update & 4.2\% & 3.8\% & **3.4\%** & **1.7\%** \\ \hline point of view & **0.1\%** & **0.2\%** & 0.0\% & 0.1\% \\ \hline process & 2.0\% & 2.3\% & **1.9\%** & **0.7\%** \\ \hline refactoring & 1.1\% & 1.3\% & **0.9\%** & **0.5\%** \\ \hline simplification & **3.7\%** & **3.1\%** & **3.1\%** & **1.4\%** \\ \hline vandalism & **13.8\%** & **6.1\%** & **16.0\%** & **4.7\%** \\ \hline verification & 7.0\% & 7.4\% & **3.8\%** & **2.7\%** \\ \hline wikification & **25.8\%** & **32.3\%** & **14.0\%** & **6.9\%** \\ \hline \end{tabular}
\end{table}
Table 3: The edit intention distribution in the first sessions (Intention Dist) and the revert ratio comparison (Revert Ratio), among non-survivors (NS) and survivors (SS). The numbers are bolded if 1-way ANOVA tests for difference between two groups are significant, with p<0.05.

 

described the regression coefficients of statistically significant edit intentions in the **Survival** column of Table 4. This logistic model achieves an Accuracy of 60.98%, Recall of 58.30%, Precision of 78.08% and F1-score of 66.76%. Editing articles for the purposes of _Process_, _Verification_ and _Wikifaction_ significantly predict the survival of newcomers, while performing vandalism is a strong negative predictor for survival.

### How Intentions Affect Article Quality

Although there are over 5.5 million articles in the English Wikipedia, fewer than 0.2% have been evaluated by Wikipedians as good articles and around 92% have been evaluated as start or stub class articles, Wikipedia's two lowest quality categories. In this section, we examine how edits with different intentions at distinct times in an article's history influence changes in its quality.

This task is framed as a prediction task, i.e. using edits' intentions and a set of control variables to predict changes in article quality. We borrowed a Article Quality Prediction Dataset released in Yang et al. (2016), which consists of the quality ratings collected in January and June, 2015 of 151,452 articles. We collected 1,623,446 revisions made to these articles between January and June 2015, by randomly sampling 10% revisions that were made to these articles during that time periods. Specifically, the outcome _article quality change_ is calculated by subtracting the previous quality score from the end quality score. The control variables include the previous article quality score, the total number of edits, the total number of editors, the changed bytes to an article, and the total number of edits to the article talk page during the six months. To construct edit-intention predictors, we summed the number of edits for each edit intention during the six months divided by the total number of revisions in this article.

Results of the linear regression model, shown in **Quality Changes** column of Table 4, show that our constructed regression model is significantly predictive of article quality changes (\(R^{2}=0.225\)). The results show that, keeping all control variables fixed, more _Copy Editing_, _Elaboration_, _Refactoring_ and _Verification_ are positively associated with improvements in article quality; in contrast, _Vandalism_, _Counter Vandalsim_, _Disambiguation_, _Process_ and _Simplification_ predict declines in article quality. The first four of these edits types often occur with reducing the article content, removing or redirecting pages. Improper use of them might be detrimental to article quality.

To determine if the effect of edit intentions on quality changes depends upon the initial quality of the article, we added the interaction terms between the previous quality score and edit percentages of different intentions (e.g., clarification x previous quality), and visualized interaction effects in Figure 2. When examining the interaction terms in more detail: the negative slope of _copy editing_ (when prev=2) suggests that, as articles increase in quality, copy editing is needed less. We found similar trends for interactions between previous quality and _elaboration_ and _verification_, which are essential for articles in the starting stages. In contrast, the positive slopes for _simplification_, _wikification_ and _process_ suggest that, as articles increase in quality, simplifying articles' content, adding proper links or reorganizing their structure becomes more important. Overall, these results reveal that different types of edit intentions are needed at different quality stages of articles.

## 6 Discussion and Conclusion

In this work, we proposed 13 semantic intentions that motivate editors' revisions in English Wikipedia. Example edit intentions include copy editing, elaboration, simplification, etc. Based

Figure 2: Interaction effect of different levels of edit intentions and different levels of previous article quality (**prev**) on article quality changes. All variables are standardized. The Y-axis measures the predictive margins and X-axis refers to different standardized levels of edit intention.

 

in a labeled corpus of revisions, we developed machine-learning models to automatically identify these edit intentions. We then examine the relations between edit intentions, newcomers survival, and article quality improvement. We found that (1) survivors tend to do more copy editing and wikification; non-survivors seem to perform more vandalism and other sophisticated types of work, and the latter often gets reverted more; (2) Different types of contributions are needed by articles in different quality stages, with elaboration and verification are needed more for articles in the starting stages, and simplification and process become more important as article quality increases.

Our proposed edit intention taxonomy and the constructed corpus can facilitate a set of downstream NLP applications. First, classifiers based on this intention taxonomy can help retrieve large scale and high quality revisions around simplification, neutral point of view or copy editing, which provides amazing corpora for studying lexical simplification, language bias detection and paraphrases. Second, as we showed in Section 5.2, determining how different edit types influence changes in articles is of great use to better the causes of quality variance in collaborative writing, such as detecting quality flaws [3] and providing insights on which specific aspects of an article needs improvement and what type of work should be performed. The ability to identify the need for editing, and specifically the types of editing work required, can greatly assist not only collaborative writing but also individual improvement of text. Moreover, even though our edit taxonomy is for English Wikipedia, it can be applied to other language versions of Wikipedia. We are now deploying the same edit intention taxonomy for Italian Wikipedia, and plan to apply it to other low resourced languages in Wikipedia. Finally, beyond the context of Wikipedia, similar taxonomies can be designed for analyzing the collaboration and interaction happened in other online contexts such as academic writing (e.g., Google Docs or ShareLatex, etc).

## Acknowledgement

This research was supported in part by a grant from Google to Robert Kraut. The first author was supported by Carnegie Mellon Presidential Fellowship. The authors would like to thank our reviewers and Wikipedian for helpful feedback.

## References

* Adler et al. (2011) B Thomas Adler, Luca De Alfaro, Santiago M Mola-Velasco, Paolo Rosso, and Andrew G West. 2011. Wikipedia vandalism detection: Combining natural language, metadata, and reputation features. In _International Conference on Intelligent Text Processing and Computational Linguistics_, pages 277-288.
* Aji et al. (2010) Ablimit Aji, Yu Wang, Eugene Agichtein, and Evgeniy Gabrilovich. 2010. Using the past to score the present: Extending term weighting models through revision history analysis. In _Proceedings of the 19th ACM international conference on Information and knowledge management_, pages 629-638. ACM.
* Anderka et al. (2012) Maik Anderka, Benno Stein, and Nedim Lipka. 2012. Predicting quality flaws in user-generated content: the case of wikipedia. In _Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval_, pages 981-990. ACM.
* Arazy et al. (2015) Ofer Arazy, Felipe Ortega, Oded Nov, Lisa Yeo, and Adam Balila. 2015. Functional roles and career paths in wikipedia. In _Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing_, CSCW '15, pages 1092-1105.
* Bronner and Monz (2012) Amit Bronner and Christof Monz. 2012. User edits classification using document revision histories. In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_, EACI'12, pages 356-366.
* Cortina (1993) Jose M Cortina. 1993. What is coefficient alpha? an examination of theory and applications. _Journal of applied psychology_, 78(1):98.
* Daxenberger (2016) Johannes Daxenberger. 2016. _The Writing Process in Online Mass Collaboration: NLP-Supported Approaches to Analyzing Collaborative Revision and User Interaction_. Ph.D. thesis, Technische Universitat.
* Daxenberger and Gurevych (2012) Johannes Daxenberger and Iryna Gurevych. 2012. A corpus-based study of edit categories in featured and non-featured Wikipedia articles. In _Proceedings of COLING 2012_, pages 711-726, Mumbai, India. The COLING 2012 Organizing Committee.
* Daxenberger and Gurevych (2013) Johannes Daxenberger and Iryna Gurevych. 2013. Automatically classifying edit categories in Wikipedia revisions. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 578-589, Seattle, Washington, USA. Association for Computational Linguistics.
* Fajley and Witter (1981) Lester Faigley and Stephen Witter. 1981. Analyzing revision. _College composition and communication_, 32(4):400-414.
* Ferschke et al. (2013) Oliver Ferschke, Johannes Daxenberger, and Iryna Gurevych. 2013. A survey of nlp methods and 

resources for analyzing the collaborative writing process in wikipedia. In _The Peoples Web Meets NLP_, pages 121-160. Springer.
* Fitzgerald (1987) Jill Fitzgerald. 1987. Research on revision in writing. _Review of educational research_, 57(4):481-506.
* Fong and Biuk-Aghai (2010) Peter Kin-Fong Fong and Robert P. Biuk-Aghai. 2010. What did they do? deriving high-level edit histories in wikis. In _Proceedings of the 6th International Symposium on Wikis and Open Collaboration_, WikiSym '10, pages 2:1-2:10, New York, NY, USA. ACM.
* Ganter and Strube (2009) Viola Ganter and Michael Strube. 2009. Finding hedges by chasing weasels: Hedge detection using wikipedia tags and shallow linguistic features. In _Proceedings of the ACL-IJCNLP 2009 Conference Short Papers_, ACL Short '09, pages 173-176.
* Halfaker et al. (2012) Aaron Halfaker, R Stuart Geiger, Jonathan T Morgan, and John Riedl. 2012. The rise and decline of an open collaboration system: How wikipedia's reaction to popularity is causing its decline. _American Behavioral Scientist_, page 0002764212469365.
* Harpalani et al. (2011) Manoj Harpalani, Michael Hart, Sandesh Singh, Rob Johnson, and Yejin Choi. 2011. Language of vandalism: Improving wikipedia vandalism detection via stylometric analysis. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, pages 83-88, Portland, Oregon, USA. Association for Computational Linguistics.
* Jones (2008) John Jones. 2008. Patterns of revision in online writing a study of wikipedia's featured articles. _Written Communication_, 25(2):262-289.
* Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of ticks for efficient text classification. _arXiv preprint arXiv:1607.01759_.
* Kittur and Kraut (2008) Aniket Kittur and Robert E. Kraut. 2008. Harnessing the wisdom of crowds in wikipedia: Quality through coordination. In _Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work_, CSCW '08, pages 37-46.
* Kraut et al. (2010) Robert Kraut, Moira Burke, John Riedl, and P Resnick. 2010. Dealing with newcomers. _Evidencebased Social Design Mining the Social Sciences to Build Online Communities_, 1:42.
* Liu and Ram (2011) Jun Liu and Sudha Ram. 2011. Who does what: Collaboration patterns in the wikipedia and their impact on article quality. _ACM Trans. Manage. Inf. Syst._, 2(2):11:1-11:23.
* Max and Wisniewski (2010) Aurlien Max and Guillaume Wisniewski. 2010. Mining naturally-occurring corrections and paraphrases from wikipedias revision history. In _Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)_, Valletta, Malta, European Language Resources Association (ELRA).
* Nelken and Yamangil (2008) Rani Nelken and Elif Yamangil. 2008. Mining wikipedia's article revision history for training computational linguistics algorithms. In _Proceedings of the AAAI Workshop on Wikipedia and Artificial Intelligence: An Evolving Synergy_, pages 31-36.
* Pfeil et al. (2006) Ulrike Pfeil, Panayiotis Zaphiris, and Chee Siang Ang. 2006. Cultural differences in collaborative authoraging of wikipedia. _Journal of Computer-Mediated Communication_, 12(1):88-113.
* Potthast et al. (2013) Martin Potthast, Matthias Hagen, Tim Gollub, Martin Tippmann, Johannes Kiesel, Paolo Rosso, Efstathios Stamatatos, and Benno Stein. 2013. Overview of the 5th international competition on plagiarism detection. In _CLEF Conference on Multilingual and Multimodal Information Access Evaluation_, pages 301-331. CELCT.
* Recasens et al. (2013) Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic models for analyzing and detecting biased language. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1650-1659, Sofia, Bulgaria. Association for Computational Linguistics.
* Tan and Lee (2014) Chenhao Tan and Lillian Lee. 2014. A corpus of sentence-level revisions in academic writing: A step towards understanding statement strength in communication. In _Proceedings of ACL (short paper)_.
* Tsounakas et al. (2011) Grigorios Tsounakas, Elefterions Spyromitros-Xioufis, Jozef Vilcek, and Ioannis Vlahavas. 2011. Mulan: A java library for multi-label learning. _Journal of Machine Learning Research_, 12:2411-2414.
* Yamaguchi and Nelken (2008) Elif Yamaguchi and Rani Nelken. 2008. Mining wikipedia revision histories for improving sentence compression. In _Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers_, pages 137-140. Association for Computational Linguistics.
* Yang et al. (2016) Diyi Yang, Aaron Halfaker, Robert Kraut, and Eduard Hovy. 2016. Who did what: Editor role identification in wikipedia. In _Tenth International AAAI Conference on Web and Social Media_.
* Yatskar et al. (2010) Mark Yatskar, Bo Pang, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2010. For the sake of simplicity: Unsupervised extraction of lexical simplifications fromwikipedia. In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_, pages 365-368. Association for Computational Linguistics.
* Zanzotto and Pennacchiotti (2010) Fabio Massimo Zanzotto and Marco Pennacchiotti. 2010. Expanding textual entailment corpora from wikipedia using co-training. In _Proceedings of the COLING-Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources_, volume 128 

T. Zesch (2012)Measuring contextual fitness using error contexts extracted from the wikipedia revision history. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pp. 529-538. Cited by: SS4.
* F. Zhang and D. Litman (2016)Using context to predict the purpose of argumentative writing revisions. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, California, pp. 1424-1430. External Links: Link Cited by: SS4.
* X. Zhang, J. Zhao, and Y. LeCun (2015)Character-level convolutional networks for text classification. In Advances in neural information processing systems, pp. 649-657. Cited by: SS4.