Venue: NAACL 2021

# WikiTalkEdit: A Dataset for modeling Editors' behaviors on Wikipedia

 Kokil Jaidka\({}^{1}\) Andrea Ceolin \({}^{2}\) Iknoor Singh\({}^{3}\)

Niyati Chhaya\({}^{4}\) Lyle H. Ungar\({}^{5}\)

###### Abstract

This study introduces and analyzes **WikiTalkEdit**, a dataset of conversations and edit histories from Wikipedia, for research in online cooperation and conversation modeling. The dataset comprises dialog triplets from the Wikipedia Talk pages, and editing actions on the corresponding articles being discussed. The exchanges occur between two turn-taking individuals and span all of Wikipedia. We show how the data supports the classic understanding of style matching, where positive emotion and the use of first-person pronouns predict a positive emotional change in a Wikipedia contributor. However, they do not predict editorial behavior. On the other hand, feedback invoking evidentiality and criticism, and references to Wikipedia's community norms, is more likely to persuade the contributor to perform edits but is less likely to lead to a positive emotion. We developed baseline classifiers trained on pre-trained RoBERTa features that can predict editorial change with an F\({}_{1}\) score of .54, as compared to an F\({}_{1}\) score of .66 for predicting emotional change. A diagnostic analysis of persisting errors is also provided. We conclude with possible applications and recommendations for future work. The dataset is publicly available for the research community at [https://github.com/kj2013/WikiTalkEdit/](https://github.com/kj2013/WikiTalkEdit/).

+
Footnote †: Code to replicate the data collection is available at [https://github.com/kj2013/WikiTalkEdi/](https://github.com/kj2013/WikiTalkEdi/).

+
Footnote †: Code to replicate the data collection is available at [https://github.com/kj2013/WikiTalkEdi/](https://github.com/kj2013/WikiTalkEdi/).

+
Footnote †: Code to replicate the data collection is available at [https://github.com/kj2013/WikiTalkEdi/](https://github.com/kj2013/WikiTalkEdi/).

## 1 Introduction

Dialogue is a language game of influence, action, and reaction that progresses in a turn-taking manner. Persuasion occurs through dialogue when a listener favorably evaluates the authority, claims, and evidentiality through the cues and arguments made by the speaker Krippendorff (1993); Schulte (1980); Durik et al. (2008).

Discussions on Wikipedia Talk pages can be useful for determining strategies that lead to an improvement of the article discussed, and for examining if they also lead to an amicable dialog exchange. Previous work Yang et al. (2016, 2017) has explored the role of editors and the types of edits made on Wikipedia, but have not related them to the ongoing conversation on the Wikipedia Talk pages.

We introduce the **WikiTalkEdit dataset**, a novel dataset for research in online collaboration. The dataset is a subset of the Wikipedia Talk Corpus available as of May 20181. It contains 12,882 dialogue triples with labels about editors' subsequent editorial (editing) behavior, and 19,632 triplets with labels corresponding to editors' emotion as manifested in their replies. Table 1 has examples from the dataset.2

Footnote 1: [https://figshare.com/articles/wikipedia_Talk_Corpus/4264973](https://figshare.com/articles/wikipedia_Talk_Corpus/4264973)

This new dataset enables various language and behavior modeling tasks. In general, the dataset is important for understanding linguistic coordination, online cooperation, style matching, and teamwork in online contexts. More specifically, it offers linguistic insights about the norms on Wikipedia, such as (i) the feedback which is associated with a positive emotion vs a positive editing action, (ii) identifying and characterizing successful editorial coordination Lerner and Lomi (2019), (iii) generating constructive suggestions based on a given Wikipedia edit, and (iv) identifying and resolving disagreements on Wikipedia before they go awry Zhang et al. (2018). In this study, we examine the first research problem. That is, we demonstrate how the dataset is helpful to compare and contrast the linguistic strategies that evoke favorable dialog responses from those evoking behavioral compliance.

## 2 Related Work

Conversational quality is largely the focus of a body of work modeling the formal Pavlick and Tetreault (2016), polite Niculae et al. (2015) and

toxic (Zhang et al., 2018) features of comments on Wikipedia, Reddit, and other online public forums. The labels in such a task are often subjective as they depend mostly on annotated or crowdsourced labels. On the other hand, gauging the impact of a conversation in terms of a reader's subsequent behavior is a rather different problem. A few studies have modeled the language of arguments to predict their upvotes Wei et al. (2016a, b); Habernal and Gurevych (2016); Tan et al. (2016). The best result reported by Habernal and Gurevych (2016) was an F\({}_{1}\) score of .35 for the task of predicting which of two arguments was better, using SVMs and bi-directional LSTMs. The study by Tan et al. (2016) reported an accuracy of 60% for predicting which argument was most likely to change the original poster's (OP's) point of view. Althoff et al. (2014) report an AUC of .67 on predicting the success on \(\sim\)5700 requests. Studies predicting users' stance Lin and Utz (2015); Sridhar et al. (2015) have done better, but do not usually factor in the feedback from a turn-taking partner, during a dialogic exchange. Furthermore, to the best of our knowledge, we did not find an equivalent study to measure the actual subsequent behavior of a conversation partner after a dialogic exchange on social media platforms, forums, or Wikipedia.

In recent years, computational linguistics has developed computational models of dialogic text that predict the emotional responses associated with any utterance. The findings suggest that interacting speakers generally reinforce each others' point of view Kramer et al. (2014); Rime (2007), use emotions to signal agreement, and mirror each other's textual cues Niculae et al. (2015). On the other hand, predicting behavioral responses is potentially a more challenging task for text modeling and prediction, and it is also less explored in the literature.

The existing research on online turn-taking behavior has focused on modeling emotional reactions, with little interest in predicting actual behavioral change. This research is discussed in more detail in the Supplementary Materials 3. For now, we contextualize the contributions of this dataset by demonstrating how it is applicable to address the following gaps in the scholarship:

Footnote 3: Available at [https://github.com/kjj2013/WikiTalkEEdit/](https://github.com/kjj2013/WikiTalkEEdit/)

* How well do language models trained on editorial feedback predict subsequent emotional and editorial change?
* What are the linguistic features of editorial feedback which predict emotional change in the person that initiates the discussion (henceforth, OP, original poster)?
* What are the linguistic features of editorial feedback which predict subsequent editorial behavior by the OP?

First, we report the predictive performance on predicting emotional and editorial behavior change from the linguistic features of the comments, using regression baselines and state-of-the-art deep learning models. Performance is evaluated as an F\({}_{1}\) score of predicted labels against the ground truth labels as implemented in _scikitlearn_. Then, we compare the linguistic features associated with emotional change with those associated with subsequent edits. Finally, we offer a diagnostic analysis of the prediction errors observed.

## 3 The WikiTalkEdit dataset

In this dataset, we describe how we collected our data from the Wikipedia Talk dataset and formulated a task around emotional and behavioral actions of an article's editors, who are taking turns in a conversation.

### Data generation process

After contributing to a Wikipedia article, the OP usually updates the Talk page with a summary of the edit. At this point, the OP may get zero or more responses, and they may respond to all, some, or

\begin{table}
\begin{tabular}{|p{99.6pt}|p{170.7pt}|} \hline
**Page** & **Multi replied** \\ \cline{2-3}  & OF: The title of this article should be “Fonctions Language Following”. That is due to one way, hundreds and in all I concentrate and on the website. [...] \\ Feinforming Camping Fellowship & E: Learned the article to Pointreprocessing Capping Fellowship accordingly. **On Thanked** \\ \hline \multirow{5}{*}{Doseid Livingstone} & OP: Wizard Like Vicents the source of the NLP’ Why did find Lake Youtson, and then a few years later touching for the source of the NLP’. [...] \\ \cline{2-3}  & CR: As a result, he found an automatic trend/highly produced half Lake Youtson until you said vice intermediate sentence. They were looking for the issue of Lake Vickents and hence like Nake, etc. But the tokenizer that Luke Youtson’s word into something tied, so the NE [..] \\ \cline{2-3}  & OR: Intersubjecting, thanked \\ \cline{2-3}  & OF: When Eric nick a format ”m pooling reasoning it is recommended countless of experimental counts of England per time dono that that does have the teaching problems probed here hit isn’t’...] \\ \cline{2-3}  & E: Is think it another cause one the 1974 countries can share it where they were when they become impossible. We could put another under these above, although others defined only 1. In contrast to the situation alphabets, co-occurrence and force the content of Re-P. **Of Be the identity words – learned and target node 1: "Imagewise the correct directories would make sense. Once run all done, I think mek-like.** \\ \hline \multirow{5}{*}{Electro Tem Composition} & OP: This is a paper I am beginning because of the trouble! It had in obtaining information about an instance Token Text Database via keyboard organ. I \\ \cline{2-3}  & E: From several the speeds play far away, it see how the attitude derivative. It is important for you to live your sources core VP/Villability, and allow other driver to a visibility threshold test (or VECO for compilers), to the ratio myself will feel if individuality cannot be asserted. Or the majority of information for this attack is my own research, with any phone displayed for vertillability. There is no other source in the world that I am aware of that his his information.[...] \\ \cline{2-3}  & \\ \end{tabular}
\end{table}
Table 1: Example talk triplets from the dataset.

 

none of them. To study the effect of editorial feedback, we defined a complete interaction between an OP and another Editor as a dialog triplet of the form \(OP\to Editor\to OP^{\prime}\).

Our dependent variables are the OP's reaction to an Editor's comment in terms of the 'emotional change' in their language and their 'editorial change' in terms of subsequent edits to the Wikipedia article.

First we downloaded the entire Wikipedia Talk Corpus available as of May 2018 and extracted 128,231 dialogue triplets. Next, we used the Wikimedia API to download the edits corresponding to each of the OP's comments in our dataset of triplets. In the following paragraphs, we further describe how we operationalized the labels for the dataset.

**Emotional change:** The emotional change label is the **signed** Euclidean distance between the positive and negative emotions of OP' and OP (see Figure 1). The positive and negative emotion measurements are calculated using the emotion dictionaries from the Linguistic Inquiry and Word Count (LIWC) Pennebaker et al. (2007). The assigned labels were manually examined by the authors for face validity.

A change over one standard deviation above the mean is coded as '1' and is a positive emotional change. A change under one standard deviation below the mean is coded as a '0' and is a negative emotional change. All other values are marked "null" as there is no evident change in emotion.

**Editorial change:** The edits, if any, performed by the OP to the article in the week following the Editor's feedback, are operationalized as a binary value ('1'=edit, '0'=no edit).

## 4 Dataset Analysis

In the following sections, we analyze what types of linguistic feedback from the Editor is effective at creating a positive emotional change or an editorial action by the OP.

In preliminary data explorations, we found no correlation between the Editor's politeness or status, and emotional or editorial change. We observed that the Editor's comments that are associated with positive comments (Mean = 273 characters, Mean Jaccard coefficient, JC = .16) are significantly shorter and have less overlap (content interplay) with the OP's comment than those associated with negative comments (Mean = 417 characters, Mean JC = .18). There was no substantial difference for editorial changes.

## 5 Predicting the response to Editor's Feedback

We examine the different linguistic features and discourse markers in predicting emotional and editorial change in the WikiTalkEdit dataset. Our independent variables comprise the linguistic features of the Editor's feedback, and the dependent variables are the OP's change in emotional and editorial behavior after receiving the feedback.

We used logistic regression and many deep learning implementations from the _pytorch-pretrained-bert_ package to predict both the emotional and editorial change of the user.

### Feature extraction

We represented the Editor's feedback as a normalized frequency distribution of the following feature sets:

- **General lexical features (500 features and 50 LDA topics):** The most frequent unigrams, and 50 topics modeled using Latent Dirichlet Allocation in Python's MALLET package, with \(\alpha\)=5.

- **Stylistic features (73 features):** These include cognitive features (discrepancy, past tense, present tense, work) and emotional features (positive emotion, negative emotion, reward) from LIWC Pennebaker et al. (2007). A politeness index was generated using Stanford's Politeness API Danescu-Niculescu-Mizil et al. (2013).

-**Syntactic features (4 features)**: The Stanford parser was used to generate dependency parses. Dependency parses were used to identify and categorize all the adjectival modifiers that occurred at least ten times in the data. We distinguished the

Figure 1: Calculation of OP’s emotional change as the signed two-dimensional Euclidean distance between OP and OP’.

 

first-, second-, and third-person pronouns. Finally, we created part-of-speech n-grams based on the dependency trees.

**- Social features (2 features):** We measured _content interplay_, the Jaccard coefficient of similarity between the unigrams of the Editor's feedback and the OP's first comment. The Editor's _status_ may pressure the OP to conform by performing edits; therefore, we quantified the Editor's experience in terms of their number of contributions to Wikipedia.

### Deep learning models

We also experimented with a variety of deep learning baselines:

* CNN: The CNN framework Kim (2014) involves applying convolutional filters followed by max-over-time pooling to the word vectors for a post.
* RCNN: The RCNN framework Lai et al. (2015), recurrent convolutional layers followed by max-pooling. A fully connected layer then follows it with a softmax for output.
* biLSTM: The word embeddings for all words in a post are fed to bidirectional LSTM, followed by a softmax layer for output Yang et al. (2016c).
* biLSTM-Attention: For each sentence, convolutional and max-over-time pooling layers are applied on the embeddings of its words. The resultant sentence representations are put through bi-LSTM with the attention mechanism Yang et al. (2016c).
* NeuralMT: Embeddings are fed into a bidirectional-GRU followed by a decoder with the attention mechanism Bahdanau et al. (2015).
* FastText: Word representations are averaged into a sentence representation, which is, in turn, fed to a linear classifier Joulin et al. (2017). A softmax function is used to compute the probability distribution over the predefined classes, and a cross-entropy loss is used for tuning. Hierarchical softmax is used to speed up the training process.
* Transformer: The architecture implemented was based on recent previous work Vaswani et al. (2017).
* OpenAI GPT: The Generative Pretrained Transformer implementation Radford (2018) with the original hyperparameter settings.
* BERT and RoBERTa: The pre-trained BERT model Devlin et al. (2018) and the Robustly optimized BERT model (RoBERTa) Liu et al. (2019), where BERT is retrained with more data and an improved methodology. Models were fine-tuned using the simple transformers library.
* XLNET: Finally, we evaluate the performance of XLNet Yang et al. (2019), which combines bidirectional learning with the state-of-the-art autoregressive model such as Transformer-XL.

In the case of CNN and BiLSTM based models, we used the referral hyper-parameters from the original implementation for all models4. For Neural MT, FastText, and Transformer based models, implementations by original authors are used. All the models were evaluated using 5-fold cross-validation with a split ratio of 80:20 for train and test set, respectively. In fine-tuning the RoBERTa model on editorial change, the model parameters included a learning rate of 9e\({}^{-6}\), 3 epochs, and train batch size of 8. For emotional change, model parameters include a learning rate of 1e\({}^{-5}\), 3 epochs, and a train batch size of 8. The maximum input sequence length was 128, which included 91% of all the inputs. The time taken was 6-8 minutes/epoch on Tesla k80, running on a Google Colab implementation. Five hyperparameter search trials were conducted with cross-validation. A manual tuning strategy was followed to identify the setting with the best performance.

Footnote 4: [https://tinyurl.com/brightmart](https://tinyurl.com/brightmart)

## 6 Results

We now examine the test-set performance of these models trained on a subset of the WikiTalkEdit dataset. The dataset for emotion analysis comprises the 15% of overall dataset where editorial feedback yielded a substantial positive or negative change in the emotion vector (i.e., the emotional change was above or below one standard deviation from the mean). Similarly, the dataset for editorial actions (edits performed) comprises the 10% of the conversations that started within 24 hours since an OP

\begin{table}
\begin{tabular}{|p{42.7pt}|p{42.7pt}|p{42.7pt}|p{42.7pt}|p{42.7pt}|p{42.7pt}|} \hline  & **Triplets** & **0s** & **1s** & **Pages** & **Users** \\ \hline
**Sentiment** & 19.632 & 7,286 & 12,346 & 19,299 & 12,531 \\ \hline
**Editorial** & 12,882 & 6,896 & 5,986 & 12,731 & 8,506 \\ \hline \end{tabular}
\end{table}
Table 2: Dataset statistics.

 

edited the page. A pairwise correlation found no relationship between emotional and editorial change (\(\rho\)= .01, p>.1). The dataset statistics are provided in Table 2.

### Predictive performance

**Baseline logistic regression models:** Table 3 shows that emotional change is more straightforward to predict than editorial change, and style provides marginally better predictive performance than content. The best performance was obtained using POS n-grams, with an F\({}_{1}\) score of .57 for predicting emotional change, and of .51 for predicting behavioral change. Unexpectedly, social features were not good predictors of emotional change.

**Deep learning models:** In comparison to the logistic regression baselines, the deep learning models in Table 4 offer a remarkable predictive advantage, especially for emotional change. The best performing deep learning classifier is trained on pre-trained RoBERTa features and reports an F\({}_{1}\) score of \(.66\) for emotional change and \(.54\) for editorial change.

### Error analysis

We observed instances of misclassification from the best logistic regression classifier and the XLNet model Yang et al. (2019). We have diagnosed the likely sources of errors in this section.

#### 6.2.1 False positives in emotional change prediction

We randomly selected an assortment of false positives predicted by a logistic regression classifier and by XLNet and have provided them in Table 55. First, we find that since the logistic regression methods rely heavily on stylistic features, the errors we identified seemed to occur when the style does not match the intended meaning:

Footnote 5: More examples of errors are provided in the Supplementary Materials.

* **Feedback about notationy and relevance**: In the first example in Table 5, we see that despite the polite feedback, the conversation was not resolved positively and resulted in negative responses.
* **Reverted edits**: Similarly, in conversations where the OP contest their reverted edits, the dialogue appears to regularly derail into further negative replies despite the civility of the Editor's feedback.

The XLNet model did not repeat these particular errors. Its errors, on the other hand, appear to be driven by fact-checks and questions:

* **Fact-checks**: In contradicting the OP with facts and personal opinions, a disagreement is sometimes implied but not obvious. The model predicts a positive emotional change, but the OP responds to the implication with a negative reaction.
* **Counter-questions**: When Editors asked questions of the OP, it appears likely that the OP would turn defensive, even if the response included facts.

#### 6.2.2 False positives in editorial change prediction

Table 6 shows the false positives in predicting editorial change. Starting with the errors from models trained on stylistic features, we observed that in general, the errors centered on:

* **Controversial topics:** The errors arising from logistic classifiers reflect ideological disagreements, often involving hot-button topics such as race and ethnicity. The OP is not likely to change their mind despite what might be a well-reasoned argument from the Editor.
* **Reverted edits:** Dialog around why edits were reverted, or content was removed are

\begin{table}
\begin{tabular}{|l|c|c|} \hline  & \multicolumn{1}{|c|}{**Sentiment change**} & \multicolumn{1}{|c|}{**Editorial change**} \\ \hline
**Model** & \multicolumn{1}{|c|}{**F\({}_{1}\)score**} \\ \hline CNN Kim (2014) & .45 & .38 \\ biLSTM & .57 & .38 \\ bilLSTM-Attention Yang et al. (2016c) & .59 & .43 \\ Neural MT Bahdanau et al. (2015) & .59 & .47 \\ RCNN Lai et al. (2015) & .61 & .36 \\ FastText Joulin et al. (2017) & .65 & .51 \\ \hline Transformer Vaswani et al. (2017) & .48 & .50 \\ OpenAI GPT Radford (2018) & .64 & .50 \\ BERT Devlin et al. (2018) & .65 & .52 \\ RoBERTa Liu et al. (2019) & .66 & **.54** \\ XLNet Yang et al. (2019) & .65 & .53 \\ \hline \end{tabular}
\end{table}
Table 4: Predictive performance with deep learning classifiers.

\begin{table}
\begin{tabular}{|p{56.9pt}|p{56.9pt}|p{56.9pt}|} \hline  & Sentiment change & Editorial change \\ \hline Features & \multicolumn{3}{|c|}{F\({}_{1}\)score} \\ \hline \multicolumn{3}{|c|}{General lexical features} \\ \hline Unigrams Topes & .54 & **.51** \\ \cline{2-3}  & .56 & **.51** \\ \hline \multicolumn{3}{|c|}{Sykinetic features} \\ \hline LIWC + Politeness & .55 & .48 \\ \hline \multicolumn{3}{|c|}{Syratic features} \\ \hline Dependency Parses & .51 & .46 \\ Adjectivid modifiers Personal pronouns & .46 & .42 \\ POS ngrams & **.57** & **.51** \\ \hline \multicolumn{3}{|c|}{Social features} \\ \hline Content Interplay + Status & .39 & .35 \\ \hline \end{tabular}
\end{table}
Table 3: Performance of Logistic Regression classifier on different features.

 

usually requests for greater clarity for documentation purposes, and are rarely followed up with edits to the page.

False positives in predicting editorial change by XLNet also appear to arise when feedback is nuanced. Aside from feedback that implicitly discourages further editing, similar to what was observed in Table 5, we also observed other types of feedback that leads to errors by the XLNet model:

* **Opinions:** Editorial feedback that uses opinions rather than facts to persuade the OP appears to lead to an edit rarely, and this was a common error observed among the predicted labels.
* **Mixed feedback:** The models also appear to get confused when the feedback included content from the page as a quote, and included suggestions but made no direct requests.

## 7 Linguistic insights

Based on the results in Table 3, in this section, we examine the stylistic, lexical, and topical features which best predict emotional and behavioral change. These findings offer us a way to examine whether emotional and editorial change are indeed different, and to compare the results against previous studies which have examined these problems in some capacity.

### Stylistic insights

Comparing the most predictive stylistic and content features suggests that emotional and editorial change have different predictors. Table 7 summarizes the most significant predictors of emotional change based on an ordinary least squares regression analysis. Positive feedback through words related to rewards and positive emotions typically predict a positive emotional change, besides the use of _stance_ words (the first person pronoun, \(I\)) and reference to past experiences (past tense). This finding is in line with the literature Zhang et al. (2018); Althoff et al. (2014). Conversely, excessive use of adjectival modifiers (e.g., comparative words or words used to emphasize quantity or impact) is associated with a negative emotional change.

The insights look very different for editorial change (Table 8). Second person pronouns and present tense, both of which occur in directed speech, are associated with editorial changes, in sharp contrast with the features that emerged in the analysis of emotional change. Aigned with this, the use of words related to criticism (discrepancy) and work is also among the significant predictors of editorial change. Among the parts of speech, comments about the content (NN, NNP) appear to reduce the likelihood of an editorial change. Except for superlative modifiers, style seems not to be relevant in this case.

These results support previous studies in showing that emotion and politeness do not always signal editorial change Hullett (2005); Althoff et al. (2014), as it is true for stylistic markers Durik et al. (2008), while direct requests Burke et al. (2007), assertiveness, evidentiality Chambliss and Garner (1996) and other content-based features usually perform better. No feature appeared to correlate with both emotional and editorial behavior. Further lexical insights are provided in the Supplementary Materials.6

Footnote 6: We further tested the effect of only positive or only negative features; we found that positive emotion is a better predictor of emotional change (F1=.42 vs. F1=.45) but not of editorial change (F1=.48 for both positive and negative features).

## 8 Insights from topics

We conducted a Benjamini Hochberg (BH)-corrected Pearson correlation of the topic features of comments by the Editor. We visualize it as a language confusion matrix introduced in recent work Jaidka et al. (2020) to compare the topics predictive of emotional vs. editorial change of the OP. The word clouds in Figure 2 show the correlation of LDA topics with emotional change on the X-axis, and the correlation with editorial change on the Y-axis. The grey bands depict zones where the topics do not have a statistically significant correlation with either emotional or editorial change. We have distinguished the themes related to content (e.g., _medicine, religion, and ethnicity_) by coloring them in red. The topics in black are related to Wikipedia's content guidelines (i.e., mentions of _NPOV, sources, cite, information_).7 These themes involve the neutrality (_neutral point of view, NPOV_), general importance (_notability_), and verifiability (_sources, evidence_) of information. Finally, the blue topics are meta-commentary centered around the elements in a Wikipedia article (mentions of _edit, page, title, section_).

Footnote 7: See [https://en.wikipedia.org/wiki/Wikipedia:Five_pillars](https://en.wikipedia.org/wiki/Wikipedia:Five_pillars)

Our analysis of the WikiTalkEdit dataset suggests that mentions of Wikipedia's guidelines are associated with a positive editorial change, but a 

negative emotional change. Suggestions based on evidence are associated with both, a positive editorial _and_ a positive emotional change. First, we look at the spread of the content-themed topics around the figure. Some of the topics related to religion (_god, church, christian_) and ethnicity (_israel, people, jewish, indian_) are associated with a negative emotional change (-.06 \(<r<\) -.02, p \(<\) .05). Content topics related to medical research and health inspire a negative emotional change but a positive editorial change (\(r=\) .05, p\(<\).05).

Next, we consider the meta-commentary about page structure (_page, title, move_ and _review, section, add_). We observe that these are associated with positive emotional changes (.06 \(<r<\) .10, p \(<\) .05), possibly because they offer concrete and minor suggestions. Those meta-commentary topics which directly request an edit or a review inspire editorial change (.03\(<\) r \(<\) .06, p \(<\) .05). Finally, topics related to the source, i.e., about Wikipedia's guidelines, generate a more nuanced reaction. Topics related to evidentiality (_source, news, evidence_) and notability (_notable, articles, deletion_) are the strongest predictors of negative emotion (-.18\(<\)r\(<\)

\begin{table}
\begin{tabular}{|p{49.7pt}||p{142.3pt}|p{142.3pt}|} \hline  & \multicolumn{2}{c|}{**Editorial change - false positives**} \\ \hline \multicolumn{2}{|c|}{**LIWC-based classifier**} & \multicolumn{1}{p{142.3pt}|}{**Page**} & \multicolumn{1}{p{142.3pt}|}{**XLNet**} \\ \hline
**Page** & **Talk triplet** & **Page** & **Talk triplet** \\ \hline \multirow{9}{*}{Arabic} & OP: The redirect to the disambiguation page is intended to assist the reader in locating [...] & OP: ==Consoles? == Consoles are non-portable video game systems, handhelds are portable ones. Should we change the uses of “Conse Games”to “Video Games” due to Mario Kart games on both types? [...] & OP: ==Consoles are non-portable videos game systems, handhelds are portable ones. Should we change the uses of “Conse Games”to “Video Games” due to Mario Kart games on both types? [...] & OP: ==Consoles are non-portable videos game systems, handhelds are portable ones. Should we change the uses of “Conse Games”to “Video Games” due to Mario Kart games on both types? [...] & OP: ==Drugtive are traffic— I have made some edits to improve the article. \\  & Ediror: "Rikipedia is not a dictionary. Adjectives are not supposed to be disambiguated in this way unless they are truly ambiguous. The noun has very clear prescedence over the rather nonstandard adjective. See talk:Arab [...] & Mario Kart games can be disambiguation. (Thus the name of the article we have no it, handheld game console.) Using just ”video game” create would instead create a new problem, because arcade game !the main thing we’re using to differentiate from, are also a kind of video games. \\  & OP: ADK, I admit that my own recent edit on this point was not the [...] & OP: ==Bamockburn == I’ve replaced a sentence on the Victorian origin of the myth with a short summary of Cooper’s research on Burnets.. [...] & Editor: Thanks. What really concerned me was [...] We need to cite/attribute this. It’s pretty rare to use bloggs, see WP:RS and WP:VERIFY. [...] & OPP: Concern shared. Thanks, we’re using the same hymnNuclear. \\ \cline{2-3}  & EMP: "The woman did not appear to be black at all (she looks slightly tamed, almost mediterranean) :Why do you think that being Black is about how a person looks? [...] & Knightly & Or: ==Bamockburn == I’ve replaced a sentence on the Victorian origin of the myth with a short summary of Cooper’s research on Burnets.. [...] \\ \cline{2-3}  & OP: "There is some merit in your argument, but race is a matter of genetics, as I see you say yourself. I think we would be a lot better off without racial [...] & Editor: Thanks. What really concerned me was [...] We need to cite/attribute this. It’s pretty rare to use bloggs, see WP:RS and WP:VERIFY. [...] & Editor: Thanks. What really concerned me was [...] We need to cite/attribute this. It’s pretty rare to use bloggs, see WP:RS and WP:VERIFY. [...] & CoTF: ==Bamockburn == I’ve replaced a sentence on the Victorian origin of the myth with a short summary of Cooper’s research on Burnets.. [...] \\ \cline{2-3}  & EMP: "Adao" & 

.10, p \(<\) .05) but they generally lead to editorial changes (.03 \(<\) r \(<\) .08, p \(<\) .05).

## 9 Discussion and Limitations

An exploration of the WikiTalkEdit dataset suggests that strategies that elicit a positive emotional change may not affect editorial behavior. Negative responses should not be the only yardstick to measure the successful outcome of a conversation. Editorial changes occur when Editors use interpersonal language in talking about evidentiality and notability. However, these strategies are also associated with a negative emotional change. Despite the apparent negative feedback, referencing norms and sources is a successful strategy to prompt behavioral compliance. In related work, social influence through mentioning community norms was more effective than the Editor's status at achieving compliance on Wikipedia; however, the latter was an important predictor in a similar modeling task on Reddit Althoff et al. (2014).

Although the findings would be correlational, there would be ways to establish cause and effect through a rigorous research design Zhang et al. (2018). In some cases, the measurements may be thrown off if the replies to feedback are appreciative, but include some negative emotion words. Secondly, inordinately long or short feedback confounds the classifiers, but we expect that improvements in accuracy can be achieved by using differential attention models that focus on the emotions expressed in the first few words in the

Figure 2: LDA topics correlated with emotional and editorial change. Topics are colored according to their theme; word size is proportional to word weight in the topic.

 

dialogic exchanges. Finally, we could encode the latent space with information about the type of editorial feedback [15], which would be helpful in predicting how the OP responds.

## 10 Conclusion and Future Applications

The WikiTalkEdit dataset offers insights that have important implications for understanding online disagreements and better supporting the Wikipedia community [32]. We recommend the use of the WikiTalkEdit dataset to model the dynamics of consensus among multiple contributors. Scholars can also use the WikiTalkEdit dataset to address issues of quality, retention, and loyalty in online communities. For instance, the insights could shed light on how new OPs can be retained as sustaining Wikipedia contributors [16]. Our exploratory analyses suggest that disagreements on Wikipedia arise over "errors": doubts that a given entry leaves no room for improvements. But errors serve a good faith purpose on Wikipedia by perpetuating participation and shared collective action [47]. The dataset would also be useful to understand how references are debated and interpreted as objective pieces of evidence [9].

**Acknowledgements:** Supported by a Nanyang Presidential Postdoctoral fellowship and Templeton Religion Trust, grant TRT-0048. Thanks to Dr. Nicholas Palomares for their early feedback.

## References

* Althoff et al. (2014) Tim Althoff, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2014. How to ask for a favor: A case study on the success of altruistic requests. In _Eighth International AAAI Conference on Web and Social Media (ICWSM 2015)_.
* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. _CoRR_, abs/1409.0473.
* Burke et al. (2007) Moira Burke, Elisabeth Joyce, Tackjin Kim, Vivek Anand, and Robert Kraut. 2007. Introductions and requests: Rhetorical strategies that elicit response in online communities. In _Communities and Technologies 2007_, pages 21-39. Springer.
* Chambliss and Garner (1996) Marilyn J Chambliss and Ruth Garner. 1996. Do adults change their minds after reading persuasive text? _Written Communication_, 13(3):291-313.
* Danescu-Niculescu-Mizil et al. (2013) Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jere Leskovec, and Christopher Potts. 2013. A computational approach to politeness with application to social factors. _arXiv preprint arXiv:1306.6078_.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. _CoRR_, abs/1810.04805.
* Durik et al. (2008) Amanda M Durik, M Anne Britt, Rebecca Reynolds, and Jennifer Storey. 2008. The effects of hedges in persuasive arguments: A nuanced analysis of language. _Journal of Language and Social Psychology_, 27(3):217-234.
* Habernal and Gurevych (2016) Ivan Habernal and Iryna Gurevych. 2016. What makes a convincing argument? empirical analysis and detecting attributes of convincingness in web argumentation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1214-1223.
* Hullett (2005) Craig R Hullett. 2005. The impact of mood on persuasion: A meta-analysis. _Communication Research_, 32(4):423-442.
* Jaidka et al. (2020) Kokil Jaidka, Salvatore Giorgi, H Andrew Schwartz, Margaret L Kern, Lyle H Ungar, and Johannes C Eichstaedt. 2020. Estimating geographic subjective well-being from twitter: A comparison of dictionary and data-driven language methods. _Proceedings of the National Academy of Sciences_, 117(19):10165-10171.
* Joulin et al. (2017) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 427-431. Association for Computational Linguistics.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1746-1751. Association for Computational Linguistics.
* Klein et al. (2019) Max Klein, Julia Kamin, and J Nathan Matias. 2019. 6 Ideas to Strengthen Wikipedia (s) with Citizen Behavioral Science. _citizensandtech.org/2019/11/research-summit-withwikimedians_.
* Kramer et al. (2014) Adam DI Kramer, Jamie E Guillory, and Jeffrey T Hancock. 2014. Experimental evidence of massive-scale emotional contagion through social networks. _Proceedings of the National Academy of Sciences_, 111(24):8788-8790.
* Kippendorff (1993) Klaus Krippendorff. 1993. Conversation or intellectual Imperialism in comparing communication theories. _Communication Theory_, 3(3):252.
* Lai et al. (2015) Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent convolutional neural networks for 

text classification. In _Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence_, AAAI'15, pages 2267-2273. AAAI Press.
* Lerner and Lomi (2019) Jurgen Lerner and Alessandro Lomi. 2019. The network structure of successful collaboration in wikipedia. In _Proceedings of the 52nd Hawaii International Conference on System Sciences_.
* Lin and Utz (2015) Ruoyun Lin and Sonja Utz. 2015. The emotional responses of browsing Facebook: Happiness, envy, and the role of tie strength. _Computers in human behavior_, 52:29-38.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. _CoRR_, abs/1907.11692.
* Luyt (2015) Brendan Luyt. 2015. Debating reliable sources: writing the history of the vietnam war on wikipedia. _Journal of documentation_.
* Niculae et al. (2015) Vlad Niculae, Srijan Kumar, Jordan Boyd-Graber, and Cristian Danescu-Niculescu-Mizil. 2015. Linguistic harbingers of betrayal: A case study on an online strategy game. _arXiv preprint arXiv:1506.04744_.
* Nunes (2011) Mark Nunes. 2011. _Error: glitch, noise, and jam in new media cultures_. Bloomsbury Publishing.
* Pavlick and Tetreault (2016) Ellie Pavlick and Joel Tetreault. 2016. An empirical analysis of formality in online communication. _Transactions of the Association for Computational Linguistics_, 4:61-74.
* Pennebaker et al. (2007) James W Pennebaker, Roger J Booth, and Martha E Francis. 2007. Linguistic inquiry and word count: LIWC.
* Radford (2018) Alec Radford. 2018. Improving language understanding by generative pre-training. _openai.com/blog/language-unsupervised_.
* Rime (2007) Bernard Rome. 2007. The social sharing of emotion as an interface between individual and collective processes in the construction of emotional climates. _Journal of Social Issues_, 63(2):307-322.
* Schulte (1980)Joachim Schulte. 1980. _Witgreenstein: an introduction_. SUNY Press.
* Sridhar et al. (2015) Dhanya Sridhar, James Foulds, Bert Huang, Lise Getoor, and Marilyn Walker. 2015. Joint models of disagreement and stance in online debate. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, volume 1, pages 116-125.
* Tan et al. (2016) Chenhao Tan, Vlad Nicolae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016. Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions. In _Proceedings of the 25th international conference on World Wide Web_, pages 613-624.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. _CoRR_, abs/1706.03762.
* Wei et al. (2016a) Zhongyu Wei, Yang Liu, and Yi Li. 2016a. Is this post persausive? Ranking argumentative comments in online forum. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_, volume 2, pages 195-200.
* Wei et al. (2016b) Zhongyu Wei, Yandi Xia, Chen Li, Yang Liu, Zachary Stallbohm, Yi Li, and Yang Jin. 2016b. A preliminary study of disputation behavior in online debating forum. In _Proceedings of the Third Workshop on Argument Mining (ArgMining2016)_, pages 166-171.
* Yang et al. (2016a) Diyi Yang, Aaron Halfaker, Robert Kraut, and Eduard Hovy. 2016a. Edit categories and editor role identification in wikipedia. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)_, pages 1295-1299.
* Yang et al. (2016b) Diyi Yang, Aaron Halfaker, Robert Kraut, and Eduard Hovy. 2016b. Who did what: Editor role identification in wikipedia. In _Tenth International AAAI Conference on Web and Social Media (ICWSM 2016)_.
* Yang et al. (2017) Diyi Yang, Aaron Halfaker, Robert Kraut, and Eduard Hovy. 2017. Identifying semantic edit intentions from revisions in wikipedia. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2000-2010.
* Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. _CoRR_, abs/1906.08237.
* Yang et al. (2016c) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J. Smola, and Eduard H. Hovy. 2016c. Hierarchical attention networks for document classification. _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, page 1480-1489.
* Zhang et al. (2018) Justine Zhang, Jonathan P Chang, Cristian Danescu-Niculescu-Mizil, Lucas Dixon, Yiqing Hua, Nithum Thain, and Dario Taraborelli. 2018. Conversations gone awry: Detecting early signs of conversational failure. _arXiv preprint arXiv:1805.05345_.