SlowGuess/NAACL_2021_OCR huggingface dataset:
The insights look very different for editorial change (Table 8). Second person pronouns and present tense, both of which occur in directed speech, are associated with editorial changes, in sharp contrast with the features that emerged in the analysis of emotional change. Aligned with this,
the use of words related to criticism (discrepancy) and work is also among the significant predictors of editorial change. Among the parts of speech, comments about the content (NN, NNP) appear to reduce the likelihood of an editorial change. Except for superlative modifiers, style seems not to be relevant in this case.
These results support previous studies in showing that emotion and politeness do not always signal editorial change (Hullett, 2005; Althoff et al., 2014), as it is true for stylistic markers (Durik et al., 2008), while direct requests (Burke et al., 2007), assertiveness, evidentiality (Chambliss and Garner, 1996) and other content-based features usually perform better. No feature appeared to correlate with both emotional and editorial behavior. Further lexical insights are provided in the Supplementary Materials.[6]
8 Insights from topics
We conducted a Benjamini Hochberg (BH)-corrected Pearson correlation of the topic features of comments by the Editor. We visualize it as a language confusion matrix introduced in recent work (Jaidka et al., 2020) to compare the topics predictive of emotional vs. editorial change of the OP. The word clouds in Figure 2 show the correlation of LDA topics with emotional change on the X-axis, and the correlation with editorial change on the Y-axis. The grey bands depict zones where the topics do not have a statistically significant correlation with either emotional or editorial change. We have distinguished the themes related to content (e.g., medicine, religion, and ethnicity) by coloring them in red. The topics in black are related to Wikipedia's content guidelines (i.e., mentions of NPOV, sources, cite, information). These themes involve the neutrality (neutral point of view, NPOV), general importance (notability), and verifiability (sources, evidence) of information. Finally, the blue topics are meta-commentary centered around the elements in a Wikipedia article (mentions of edit, page, title, section).
Our analysis of the WikiTalkEdit dataset suggests that mentions of Wikipedia's guidelines are associated with a positive editorial change, but a
<sup>6</sup>We further tested the effect of only positive or only negative features; we found that positive emotion is a better predictor of emotional change (F1=.42 vs. F1=.45) but not of editorial change (F1=.48 for both positive and negative features).
See https://en.wikipedia.org/wiki/Wikipedia:Five-pillars
Sentiment change - false positives
Page	Talk triplet	Page	Talk triplet
Kiev	OP: I am so confused! What is the policy regarding names of towns?[...]Editor: Danny, there's no real policy. If there is a name which is widely used for one very famous city (e.g. Paris, Rome, Athens), then the article with that name should be about that city. [...]OP': Actually, I have a problem with this. Yes, Paris, Rome, and Athens immediately bring to mind great European cities, but why should we play favorites?[...]	Jeff Ken-net	OP: ==Massive POV/CRUFT should be deleted== I recommend restoring my edits. This article is filled with POV and cruft. I did not think they would require community consensus but it is what it is. Also, why no mention of the psycho who Issaulted him (Fergal Downey)?'Editor: When I looked at your edits I was just over-whelmed by the scale of them. Obviously you've [...] What do you want to remove? What do you want to add? [...]OP': I guess sometimes I am too bold.
Moors	OP: Several days have gone by and [user] has still failed to give me a logical reason for his reversion. I changed the article back to my previous edit and made some [...]Editor: In other words, you have no consensus for the changes. I'm not surprised hasn't replied, since you've got further and further from any actual comment on the content of the article. I'm not even sure [...]OP': Don't revert due solely to no consensus any of Wikipedia's basic editing policies? This is the second time I have been reverted without a specific reason why.[...]	Religion in Swazi-land	OP: ==Improving the article== I have made some edits to improve the article.Editor: Can you please explain why you changed CIA statistics?'OP': ==Reverted page== I disagree with the edits by , I therefore reverted it to my last edit on July 15.
Table 5: Error diagnostics for predicting sentiment change. In this case, our models predicted a positive change ('1'), but the sentiment actually turned negative.
Editorial change - false positives
LIWC-based classifier	XLNet
Page	Talk triplet	Page	Talk triplet
Arabic	OP: The redirect to the disambiguation page is intended to assist the reader in locating [...]Editor :Wikipedia is not a dictionary. Adjectives are not supposed to be disamiguated in this way unless they are truly ambiguous. The noun has very clear presedence over the rather nonstandard adjective. See talk:Arab [...]OP' : I think you are doing a disservice to readers. I am not going to get in a revert war with you, but will ask for input from others.	Mario Kart	OP: == Consoles? == Consoles are non-portable video game systems, handhelds are portable ones. Should we change the uses of "Console Games" to "Video Games" due to Mario Kart games on both types? [...]Editor: Handhelds are really just a type of video game console. (Thus the name of the article we have on it, handheld game console.) Using just "video game" create would instead create a new problem, because arcade game | the main thing we're using to differentiate from, are also a kind of video games(OP': Ah, okay, I thought there was a big difference. Thanks for clearing this up! [...]
Oona King	OP :OK, I admit that my own recent edit on this point was not the [...]Editor: *The woman did not appear to be black at all (she looks slightly tanned, almost mediterranean) :Why do you think that being Black is about how a person looks? [...]OP' : There is some merit in your argument, but race is a matter of genetics, as I see you say yourself. I think we would be a lot better off without racial [...]	Knights Templar legends	OP: == Bannockburn == I've replaced a sentence on the Victorian origin of the myth with a short summary of Cooper's research on Burnes.[...]Editor: Thanks. What really concerned me was [...] We need to cite/attribute this. It's pretty rare to use blogs, see WP:RS and WP:VERIFY. [...]OP': Concern shared. Thanks, we're using the same hymn-sheet.
Table 6: Error diagnostics for Editorial change. In this case, our models predicted edits ('1'), but no edit was actually made.
negative emotional change. Suggestions based on evidence are associated with both, a positive editorial and a positive emotional change. First, we look at the spread of the content-themed topics around the figure. Some of the topics related to religion (god, church, christian) and ethnicity (israel, people, jewish, indian) are associated with a negative emotional change  $(-.06 < r < -.02, p < .05)$ . Content topics related to medical research and health inspire a negative emotional change but a positive editorial change  $(r = .05, p < .05)$ .
Next, we consider the meta-commentary about
page structure (page, title, move and review, section, add). We observe that these are associated with positive emotional changes (.06 < r < .10, p < .05), possibly because they offer concrete and minor suggestions. Those meta-commentary topics which directly request an edit or a review inspire editorial change (.03 < r < .06, p < .05). Finally, topics related to the source, i.e., about Wikipedia's guidelines, generate a more nuanced reaction. Topics related to evidentiality (source, news, evidence) and notability (notable, articles, deletion) are the strongest predictors of negative emotion (-.18 < r<
Sentiment change
Feature	Coefficients	Examples
First person pronouns	.08**	I, me
Past tense	.04**	ago, did, talked
Positive emotion	.06**	great, nice, sweet
Reward	.06**	take, prize, benefit
Comparative modifiers	-.08*	significant amount, in particular
Quantitative modifiers	-.06*	many articles, most cases, vast majority
JJ NN (adjective+noun)	-.04*	relevant description, rare possibility
IN VBZ (preposition+verb)	-.04*	that is, in position
VBZ JJ (verb+adjective)	-.03*	seems correct, is only
Table 7: Analysis of the features significantly associated with a sentiment change, with  $* \mathrm{p} < {10}^{-3}, *  * \mathrm{p} < {10}^{-6}$  .
Editorial change
Feature	Coefficients	Examples
Second person pronoun	.03*	you, your
Present tense	.01*	today, is, now
Work	.01*	job, work, source
Discrepancy	.01*	should, would
Superlative modifiers	.03*	great deal, great job, best place
IN (preposition)	-.04*	of, is, in, off, from
NNP (proper noun)	-.04*	axelboldt, gulag, wales
NN (noun)	-.04*	mistake, theory, problem
Table 8: Analysis of the features significantly associated with an editorial change, with  $* \mathrm{p} < {10}^{-3}, *  * \mathrm{p} < {10}^{-6}$  .
Figure 2: LDA topics correlated with emotional and editorial change. Topics are colored according to their theme; word size is proportional to word weight in the topic.
.10,  $\mathrm{p} < .05$  but they generally lead to editorial changes (.03 < r < .08, p < .05).
An exploration of the WikiTalkEdit dataset suggests that strategies that elicit a positive emotional change may not affect editorial behavior. Negative responses should not be the only yardstick to measure the successful outcome of a conversation. Editorial changes occur when Editors use interpersonal language in talking about evidentiality and notability. However, these strategies are also associated with a negative emotional change. Despite the apparent negative feedback, referencing norms and sources is a successful strategy to prompt behavioral compliance. In related work, social influence
through mentioning community norms was more effective than the Editor's status at achieving compliance on Wikipedia; however, the latter was an important predictor in a similar modeling task on Reddit (Althoff et al., 2014).
Although the findings would be correlational, there would be ways to establish cause and effect through a rigorous research design (Zhang et al., 2018). In some cases, the measurements may be thrown off if the replies to feedback are appreciative, but include some negative emotion words. Secondly, inordinately long or short feedback confounds the classifiers, but we expect that improvements in accuracy can be achieved by using differential attention models that focus on the emotions expressed in the first few words in the
dialogic exchanges. Finally, we could encode the latent space with information about the type of editorial feedback (Yang et al., 2017), which would be helpful in predicting how the OP responds.
10 Conclusion and Future Applications
The WikiTalkEdit dataset offers insights that have important implications for understanding online disagreements and better supporting the Wikipedia community (Klein et al., 2019). We recommend the use of the WikiTalkEdit dataset to model the dynamics of consensus among multiple contributors. Scholars can also use the WikiTalkEdit dataset to address issues of quality, retention, and loyalty in online communities. For instance, the insights could shed light on how new OPs can be retained as sustaining Wikipedia contributors (Yang et al., 2017). Our exploratory analyses suggest that disagreements on Wikipedia arise over "errors": doubts that a given entry leaves no room for improvements. But errors serve a good faith purpose on Wikipedia by perpetuating participation and shared collective action (Nunes, 2011). The dataset would also be useful to understand how references are debated and interpreted as objective pieces of evidence (Luyt, 2015).
Acknowledgements: Supported by a Nanyang Presidential Postdoctoral fellowship and Templeton Religion Trust, grant TRT-0048. Thanks to Dr. Nicholas Palomares for their early feedback.
Tim Althoff, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2014. How to ask for a favor: A case study on the success of altruistic requests. In *Eighth International AAAI Conference on Web and Social Media* (ICWSM 2015).
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473.
Moira Burke, Elisabeth Joyce, Tackjin Kim, Vivek Anand, and Robert Kraut. 2007. Introductions and requests: Rhetorical strategies that elicit response in online communities. In *Communities and Technologies* 2007, pages 21-39. Springer.
Marilyn J Chambliss and Ruth Garner. 1996. Do adults change their minds after reading persuasive text? Written Communication, 13(3):291-313.
Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A computational approach to politeness
with application to social factors. arXiv preprint arXiv:1306.6078.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
Amanda M Durik, M Anne Britt, Rebecca Reynolds, and Jennifer Storey. 2008. The effects of hedges in persuasive arguments: A nuanced analysis of language. Journal of Language and Social Psychology, 27(3):217-234.
Ivan Habernal and Iryna Gurevych. 2016. What makes a convincing argument? empirical analysis and detecting attributes of convincingness in web argumentation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1214-1223.
Craig R Hullett. 2005. The impact of mood on persuasion: A meta-analysis. Communication Research, 32(4):423-442.
Kokil Jaidka, Salvatore Giorgi, H Andrew Schwartz, Margaret L Kern, Lyle H Ungar, and Johannes C Eichstaedt. 2020. Estimating geographic subjective well-being from twitter: A comparison of dictionary and data-driven language methods. Proceedings of the National Academy of Sciences, 117(19):10165-10171.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427-431. Association for Computational Linguistics.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751. Association for Computational Linguistics.
Max Klein, Julia Kamin, and J Nathan Matias. 2019. 6 Ideas to Strengthen Wikipedia (s) with Citizen Behavioral Science. citizensandtech.org/2019/11/research-summit-with-wikimedians.
Adam DI Kramer, Jamie E Guillory, and Jeffrey T Hancock. 2014. Experimental evidence of massive-scale emotional contagion through social networks. Proceedings of the National Academy of Sciences, 111(24):8788-8790.
Klaus Krippendorff. 1993. Conversation or intellectual Imperialism in comparing communication theories. Communication Theory, 3(3):252.
Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent convolutional neural networks for
text classification. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI'15, pages 2267-2273. AAAI Press.
Jürgen Lerner and Alessandro Lomi. 2019. The network structure of successful collaboration in wikipedia. In Proceedings of the 52nd Hawaii International Conference on System Sciences.
Ruoyun Lin and Sonja Utz. 2015. The emotional responses of browsing Facebook: Happiness, envy, and the role of tie strength. Computers in human behavior, 52:29-38.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Brendan Luyt. 2015. Debating reliable sources: writing the history of the vietnam war on wikipedia. Journal of documentation.
Vlad Niculae, Srijan Kumar, Jordan Boyd-Graber, and Cristian Danescu-Niculescu-Mizil. 2015. Linguistic harbingers of betrayal: A case study on an online strategy game. arXiv preprint arXiv:1506.04744.
Mark Nunes. 2011. Error: glitch, noise, and jam in new media cultures. Bloomsbury Publishing.
Ellie Pavlick and Joel Tetreault. 2016. An empirical analysis of formality in online communication. Transactions of the Association for Computational Linguistics, 4:61-74.
James W Pennebaker, Roger J Booth, and Martha E Francis. 2007. Linguistic inquiry and word count: LIWC.
Alec Radford. 2018. Improving language understanding by generative pre-training. *openai.com/blog/language-unsupervised*.
Bernard Rimé. 2007. The social sharing of emotion as an interface between individual and collective processes in the construction of emotional climates. Journal of Social Issues, 63(2):307-322.
Joachim Schulte. 1980. Wittgenstein: an introduction. SUNY Press.
Dhanya Sridhar, James Foulds, Bert Huang, Lise Getoor, and Marilyn Walker. 2015. Joint models of disagreement and stance in online debate. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 116-125.
Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016. Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions. In Proceedings of the 25th international conference on World Wide Web, pages 613-624.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762.
Zhongyu Wei, Yang Liu, and Yi Li. 2016a. Is this post persuasive? Ranking argumentative comments in online forum. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, volume 2, pages 195-200.
Zhongyu Wei, Yandi Xia, Chen Li, Yang Liu, Zachary Stallbohm, Yi Li, and Yang Jin. 2016b. A preliminary study of disputation behavior in online debating forum. In Proceedings of the Third Workshop on Argument Mining (ArgMining2016), pages 166-171.
Diyi Yang, Aaron Halfaker, Robert Kraut, and Eduard Hovy. 2016a. Edit categories and editor role identification in wikipedia. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1295-1299.
Diyi Yang, Aaron Halfaker, Robert Kraut, and Eduard Hovy. 2016b. Who did what: Editor role identification in wikipedia. In Tenth International AAAI Conference on Web and Social Media (ICWSM 2016).
Diyi Yang, Aaron Halfaker, Robert Kraut, and Eduard Hovy. 2017. Identifying semantic edit intentions from revisions in wikipedia. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2000-2010.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J. Smola, and Eduard H. Hovy. 2016c. Hierarchical attention networks for document classification. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, page 1480-1489.
Justine Zhang, Jonathan P Chang, Cristian Danescu-Niculescu-Mizil, Lucas Dixon, Yiqing Hua, Nithum Thain, and Dario Taraborelli. 2018. Conversations gone awry: Detecting early signs of conversational failure. arXiv preprint arXiv:1805.05345.