Why Should This Article Be Deleted? Transparent Stance Detection in Multilingual Wikipedia Editor Discussions
Authors: Lucie-Aimée Kaffee
Arnav Arora
Isabelle Augenstein
Venue: EMNLP 2023 Main (2023)
Keywords: Wikipedia
deletion discussion
stance detection
content moderation
policy
multilingual
TL;DR: Predicting editors' stance and reasoning in Wikipedia deletion discussions across three languages

Abstract: The moderation of content on online platforms is usually non-transparent. On Wikipedia, however, this discussion is carried out publicly and editors are encouraged to use the content moderation policies as explanations for making moderation decisions. Currently, only a few comments explicitly mention those policies -- 20% of the English ones, but as few as 2% of the German and Turkish comments.
To aid in this process of understanding how content is moderated, we construct a novel multilingual dataset of Wikipedia editor discussions along with their reasoning in three languages. The dataset contains the stances of the editors (keep, delete, merge, comment), along with the stated reason, and a content moderation policy, for each edit decision. We demonstrate that stance and corresponding reason (policy) can be predicted jointly with a high degree of accuracy, adding transparency to the decision-making process. We release both our joint prediction models and the multilingual content moderation dataset for further research on automated transparent content moderation.

Why Should This Article Be Deleted?
Transparent Stance Detection in Multilingual Wikipedia Editor Discussions
Lumée-Aimée Kaffee"
Arnav Arora
Isabelle Augenstein
Hasso Plattner Institute
University of Copenhagen
University of Copenhagen
Germany Lucie-aimes kaffee@hpi.de
Denmark
Denmark
aarédi Serddikkuu.. ku.dk dk
augensteinédi.. wtt.ediikkk.c.. ku.dk
Abstract
agreed on. It has previously been utilised, us- ing a wisdom-of-the-croode approach, for mod- cration in specific scenarios, for instance, fact- checking (Hardalov et al., 2022b). Moderator de- cisions, such as changing the status of a subreddit to private in protest (Porter, 2023), is typically a decision made in dialogue with the community and based their feedback and previously agreed upon policies for that community.
The moderation of content online platforms is usually non-tr trantranpprret. On Wikipedia, how- ever, this discussion carried out publicly and editors are encouraged to use the content moderation policies as explanations for making moderation decisions. Currently, only a few comments explicitly mention those policies 20% of the English ones, but as few as 2% of the German and Turkish comments. To aid in this process of understanding how content is moderated, we construct a novel multilingual dataset of Wikipedia editor discussions along with their reasoning in three languages. The dataset contains the stances of the editors (keep, delete, merge, comment), along with the stated reason, and content moderation policy, each edit decision. We demonstrate that stance and corresponding reason (policy) can be pre- dicted jointly with high degree of accuracy, adding transparency to the decision-making process. We release both our joint prediction models and the multilingual content modera- tion dataset for further research on automated transparent content moderation."
As stance detection is crucial task for under- standing the reasoning for content moderation deci- sions, we here propose a method to explain stances with existing moderation policics. Policies in gen- eral can take various shapes and forms (Robyn Ca- plan, 2018). For many online platforms, these poli- cies are hidden in Terms and Conditions documents and point to what is permissible or not on plat- form, typically mitigate potential harms (Arora et al., 2023). For others, they are explicitly stated for the community to refer to, where these policies should be included in the considerations when con- tent discussed (Gillespie, 2018). Communities on those platforms further have norms and policies of their own (Fiesler et al., 2018). For members of such communities, it is imperative to be able to comprehend why their content was deleted based on these policies. For the platforms themselves, highlighting such policies ing the trust and safety of the platform and also required by upcoming regulation (Kaminski and Urban, 2021). Finally, for the content moderators on those platforms, navigating through a large num- ber of legalese-heavy policies can be challenging and time-consuming, However, to the best of our knowledge, supporting content moderators with automated tools that jointly predict moderator's decision and the corresponding policy has not been explored in prior research.
1 Introduction
Moderators typically discuss content moderation decisions they want to take collectively. In those discussions, it is crucial to understand the stances of the discussion participants. As moderators of- ten refer to content moderation policies (Gillespie. 2018), these be proposed solution for the problem hand. There- fore, understanding the stance of moderators and how they come to a decision is crucial for com- munity members to ensure that the decisions are made based on the policies the community has
crucial for maintain-
justification for their
Both authors contributed equally this work Q
Cooee
com/ copenlu/
Such discussion process referring to policies exists, for example, on Wikipedia, a community- edited encyclopedia. Here, policies ensure the qual-
wikt wiki-stance stance
Dataset: @ https: / /hugg aage.cecco dato/attssss/////// copenlu/ weteenuuwwieee //wikiiiiiii.i. stance

---

ity of the written articles and that standards are kept. For example, the Manual of Writing describes how an article should be written, the Reliable sources policy ensures that enough and high-quality refer- ences are included in an article, and the Notability policy ensures that only topics notable enough for an encyclopedic entry are included on Wikipedia. Some form of these policies exists across different language Wikipedias, ensuring that the respective community standards in each language are clearly communicated and upheld.
Deletion of Beta Kappa Gamma
lopic
blatant advertising, Fails [WP:NOTE]] The whole article reads like an advertisement
[[delete]]
example
[WP:NOTEI]
policy montion
poliyy Mie
Figure I:
Example comment from the dataset
Kappa (CLS)
advertising. Fails.
The whole article reads like an advertisement <SEP>
an article does not live up to the communities' standards, editors on Wikipedia can recommend the article for deletion. These deletion discussions are form of content moderation process, in which ed- itors cite established policies to argue for the dele- tion of, e.g., group or post. Across all Wikipedia language versions, the process very similar: An article is suggested for deletion, Deletion Dis- cussion takes place in which the editors support or oppose the deletion the article, or propose to merge it with other articles or comment on the discussion. For an example comment in such a dele- tion discussion, see Figure 1. Eventually, an admin Wikipedia takes action using the input of the discussion. A variety of guidelines help editors make this ?
WilppeaarCviiityy
keep
WhiipedaNNotsbip
merge
Wkkpadi:Advertiing
delola
Wikipedssnnooittttee mil
comment
Figure Overview of the approach for policy predie- tion and stance detection
language Wikipedias.
developed
To this end, our contributions are as follows:
When discussing the deletion of articles, many editors refer to existing policies in Wikipedia, to base their arguments on these evidence documents. In discussions, referring to evidence documents is helpful, as it grounds arguments in more objec- tive points and previously agreed-upon standards (Xiao and Askin, 2014; Schneider et al., 2012). Automating the task of getting overview and proposing policies cite is crucial help admins make consistent decisions, and for users to under- stand how and based on which policies decisions are discussed. Currently, there limited work in the area of transparent content moderation, and none of it has considered policies in community discussions as explanations reasoning for stance. We therefore propose multi-task setup in which stance is detected alongside the prediction of a jus- tification in the form of a policy, as described in Figure 2. As different language Wikipedias have different sizes in terms of articles and accordingly deletion discussions, we further propose mul- tilingual setup, aligning policies across different
A transparent stance dataset in three lan- guages (English, German, and Turkish), anno- tated with stance as suggested by the editor, well as policies arguments for their stance, and multilingual version in which policies are aligned across languages;
Benchmarking of Transformer-biss based models on this multilingual dataset;
An approach which leverages multi-task setup to predict stance along with policy justification for the stance;
Empirically demonstrated improvements of this multi-task setup the single task setup for low-resource scenarios (Turkish).
Related Work
Policies for Content Moderation Online content moderation and, with it, the policies that moder- ators use are indispensable on the current online platforms (Langvardt, 2017: Gillespie, 2018). Kit- tur and Kraut (2010) describe that once a commu- nity grows, its implicit rules need to formalised
"See for example, deletion policy on English Wikipedia: ht attps:/ //w////.vv k ripedia org/wt //ii pidela Deletion, caciioonooliiy__eeeeeeeeeeee

---

into standards, guidelines, or policies. Social me- dia platforms, such as Facebook, have policies to aid content moderators in deleting or keeping posts of users. Conflict arises when those policies are applied non-transparently (Sablosky, 2021).
Transparent Stance Detection Stance detection well-studied task, with prior work covering dif- ferent ways of formalising it, considering variety of different targets and forms of text, as well as different domains (Hardalov et al., 2021, 2022a,b). Less well-studied, however, is explainable stance detection. Providing a reason for a stance pre- diction is important to make it comprchensible and transparent users why that decision was made (Draws et al., 2023). One approach make stance more explainable is have human- annotated rationales for stance detection in a train- ing dataset, which model can then learn to predict for unseen text (Jayaram and Allaway, 2021). An- other approach is to use a topic modeling approach to extract stance labels alongside textual justifica- tions for these labels. (Gómez-Suta et al., 2023) However, no existing approaches have leveraged policies to explain the stance labels predicted, de- spite these being natural explanations of stance content moderation discussions.
As content moderation is a crucial topic, there have been proposals to automate aspects of it. Risch and Krestel (2018) propose automated method to suggest the deletion of social media comments based on general guidelines, such as insults, discrimination, and defamation. They do however not analyse the discussion of content mod- erators themselves. Ribeiro et al. (2023) suggest that automated content moderation could increase adherence to community standards. In our paper, we focus on the policies used in Wikipedia. They are created and maintained by community mem- bers, as any Wikipedia page can be edited by any user. These policies play an important role in nego- tiating community standards and have a wide range functions (Butler et al., 2008). To the best of our knowledge, there is no prior work on automatically detecting policies in content moderator discussions justification for their decisions.
Transparent Stance Detection
In content moderation, obtaining an overview of the attitudes of different moderators towards deci- sion is crucial to reach a consensus, The reasoning of the proposed decisions by moderators is idcally grounded in agreed upon community standards. In Wikipedia, deletion discussions are a forum for ex- changes about whether an article should be deleted or kept on the platform. In our work, we aim to support the admins, who make the final decision on whether an article should be deleted or kept, by automating the process of getting an overview of the different attitudes. Given a comment on topic. such as the example comment in Figure 1, we aim to predict the stance of the user along with their rea- soning in form of a policy, as displayed in Figure 2. The task of transparent stance detection consists of two subtasks: 1) policy prediction, in which we predict the reasoning for a decision in the form of policy; and 2) stance detection, in which the stance of the comment is detected. Transparent stance detection aims to predict the policy and stance of a comment alongside each other. We explore these tasks in a multilingual manner, leveraging knowl- edge from high-resource language Wikipedias to improve task performance on lower resourced ones.
Deletion Discussion on Wikipedia Xiao and Askin (2014); Schneider et al. (2012) find that in deletion discussions, rationales stated by the editors are rooted in established policies. One of the im- portant aspects for editors contributing the dele- tion discussions is the familiarity with the norms of these discussions (Geiger and Ford, 2011), in which citing existing policies can be great help.
The deletion discussions have been analysed in an automated manner to understand how on- line communities communicate. Xiao and Sitaula (2018) detect sentiment in the deletion discussions English Wikipedia and find that positive senti- ments tend indicate an article being kept, and negative sentiments tend to indicate that the arti- cle will be deleted. Mayfield and Black 2b, model Wikipedia deletion discussions as group decision-making processes. To this end, they use stance detection to evaluate each user's attitude. They show the importance of understanding stance in the Wikipedia deletion discussions. However, they only work on subset of the deletion discus- sions in the English Wikipedia. Park al. (2015) propose a method to identify online user arguments in need of support, but do not take the next step of proposing supporting documents.
Policy Prediction Wikipedia deletion discus- sions, rationales for opinions should be rooted the policies on Wikipedia, and mostly are, even if

---

only implicitly (Xiao and Askin, 2014; Schneider al., 2012). Referring the policy documents di- rectly in discussions can be helpful for other users to understand the reasoning easily. Still, a large part the comments do not directly refer to a pol- icy on Wikipedia (see Section 4). For our task of transparent stance detection, the policy document can be justification for the decision (stance) the moderator proposes in their comment. Therefore, we predict the relevant Wikipedia policy given an editor's comment. This could be extended to com- ments that do not yet refer to a policy and support editors and moderators to gain an easy overview over opinions and their rationales. In our paper, we leverage the fact that the policies are used as justi- fication to explain the stance of the user, by using it for transparent stance detection (see Section 3).
Language
train
test
dev
#pol
en
372,033
43,776
21,961
94
de
7,320
862
455
48
684
202
44
33
Table 1: Dataset split statistics
Label
#instances
en
de
tr
delete
279063
4805
433
keep
108273
3394
252
comment
30974
395
224
merge 19460
43
21
Table 2: Distribution of stance labels in the dataset
icy prediction and stance detection as part of a larger framework, thereby creating a transparent stance detection setup. In this transparent stance setup, given comment and topic, predict the stance alongside the policy, justifying the stance. In content moderation, this setup can be used understand and summarise the attitudes of different moderators and clarify which community standards or policies form the basis of moderation decision.
Stance Detection Stance Detection has histori- cally been used for understanding diseussions at
large. allows us to understand an individual's position towards a topic. We define stance along the commonly adopted definition the field of natural language processing, as the expression of a speaker's 's standpoint and judgement towards given proposition (Biber and Finegan, 1988). In deletion discussions, the stances of users given votes to decide whether article should be kept or deleted, i.e. the proposition is the deletion of given article, and the standpoint is expressed through keep, delete, merge, or comment sugges- tions (which we use as stance labels) that the au- thor argues for in their comment. Any editor can contribute to these discussions. Automating the detection of stances in moderators' comments can be helpful to content moderation large, as it al- lows moderators to get high-level overview of the opinions voiced a discussion. Prior work has suggested the same for detecting the stances of news articles so that fact-checkers can quickly get a quick overview and take an informed judgement about the veracity of an article on a topic (Pomer- leau and Rao, 2017). Based on user's 's comment and the name of the article discussed, we predict the stance of the comment towards the deletion or retention of the article.
Multilingual Transparent Stance Detection Different Wikipedia languages drastically vary in size in terms of the number of articles as well editors. Each of the different Wikipedia language communities creates policies so as to have stan- dards to point new editors to. Across Wikipedias, cultural norms and standards may differ (Hara et al., 2010), but there is still high overlap the guidelines they write. As such, different Wikipedia language versions can benefit from each other terms of identifying new guidelines to add or old ones to adapt. This is particularly true for lower resoureed Wikipedias, in which there are fewer ed- itors maintain the communities' policies and fill gaps when new policy should be created.
4 Dataset
We create three new datasets, containing annota- tions of stances and policies in articles for deletion discussions in the English, German, and Turkish Wikipedias. The datasets cover the time span from 2005 (2006 for Turkish) 2022. Each instance in the dataset consists of the title of the page that is dis- cussed for deletion as the topic, the stance towards
Transparent Stance Detection For users of platform, is crucial understand why content moderation decisions are made. Our paper con- tributes to the field of stance detection in content moderation by jointly considering the tasks pol-

---

the topic, i.e deletion of the article, the comment of the user arguing for or against deletion, and the pol- icy they cite to justify their argument. An example comment, shown in Figure 1, is "[[delete]] blatant advertising. Fails [[WP:NOTE]]. The whole arti- cle reads like an advertisement", where delete is the stance of the editor and the mentioned policy is Notability. More examples from the dataset can be found in the Appendix in Table 5.
glish with basic understanding of Turkish. The number of policies in our dataset varies consider- ably by language, see Table 1. As the majority of comments refer to only one policy, we keep only one policy per comment by selecting the first pol- icy mentioned. We further remove all mentions of policies from the comments using regular expres- sions, which often breaks grammaticality of the sentence but is necessary to prevent leakage of la- bel information. For future work, we also release a unperturbed version of the dataset, with the policy mentions intact. The stance labels (keep, delete, merge, and comment), can be expressed in different forms or spelled differently. We manually identify the different ways the labels might be expressed and aggregate them into the four standard labels. In the Appendix, Table 2, we list all versions of each of the labels across the three languages considered. To maintain privacy, and since this information is not relevant to our experiments, we remove user- names from the comments using regular expres- sions. The dataset is split in train/test/dev, where the split for English and German is 80%//5%/ but due to the low number in comments in Turkish, we decided to alter the split for Turkish to have at least 200 test examples. We use this dataset for the task of policy prediction and stance detection. as well as transparent stance, which we predict stance alongside policy, see Section 6.
4.1 Dataset Creation
To create the dataset, we identify the article dele- tion discussion archive pages for the English, Ger- man, and Turkish Wikipedia respectively, and re- trieve all deletion discussions in the considered time frame through the respective MediaWiki APIs?. From those pages, we select comments which mention a Wikipedia page, identified by the prefix [ [WPP or [ [Wikipedia:. We find that these generally refer to policies, or policy abbre- viations such as WP:NOTE in our example. If the policy abbreviations link to a policy page, the Wiki- media API resolves them and returns the actual policy or Wikipedia page title. For each of the three languages, retrieve the full policy page through the Wikimedia API, manually select the policies that are actual policy pages, and discard other Wikipedia pages, such as articles. We further discard all policies that are mentioned infrequently (100 in English, 10 for German, and 2 for Turk- ish, due to the varying data set sizes) across all comments in the respective language deletion dis- cussions. The final policies include, for example, the notability criteria in the English Wikipedia*, in German Relevanzkriterrenn,, in Turkish kayda degerlik To collapse sub-polices with the same or similar meaning, or subcategories one policy into the main policy, we merge them based on the link of the sub-policy to the main policy in the pol- icy page text, e.g., notability criteria for specific article types such Wikipedia:Notabiiin (music) policy.
Label Analysis We sampled subset of 100 ran- domly selected comments from the English test dataset for a preliminary human annotation study on this subset, for both stance and policy prediction. One annotator familiar with the English Wikipedia read the comments as present in the dataset, i.e., where stance and policy are removed, and then, for each comment, selected the matching stance and policy from the list of 94 English Wikipedia poli- cies. The accuracy scores between our annotation and the labels (which were self annotated by the Wikipedia editors) are 88% and 55% for stance and policy prediction respectively. We attribute the rela- tively lower score for policy prediction to the large number of labels (94), i.e., policies, with random prediction baseline of 1%. These policies also at times overlap with each other, e.g.. Wikipedia: Lists and Categoriess, lists, and navigation templates. Further, the comments are at times quite short, making challenging for the annotator to select the correct policy.
were merged into the Wikipedia: This was done manually based on the original as well as machine translated versions of the policy texts by an annotator proficient in German and En-
English Wikipedia API, German Wikipedia API, Turkish Wikipedia API
WLkipsddia:NNotabbiiit
"htrps://tr.w/kklpedd

---

4.2 Multilinguul Dataset
makes it important to ensure that models do not overfit when predicting policies. Comments, af- ter removing policies and other information as de- scribed above, have on average 248.3 (English), 342.3 (German), and 381.7 (Turkish) characters. There is a high variance between the languages re- garding how often policies are referred to in the discussions: 21.4% of English comments refer to policy, however only 2.7% of German, and 2.2% of Turkish comments refer to a policy in the ob- served time Trame.This explains the small size of the dataset for these languages, but also shows the need for automating the references to policies, as proposed in Section 3.
Since Wikipedia policies are connected across languages by links, we can leverage these cross- language links to reuse the mentions of the same policies in other languages' comments for lan- guages with lower resources. Non-English lan- guage Wikipedias may have fewer articles and therefore fewer discussions overall, but also, cru- cially for this paper, the use of policies might be limited as fewer policies exist. Aligning and pre- dicting policies across languages has the advantage that a cross-lingual model can implicitly learn fur- ther relationships between policies, and thereby achieve an improved performance for target lan- guages at test time. In future work, this might enable Wikipedias with fewer policy pages to auto- matically find gaps in which policies already exist- ing in other language Wikipedias they might want establish.
5 Experimental Setup
We consider four setups for our evaluation of indi- vidual and joint prediction of stance and policy: i) Single task where we consider the tasks outlined in Section 3 individually while only using the training set for that language; ii) Multi-task where we do joint prediction of both the tasks, while only using the dataset for the corresponding language; iii) Sin- gle task Multilingual, where we train on the multi- lingual dataset for all languages but test on a single language for one task; and iv) Multi-task multilin- gual, where we train on the multilingual dataset and test on single language, predicting both stance and policy jointly. We use three types of Trans- former models for our experiments, BERT (Devlin et al., 2019), mBERT (Devlin al., 2019), and XLM-R (Conneau et al., 2020). For each of the sctups, we run a learning rate sweep between [5c-4, 5e-5, 5e-6], taking the best model from each. We train the models with a batch size of 8, optimizing for accuracy on the development set during policy prediction and macro-averaged F1 on the develop- ment set during stance detection joint prediction. For the single language experiments, we use the language specific version of BERT, mBERT, and XLM-R, whereas for the multilingual experiments, we only use the multilingual models mBERT and XLM-R. The models were run for 5 epochs for the English and multilingual setups and 500 epochs for the smaller German and Turkish datasets. To avoid overfitting, we use carly stopping mecha- nism based on the aforementioned evaluation met- ric with a patience of 5. We report macro-averaged Fl scores for the task of stance detection as it is 4- way classification task with labels skewed towards the delete label. For policy prediction, however,
We then create a dataset, which concatenates Turkish, German, and English for training and val- idation based on the datasets introduced in Sec- tion 4.1. The comments in the test data are the same in the original test data for the three lan- guages, i.e., we provide a test dataset for each of the three languages. The stance detection labels are the same for all languages and can thus easily be mapped to one another. For policies, we create a super set across policies present in all 3 languages. In this super set of policies, we kept the policies intact that only exist in one language, and merged the policies that exist across two or all three lan- guages. To merge the policies, we use the fact that the policy pages link to each other on Wikipedia (inter-language links), in the same way different language Wikipedia articles on the same topic link to each other. When we merge the policies, we refer to the English title in the other languages. This super set of policies finally contains 116 poli- cies across three languages by linking total of 63 English and German policies to English. Hence, when performing policy prediction for Turkish, model can leverage the presence of those policies in English or German.
4.3 Dataset Statistics
The number of mentioned policies is heavily skewed towards the notability policy in each lan- guage, with 56% of comments mentioning the no- tability policy in English, 45% of comments in German, and 59% of comments in Turkish, which

---

single task
multi-task
multiling, single task
multiling, multi-task
en
de
tr
en
de
tr
en
de
tr
en
dc
tr
Random
0.20
0.19
0.17
Majority
0.19
0.18
0.15
lang_ BERT
0.80
0.51
0.48
0.80
0.45
0.54
XLMR 0.79
0.48
0.48
0.79
0.40
0.50
0.78
0.43
0.51
0.77
0.30
0.31
mBERT
0.78
0.50
0.41
0.77
0.37
0.45
0.79
0.41
0.48
0.76
0.38
0.47
Table 3: Macro-averaged Fl scores for stance detection, for the English, German, and Turkish datasets for different setups; single task, multi-task, and both setups for the multilingual dataset. For the monolingual models (lang_BERT) BERT), BERT, pre-trained on English, German, and Turkish data respectively. The best result across all setups for each language is underlined.
single task
multi-task
multiling. single task multiling. multi-tasé
en
de
en
de
tr
en
de
tr
en
de
Random
0.01
0.02
0.06
Majority
0.55
0.45
0.62
Ing BERT
0.76
0.72
0.69
0.72
0.61
0.68
XLMR
0.77
0.68
0.63
0.68
0.53
0.63
0.76
0.67
0.74
0.63
0.46
0.60
BBERT
0.75
0.75
0.66
0.67
0.56
0.65
0.75
0.68
0.70
0.66
0.51
0.60
Table Accuracy scores for policy prediction, for the English, German, and Turkish datasets for different setups; single task, multi-task, and both setups for the multilingual dataset. For the monolingual models (lang BERT), we use BERT, pre-trained on English, German, and Turkish data respectively. The best result across all setups for each language underlined.
the number of labels are substantially higher, mak- ing per-label examples scarce, especially for the smaller datasets. Hence, in this scenario, we report accuracy scores.
In this setting, both tasks are Iearnt jointly, allowing the model to leverage important label information across the tasks. In order to further enhance this sharing, we freeze the first layers of the encoder for each model. Moreover, stabilise the losses and balance the learning between the two tasks, we alternate between the losses in the ratio 3:1 for stance detection and policy prediction every step. Empirically, we find that this allows for the models' losses to converge for both tasks without the models overfitting.
Baselines To ground the results, we provide a Random baseline and Majority Class baseline for the single task setups. The former is calculated by assigning a random label from the label set to each test instance while the latter is calculated based on distribution of the labels in the test set.
Single task setup For the single task setup, use the standard fine-tuning procedure for the mod- els. We take [CLS] representation of the com- bined topic and comment text and fine-tune a se- quence classification head to make the predictions for stance and policy respectively.
Multilingual setup Since the German and Turk- ish Wikipedias are smaller than the English one, the datasets reflecting their deletion discussions contain fewer instances as well. To leverage shared learning of the task across languages, in this setup, we train the models on joint set of comments from the English, German, and Turkish datasets and tested single language test set. While stance labels are common (with translated labels) across the three languages, policies across each language Wikipedia differ, making it challenging to consolidate them to a single set of labels. To
Multi-task setup To make joint prediction, multi-task learning framework (MTL) with hard parameter sharing (Caruana, 1993) shown in Figure This is inspired by prior work which successfully employs MTL for joint classification and explanation generation (Atanasova et al., 2020).

---

circumvent this, we manually align the Turkish and German policies with the English policies. As poli- cies can differ across languages, there not an exact match for all policies. However, largely the policies do align across languages, see Section 4.2.
to single-task ones, see multi-task results in Table 4 and 3. For Turkish, the performance improves com- pared to the single task setup for stance detection. This demonstrates that identifying the relevant con- tent policy for a comment can also be beneficial for the predicting the stance of a comment, especially for an extreme low resource setting. For German, however, the policy prediction task suffers a drop in performance in this setting.
6 Results
In the following, we present the results for the experiments we conducted in order transparently detect a stance of a users comment, i.c., detect stance alongside the policy used as a justification for the stance.
6.3 Multilingual learning
In the experiments on the joint dataset of all lan- guages (Tables 3 and 4), under the multilingual single-task column, we find similar results as for the multi-task setting. In the very low resource scenario of Turkish, we see an improvement in per- formance across both stance detection and policy prediction. However, for German, there is per- formance drop across both tasks. The results for English remain consistent in this setting.
6.1 Single Task
For the tasks of policy prediction and stance detec- tion, individually trained and tested, we find that the models perform reasonably well compared to the majority and random baselines for both tasks. The results for the policy prediction task can be found in Table 4, and the results for the stance de- tection task in Table 3, under the single task column. Between the languages, we can see the impact of the different dataset sizes, especially for the stance detection task. Here, all three models trained on the smaller German and Turkish datasets perform substantially worse compared to ones trained on the English set. However, policy prediction is task where models for all three languages achieve comparable results, suggesting that seeing more instances does not necessarily help in being able to predict the relevant policy for a comment, where the number of labels is much larger. We analyse the most common bi-grams using LIME, and find that the most salient features learned are largely differ- ent lexicalisation of the labels, e.g., not enough for the delete class (see Appendix B.1). For the mono- lingual English BERT model comment most com- monly confused with delete, based on its confusion matrix(sea Appendix B.2). The overall scores are promising and suggest that the tasks of stance de- tection and policy prediction on the deletion dis- cussion comments are feasible to automate.
For the multilingual multi-task setup, the com- bined superset of aligned policies is 1 116, higher than policy labels for any one language, making this setup more challenging and not directly com- parable to the other single task and multi-task set- tings for policy prediction. The scores for policy prediction paint a similar picture, with a drop in performance compared to the other settings. For stance as well, contrary to prior results in the litera- ture, where cross-lingee lingual learning has substamtially improved performance on the task (Hardalov et al.. 2022a), we find that learning from high resource languages does not aid in our multi-task setting.
7 Discussion and Conclusion
Making content moderation decisions more trans- parent is crucial for users to understand why de- cisions were ultimately made and to understand how they were discussed. Prior work has shown that explanations for content removal utilised in content moderation. Adding them leads to more user engagement, reducing the odds of fu- ture post removals (Jhaver et al., 2019). Moreover. a mandate of providing users with an explanation for why their post or content was deleted is also included in upcoming regulation (Kaminski and Urban, 2021). Thus, making content moderation more transparent and explainable is beneficial both online platforms and end users.
under-
6.2 Multi-task
To support content moderators and give users in- sights into both the stances of editors and relevant content policies in discussion, we propose pre- dict stances and policies jointly by combing the two tasks in a multi-task model setup (Section 5). While the setting is more challenging, we find that multi-task models can achieve comparable scores
To aid in this moderation, we need improved NLP methods for stance detection. However, re-

---

sources for NLP across languages are heavily skewed (Joshi et al., 2020). This is especially the case for stance detection where multilingual re- sources are scarce (Hardalov et al., 2022b). In an effort to address these issues, we create a multi- lingual dataset (consisting of three languages: En- glish, German, and Turkish) based on Wikipedia delction discussions, annotated with the stance of the discussion participant and the policy they re- fer to their comment as justification. Given a comment in a content moderation discussion, we predict the stance of the moderator alongside jus- tification in the form of policy.
stance detection should be applied and tested on more platforms to encourage more transparent and comprehensible content moderation. As policies for content moderation differ across platforms, our model not able to directly predict other platforms' policies in zero-shot setting. However, the model could be fine-tuned on another platforms' policies and potentially leverage the policy-content relation- ship observed Wikipedia We believe that the general framework that we propose as well the challenges faced in defining the problem as clas- sification task has implications beyond Wikipedia, to most platforms where norms or policies are ex- plicitly defined. We would also look to further in- vestigate the arguments used within the comments, along with the policies mentioned, to increase the transparency and potentially model performance across the tasks.
For an end-user of Wikipedia, the joint predic- tion setup proposed here is particularly insightful as they get high level overview of the votes for or against deletion of an article along with the policy cited as justification for the stance. For Wikipedia editors themselves, such setup can help establish good practices of citing Wikipedia content policies in their comments potential candidates provided by the model. The setup can also help make speedy decision through a hird's-eye view of the discussion with preservation of some of the nuance beyond just keep or delete labels. Our results show that it is possible to learn the stance and policy pre- diction tasks jointly, aiding in the transparency of automatic content moderation models. For Turk- ish, he multi-task setup outperforms the monolin- gual setup for stance, demonstrating that leverag- ing the information across the two tasks can bene- fit their predictions for low-resource settings. As Wikipedias across languages vary in size in terms articles and therefore discussion pages, we also experiment with multilingual setup to support Wikipedia languages with fewer data resources. To this end, we align policies across languages where possible. We find that for Turkish, the smallest dataset of our experiments, this multilingual train- ing can improve performance, especially for the policy prediction task. As there are fewer policy pages overall for Turkish and German, such a setup could in the future also be used to predict policies missing in one language that exist already in an- other Wikipedia language. Community norms vary and it is important to allow for their diversity to exist while still making use of information from norms of other communities. Our setup allows for such learning while still supporting the editors in making informed decisions based on the editor discussions. In the future, the task of transparent
Acknowledgements
This research was partially funded by a DFF Sapere Aude research leader grant under grant agreement No 0171-00034B, as well as by the Pioneer Centre for AI, DNRF grant number PI.
Limitations
One natural limitation of our approach is the fact that not all data of deletion discussions can be lever- aged as not all comments refer to policies (see Sec- tion 4). This severely limits the size of our dataset. Further, in the multilingual setup, we might predict policies for, e.g., Turkish or German, which are matching the comment's 's content but do not exist in the language Wikipedia's 's policies yet.
Ethics Statement
As the dataset is real data, taken from Wikipedia deletion discussions, it is important to handle it with care; there might be sensitive content the data and one should not use it to identify individ- ual editors. The latter we addressed by removing usernames wherever possible. For more details on the dataset, we provide the datasheet for datasets, see Appendix C.
References
Arna Arora, Preslav Nakov, Momchil Hardalov. Sheikh Muhammad Sarwar, Vibha Nayak, Yoan Dinkov, Dimitrina Zlatkova, Kyle Dent, Ameya Bhatawdekar, Guillaume Bouchard, and Isabelle Au- genstein. 2023. Detecting harmful content on online

---

platforms: What platforms need vs. where research efforts go. ACM Computing Surveys.
R. Stuart Geiger and Heather Ford. 2011. Participation in wikipedia's 's article deletion processes. In Proceed- ings of the 7th International Symposium on Wikis and Open Collaboration, 2011, Mountain View, CA, USA, October 3-5, 201 pages 201-202. ACM.
Pepa Atanasova, Jakob Grue Simonsen, Christina Li- oma, and Isabelle Augenstein. 2020. Generating fact checking explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7352-7364, Online. Association for Computational Linguistics.
Tarleton Gillespie. 2018. Custodians of the Internet: Platforms, content moderation, and the hidden deci- sions that shape social media. Yale University Press.
Douglas Biber and Edward Finegan. 1988. Adver- bial stance types english. Discourse Processes,
Manuela Gómez-Sur Sutee,SSta, Julián David Echeverry Correa, and José A. Soto Mejía. 2023. Stance detection in tweets: A topic modeling approach supporting ex- plainability. Expert Syst. Appl., 214: 19046.
(1()):
Brian Butler, Elisabeth Joyce, and Jacqueline Pike. 2008. Don't now, but we' ve created a bureaucracy: the nature and roles of policies and rules in wikipedia. In Proceedings of the SIGCHI conference on human factors in computing systems, pages 1101-1110.
Noriko Hara, Pnina Shachaf, and Khe Foon Hew. 2010. Cross-cultural analysis of the wikipedia community. J. Assoc. Inf. Sei. Technol., 61(10):209-21100
Momchil Hardalov, Amav Arora, Preslav Nakov, and Isabelle Augenstein. 2021. Cross-domain label- adaptive stance detection. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 9011 9011-9028, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Rich Caruana. 1993. Multitask learning: A knowledge- based source of inductive bias. In Proceedings of the Tenth International Conference on International Conference on Machine Learning, ICML''9, page 41-48, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Out, Luke 7ettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 8440- 8451, Online. Association for Computational Lin- guistics.
Momchil Hardalov, Arnv Arora, Preslav Nakov, and Isabelle Augenstein. 2022a. Few-shot cross-lingual stance detection with sentiment-bused pre-training. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10)) 10729-1077.
Momchil Hardalov, Amnv Arora, Preslav Nakov, and Isabelle Augenstein. 2022b. survey on stance de- tection for mis- and disinformation identification. In Findings of the Association for Computational Lin- guisties: NAACL 2022, pages 1259-1277, Seattle, United States. Association for Computational Lin- guistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
Sahil Jayaram and Emily Allaway. 2021. Human tionales as attribution priors for explainable stance detection. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 7-11 November, 2021, pages 5540- 5554. Association for Computational Linguistics.
Tim Draws, Karthikeyan Natesan Ramamurthy, Ioana Baldini, Amit Dhurandhar, Inkit Padhi, Benjamin Timmermans, and Nava Tintarev. 2023. Explainable cross-topic stance detection for search results. In Proceedings of the 2023 Conference on Human In- formation Interaction and Retrieval, CHIIR 2023, Austin, TX, USA, March 19-23, 2023, pages 221-235. ACM.
Shagun Jhaver, Amy Bruckman, and Eric Gilbert, 2019. Does transparency in moderation really matter? user behavior after content removal explanations on reddit. Proc. ACM Hum. -Comput Interact. 3(CSCW).
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282 6293, Online. Association for Computational Linguistics.
Casey Fiesler, Jialun Jiang, Joshua McCann, Kyle Frye, and Jed Brubaker. 2018. Reddit rules! characterizing an ecosystem of governance. Proceedings of the International AAAI Conference on Web and Social Media, 12(1).
Timnit Gebru, Jamie Morgenstern, Briana Vec- chione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III and Kate Crawford. 2021. Datasheets for datasets. Communications ACM, 64( (222699...
Margot E. Kaminski and Jennifer M. Urban. 2021. The right to contest ai. Columbia Law Review. 121(7)1157-2244.

---

Aniket Kittur and Robert E Kraut. 2010. Beyond wikipedia: coordination and conflict in online pro- duction groups. In Proceedings of the 2010 ACM conference on Computer supported cooperative work, pages 215-224.
Jodi Schneider, Alexandre Passant, and Stefan Decker, 2012. Deletion discussions in wikipedia: decision factors and outcomes. In Proceedings of the Fighth Annual International Symposium on Wikis and Open Collaboration, WikiSym 2012, Austria, August 27- 29, 2012, page 17. ACM.
Kyle Langvardt. 2017. Regulating online content mod- cration. Geo. LJ, 106:1353.
Lu Xiao and Nicole Askin. 2014. What influences line deliberation? A wikipedia study. Assoc. Inf. Sci. Technol. 65(5):98--110,
Elijah Mayfield and Alan Black. 2019a. Stance classifi- cation, outcome prediction, and impact assessment: NLP tasks for studying group decision-making. In Proceedings of the Third Workshop on Natural Lan- guage Processing and Computational Social Science, pages 65-77, Minneapolis, Minnesota. Association for Computational Linguistics.
Lu Xiao and Niraj Sitaula. 2018. Sentiments in wikipedia articles for deletion discussions. In Trans- forming Digital Worlds 13th International Confer- ence, iConference 2018, Sheffield, UK. March 25-28, 2018, Proceedings, volume 10766 of Lecture Notes in Computer Science, pages 81-86. Springer.
Elijah Mayfield and Alan W. Black. 2019b. Analyzing wikipedia deletion debates with group decision- making forecast model. Proc. ACM Hum. Comput. Interact., (CCSCC):00::12200 206::26.
A Dataset Creation
As the decision of each moderator is written by hand, labels with the same meaning were merged into one of the four main labels by language. Com- ments without a decision label and comments which were ambiguously labelled were discarded. In Table 2, we detail which strings were merged into which labels.
Joonsuk Park, Arzoo Katiyar, and Bishan Yang. 2015. Conditional random fields for identifying appropri- ate types of support for propositions in online user comments. In Proceedings of the 2nd Workshop on Argumentation Mining, pages 39-44, Denver, CO. Association for Computational Linguistics.
Dean Pomerleau and Delip Rao. 2017. Fake news challenge stage 1 (FNC-I): Stance detection. https//www..faknnnwwccaalllggggp
B Model Analysis
Jon Porter. 2023. Major reddit communities will go dark to protest. threat third-party apps. The Verge.
To get a better understanding of the dataset, model behaviour, and its failure cases, we conduct analy- sis of the English subset of the data outlined below.
Manoel Horta Ribeiro, Justin Cheng, and Robert West, 2023. Automated content moderation increases ad- herence to community guidelines. Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 4 May 2023, pages 2666- 2676. ACM.
B.1 Most Salient Features
To get a sense of what are the most salient words corresponding to each of the labels, we train a lo- gistic regression classifier using default parameters on TF-IDF features for the English training set. Us- ing LIME (Ribeiro et al., 2016), we then plot the most salient bi-grams associated with each label as shown in Figure 4. We expressions like fails and and not enough are very highly weighted with the delete class and nega- tively weighted for the keep class, which makes intuitive sense. Similarly, clearly passes and easily passes, which are presumably highlighting policy that the article fulfils are associated with the keep class, further demonstrating the intertwined nature of editor stance and policy references. The bi-gram per nom, an abbreviation for per nominator or per nomination, has positive weights for the delete and merge class while negative ones for the other two, indicating that when the article's nomination for deletion is referenced, the article tends to be voted
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should I trust you?": Explain- ing the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Fran- cisco, CA, USA, August 13-17, 2016, pages 1135- ACM.
that negative
Julian Risch and Ralf Krestel. 2018. Delete or not delete? semi-automatic comment moderation for the newsroom. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC- 2018), pages 166-176, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Robyn Caplan. 2018. Content context moderation? Data & Society.
Jeffrey Sablosky. 2021. "dangerous organizations: Face- book's 's content moderation decisions and ethnic vis- ibility myanmar". Media, Culture Society, 101--100

---

Topic
Comment
Stance
Policy
Deletion of Scot- tish Nursery Nurses Strike
per #NEWS. No source to establish lasting significance of this delete
What Wikipedia
ticular strike
not
Deletion of Motiva- tional press
Fails by a wide margin. see a lot of article mentioning the publisher but on taking a closer look, they are all trivial mentions. In fact some of them are made by certain Justin Sachs who works for the company. had a look at the article creator's s talk page
delete
Notability
Deletion of T. Ra-
per with major award noted above
keep
Notability
jshekar
Deletion
The beauty contests
taken part are not major, couldn't find any significant coverage reliable sources about her. She does not satisfy the as result
delete
Notability
Sabeetha
Wan-
niarachchi
Deletion of No
the album has received coverage [Link] Pitchfork. [Link]The Fader, [[ink]Exelaim!,, and [Link] XXL., all of which WikiProei Al- considered reliable sources Additional coverage exists hume/Souress/Albums WikiProjeet. [Link] Village Voice [Link] BrooklynVegan, With these mind, the alhum meets and
keep
Notability
Mountains
Manhattan
Löschung von Fleur Klingelberger
1. Zwischeparken bis Erreichen der IRelevanz, z.Z. im ANR
delete
Notability
Löschung
Relevanzkriterien wwtp://dwwwipiiii...o
wwk.ippdia.og//vwki//eeE. wtg/wwki//w.RKKKK bei Verkehr-
keep
Do not disrupt Wikipedia to il- Justrate a point
von Münsterland- Express
Rhein-
swege und -bauwerke enthalten nach meiner Lesart "Eisenbahn- strecke". Generelle Diskussionen sollien in anderem Rahmen geführt werden. Sehe aber das Dilemmaa welches uns hier wieder einmal vor die FüBe wirft
Löschung
von
Die Daten sind international nicht vergleichbar und von miser- aheler Datenqualität; selbst wenn man die deutlich hesseren Daten von Johns Hopkins University nehmen würde, ist das schlicht Schrott gemäß Also Ich weiß, wird nicht passieren, weil ist ja belegt!!!!eenselI
delete
No original re- scarch
COVID-19.
Pandemie/Statistik
Löschung von Gus- tav Wohlgemuth
Habe Artikel ein bisschen ühhraabeetee, mit Quellen und einem Literaturhinweis versehen. Wohlgemuth war Professor in Leipzig und erfüllt schon deshalb die Wwiieenschateeeeleeeeenaznkkiiie Deshalb (und natürlich auch, weil ich das Ding nicht umsonst bear- beitet haben will :-D
keep
Notability
Löschung Nagelpflegemittel
von
Einträge zwei anerkannten Lexika, daher ist #Allgemeine haltspunkte Relevanz/Relevan gegeben. LAE Fall I: Die Be- gründung des Löschantrags trifft eindoutig nicht oder nicht mehr zu
keep
Notability
Sigortam.net' in sil- inmesi
Yukanda "litgili kayda degerlik kriterlerimizin ikinci maddesi (Site veya sitedeki içerik, bir kurum veya yayı orgammin verdigi. iyi bili- nen ve bagumsiz bir ödul almış olmal)) geregince KD kabul edilmiyor mu?" şeklinde bir yorum yapmıştim. Bu kriteri karzıladigandan kalmalı madde
keep
Notability
Kurtuluş Yolu'nur nun silinmesi
KD oldugunu düşüüen arkadaşlar, kanyy hangi kriterleri göz önüne alarak vardıklarım belirtebilir mi? ilgili kriterler Iburada
comment
Notability
Ferec'in in silinmesi
Ferec grubu Ikayda degerlik ve 6' ya net bir şekilde uyuyor. 4'ünnti maddeye de uyması için bir kaç ay
keep
Notability
Emin Atli' nu silin- mesi
M6' dan direkt silinebilir. kriterleriai karşılamyor
delete
Notability
Zekeriya Onge'nin nin silinmesi
bu varsa da olmalıdır diye değil, 'deki kriterlere uygun olup ol- madığun bakılarak karar veriliyor. -Kullameı Kebel:
delete
Notability
Table Dataset examples for the three languages. Topic the target of the stance detection, Comment where the stance itself is expressed by the Wikipedia editor. Links in the table are replaced with [Link] for readability

---

250000
4000
Number of instences
3000
150000 0
Numbes of
100000
1000
90000
zoumes Rrieoo grodiage eftee
Eoan cortaaa mottuo ON Toleere cesteei
Emmoeee eeppon up Ditie andiiiid
aocuan Bang, wn
an pesteee cotte do pattdt-
ermenn you aepprttenn
sonn agu peroonee yottr wit
Ar and de snytu wood so nor
une do thoa cepie tos
aanee of sqptio
cateecti de
Covaaiiiin aeeeereee Rurver
ceateee un parii sppeee piti dnt Ceeeeeeee
(
Number ot Indtano
500
400
300
200
100
unt
Unspeedee degin Sadeaaae seceeee gog regeridd
Wiippedi leth comue poroonand
(c)
Figure 3: The 15 most frequently used policies across comments for (a) English, (b) German, and (c) Turkish.

---

for deletion or merging. The bi-gram with the high- est weight can you, which is intuitive since request for a change is being made.
for keep. For the comments, the topic in the form of the article discussed for deletion is provided.
Further, each comment annotated with one pol- icy they mention in the decision. We provide two datasets, one in which the comment is left as it was provided by the editor and one where all poli- cies are removed from the comment. We provide the dataset for three different language Wikipedias; English, German, and Turkish.
B.2 Error Analysis
We performed a brief error analysis of the best model for stance detection task in English i.c. En- glish BERT. Below, we provide the confusion ma- trix (Table 7), Fl scores per label (Table 8) as well as examples of comments which the model gol wrong with the highest probability (Table 9). Look- ing at the confusion matrix and Fl scores per label, the comment category is clearly the category the model has the hardest time with, often being mixed up with keep or delete. This makes intuitive sense as comments conceptually capture a broader range statements than the other categories. The model has tendency to over-predict delete due to its over-representation in the dataset, leaving scope for future studies to address this skewness. Look- ing examples of erroneous predictions, can see instances of the most frequent errors (delete- keep, delete-comment) made by the model.
How many instances are there in total (of each type, if appropriate)? The dataset consists of 437,7 777,770 comments for English, 8,637 comments for German, and 930 comments for Turkish, split into train, test, and dev. The split for Turkish is not in the same proportion for the other languages, to ensure reasonable amount of examples in the test set.
Does the dataset contain all possible instances or is it sample (not necessarily random) of instances from larger set? The dataset is lim- ited to comments, that contain policies. The full dataset of all comments with stance annotations contains 2,047,698 comments for English, 319,576 comments for German, and 43,043 comments for Turkish. This means, our dataset contains 21.4% of English, 2.7% of German, and 2.2% of the com- ments of the overall dataset.
Datasheet for Datasets
the following, we provide the Datasheet for Datasets as described by Gebru et al. (2021).
C.1 Motivation
What data does each instance consist of? The comments are processed to remove policies and are annotated with one policy per comment along the merged stance labels. For the topics, we added Deletion of to the article title.
For what purpose was the dataset created? Was there a specific task in mind? The aim of the dataset is to support automated content moderation by providing justification for decision of mod- erator well as predict the decision based on the comment.
Is there label or target associated with each in- stance? The labels are taken from the comment itself, i.e., the first policy mentioned in the com- ment along the stance label provided by the editor. The stance labels are processed to fit into the four categories of delete, keep, merge, comment. Fur- ther, the name of the article that is discussed for deletion is provided.
Who created the dataset (e.g., which team, re- search group) and on behalf of which entity (e.g., company, institution, organization)? The re- searchers listed as authors from their respective institutions.
Who funded the creation of the dataset? See acknowledgements.
Is any information missing from individual in- stances? Where possible, we removed the time stamp and mentions of usernames. The timestamps were removed as they are not relevant to the task. and the usernames were removed where possible for anonymity. As this is done automatically us- ing regex, there might be non-standard ways of mentioning usernames or times, that could not be removed
C.2 Composition
What do the instances that comprise the dataset represent (e.g., documents, photos, people, coun- tries)? The dataset contains Wikipedia editor's comment in deletion discussion, along with their decision between delete, keep, merge, or comment. The stance labels are merged from the larger la- bels, e.g., speedy keep is merged into the decision

---

y=comment features
y=delete top features
y=keep top features
y=merge top features
Weight'
Feature
Waight"
Feature
Waight"
Feature
Weight
Feature
+4.743
can you
+9.081
fails and
+6,904
clearly passes
+9.225
to list
+4,458
my nomination
+7.951
per nom
+6.321
easily passes
+8.107
or delete
+3.989
this discussion
+6.685
non notable
+6.097
clearly notable
+6.799
search term
+3.986
you
+6.265
reliable
118638
more positive..
+5.690
per nom
+3.984
not
+6.091
not enough
125579
negative..
+5.574
is blp
+3.977
this afd
+5.913
evidence
-61118
not notable
standalone
+5.481
105779 more positive.... ...
+5.328
sources
-6.236
non notable
article
138438
negative... per above
+5.232
fails the
-6.279
reliable
+5.326
the redirect
-4,617
+5.131
not notable
-6.389
none
+5.230
suggested
-4.747
per the
111174
positive...
-7.252
per nom
independent
+5,206
-5.877
per and
133043
more negative...
-9.100
not enough
notability
-6.390
per nom
-5.246
delete
-10.010
fails and
+5.038
to the
73651 more positive...
170566 more negative
Figure 4: Most salient bi-grams for each label in the training set
Are relationships between individual instances made explicit (e.g,, users' movie ratings, social network links)? Different comments are con- nected by the topic they discuss, i.e., the title of the article that is under discussion for deletion.
tiality, data that includes the content of individu- als' non-public communications)? As it is real data, it is possible that data is included that falls under these concerns. However, as the data is pub- lic and comments can be edited by the editors after posting, we believe that this data should be free of confidential information.
Are there recommended data splits (e.g., train- ing, development/validation,, testing)? The train/test/dev splits are provided. They are standard splits of 80/15/5 for English and German. Due to the small dataset size of the Turkish dataset, we de- cided alter the split so that there is reasonable number of comments in the test set (at least 200 comments).
Does the dataset contain data that, if viewed di- rectly, might be offensive, insulting, threatening, or might otherwise cause anxiety? As deletion discussion data contains interactions of people different language Wikipedias, that there is no offensive data, and we would like this to be considered for future reuse. Particularly, as articles of real people might be under discussion of deletion, this data should be handled with care.
cannot ensure
Are there any errors, sources of noise, or re- dundancies in the dataset? As automatically process real user comments, errors are unavoidable, but to the best of our knowledge, we limited them to a minimum. There might be minor errors in the stance labels. Further, as we are working with only the first policy mentioned, there might be important policy mentions that are overlooked. However, as the median number of policies mentioned for all three datasets is 1, believe that this will have very little impact on the tasks working with poli- cies.
Does the dataset identify any subpopulations (e.g., by age, gender)? The dataset removes user- names where possible. Where usernames might have been overlooked, we would like to point out that this is how the editors want to identify and is not necessarily connected to their real-life identity.
Is it possible to identify individuals (i.e., one or more natural persons), either directly or indi- rectly (i.e., in combination with other data) from the dataset? As above, remove usernames where possible. If usernames are accidentally in- cluded the dataset, it is only possible to identify the user on Wikipedia.
Is the dataset self-contained, does it link to or otherwise rely on external resources (e.g., web- sites, tweets, other datasets)? The dataset self- contained, no other resources are needed to work with it for the described tasks.
Does the dataset contain data that might be con- sidered sensitive in any way (e.g., data that re- veals race or ethnic origins, sexual orientations, religious beliefs, political opinions or union
Does the dataset contain data that might be con- sidered confidential (e.g., data that is protected by legal privilege or by doctor-patient confiden-

---

memberships, or locations; financial or health data; biometric or genetic data; forms of gov- ernment identification, such as social security numbers; criminal history)? Only data from the discussion itself is included, which should not be able to identify any sensitive characteristics of any of the editors contributing to these conversations.
topics, and the linking policies across languages. was conducted by the authors.
Over what timeframe was the data collected? The dataset was collected over the course of few days at the end of 2022, and contains data from 2005 (2006 for Turkish) until 2022.
Were any ethical review processes conducted (e.g., by institutional review board)? No ethical review process was conducted.
C.3 Collection Process
How was the data associated with each instance acquired? The data was scraped from the pub- lie archive of deletion discussions of Wikipedia in English, German, and Turkish. This data can be accessed at English, German, Turkish.
Did you collect the data from the individuals in question directly, or obtain via third parties or other sources (e.g., websites)? The data was collected from the English, German, and Turkish Wikipedia deletion discussion pages (see above for the links).
What mechanisms or procedures were used to collect the data (e-g,, hardware apparatuses or sensors, manual human curation, software pro- grams, software APIs)? We collected the topics and comments in an automated manner from the discussion pages. We then processed the stance la- bels by semi-an aumiiialtmaaiclllyy merging them into the four stance labels described above. The policies were automatically extracted by looking for the [[iiippdi.a] (or [ [[vik] iedi: for Turk- ish) prefix. As this leaves us with large number of links that might contain also other Wikipedia pages, such WikiProjects, we manually processed the policies by merging them and removing comments linking to irrelevant pages. We further manually edited the Turkish topics to contain the equivalent of Deletion of in Turkish, as vowel harmony made it difficult to automatically add this. For English and German Deletion of and Löschung von respec- tively were added automatically. The alignment of policies across the three languages was done manually, relying on the Wikipedia interwiki links, which link Wikipedia pages across languages.
Were the individuals in question notified about the data collection? It was not possible con- tact individuals, as the data was on Wikipedia, we could not notify each user about the collection of the data.
Did the individuals in question consent to the collection and use of their data? N/A
If consent was obtained, were the consenting individuals provided with a mechanism to voke their consent in the future or certain uses? N/A
Has an analysis of the potential impact of the dataset and its use on data subjects (e-g,, a data protection impact analysis) been conducted? As we removed usernames where possible, we be- lieve the dataset should not have direct impact the data subjects.
C.4 Prepooesinn/deeninglabbing
Was any preproesiig//deanigglaaliii of the data done (e-g., discretization or bucketing, tok- enization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? The usernames, timestamps, and policies were removed from the comments.
If the dataset is a sample from larger set, what the sampling strategy (e.g., determin- istic, probabilistic with specific sampling prob- abilities)? We only retain comments containing stances and policies. This sample might not be representative of the overall deletion discussion but was necessary for the task.
Was the "raw" data saved in addition to the pre- processe/clemeed/abeld data (e.g., to support unanticipated future uses)? We provide ver- sion of the dataset in which the comment contains the policies. We do not provide version with user- names and timestamps. This data can be found "raw" on the Wikipedia discussion pages linked above.
Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e-g., how much were crowdworkers paid)? The mamual editing of, e.g., the policies, the editing of the Turkish

---

Is the software that was used to prepro- cess/clean/label the data available? https: //github.
Have any third parties imposed IP-based other restrictions on the data associated with the instances? Wikipedia data is licensed under CC-BY-SA and GFDL, which has to be respected for further distribution of the data.
C.5
Uses
Has the dataset been used for any tasks already? The dataset has been used for the tasks described in this paper.
Do any export controls or other regulatory re- strictions apply to the dataset or to individual instances? Wikipedia data is licensed under CC- BY-SA and GFDL, which has be respected for further distribution of the data.
there repository that links any or all pa- pers or systems that use the dataset? We aim to list them on the GitHub or HuggingFace dataset page.
C.7 Maintenance
Who will be supprrtinghssttngmaiittinnng the dataset? The authors can be contacted for the maintenance of the dataset.
What (other) tasks could the dataset be used for? The dataset could be used for further experi- ments in the field of content moderation.
How the owner//curator/naaager of the dataset be contacted (e.g,, email address)? The first authors listed above can be contacted.
there anything about the composition of the dataset or the way it was collected and pre- processed/cleaned/lbbele that might impact fu- ture uses? The described removal of usernames, timestamps, and policies should be considered in the future use of the dataset.
Is there an erratum?
No.
Will the dataset be updated (e.g., to correct label- ing errors, add new instances, delete instances)? As of now; there are no plans to update the dataset beside correcting errors the existing version of the dataset. However, it could be recreated for newer deletion discussions.
Are there tasks for which the dataset should be used? Any work related to identifying Wikipedia users is discouraged.
If the dataset relates to people, there applica- ble limits the retention of the data associated with the instances (e-g., were the individuals in question told that their data would be retained
C.6 Distribution
Will the dataset be distributed to third parties outside of the entity (e-g., company, institution, organization) on behalf of which the dataset was created? The dataset will be freely accessible under a permissive license.
for a fixed period of time and then deleted)? N/A
Will older versions of the dataset continue to be supportel/nsteennnntannn???
How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? The dataset will be freely available GitHub and at common website for dataset distribution.
If errors de- tected in the dataset, e.g., missed usernames in the automatic processing the comments, the authors are committed to fixing these errors.
If others want to extend/augment/huid on/contribute to the dataset, is there mechanism for them to do so? The dataset will be available on HuggingFacc datasets(https:: http:://////ggggg..a...... co/ datasets http://oppnn//l//eeeeeeeee stanceesstaeee, where issues can be opened to notify the authors. The authors can also be contacted via email.
When will the dataset be distributed? The dataset will be distributed after acceptance this paper.
Will the dataset be distributed under a copy- right or other intellectual property (IP) li- cense, and/or under applicable terms of use (ToU)? As the text is derived from Wikipedia, and Wikipedia licensed under the creative com- mons licenses (CC-BY-SA) and GNU Free Doc- umentation License (GFDL), we will license the dataset under a similar, compatible dataset license.

---

Label
Label comment
English
keep
keep, oppose, save, retain, keeep, preserve, stay
delete
delete, support, agree, del comment, note, question, reply, response, clarification, answer, request, query, neutral, abstain, commmment, undecided, uncertain, not sure, unsure, no opinion, coment, fyi, vote merge, redirect, move, rename, rewrite, split, relist
comment
merge
German
keep
behalten, lac, bleibt, relevant, bleiben, nein, behalt, erhalten
delete
löschen, gelöscht, sla, 1ösch, løschen, loeschen, weg, wech, entsorgen, hinfort tage, sieben tage, neutral, überarbeiten, unentschieden verschieben, redirect, weiterleitung
Fl score
comment
comment
0.47
delete
0.94
merge
keep 0.89
Turkish
merge 0.76
keep
kalsın, kalsin, silinmesin, kalması silinsin, silinmeli, sil yorum, çekimser, cevap, soru, tarafsiz, ping, seslen, mesaj, düzenle, yanıt, kararsız, geçersiz aktarılsın, birleştirilsin, aktanImasıı, taşınsın
delete
Table 8: Macro-averaged Fl scores per label for English BERT in the single task setting
comment
merge
Table 6: Labels for each language and text representa- tions of each of these labels that were merged into the labels, the three languages
comment
delete
keep
merge
comment
1083
1121
866
22
delete
188
27092
498
112
keep
182
795
9830
51
merge
19
456
150
1311
Table 7: Confusion Matrix for Stance Detection with BERT English

---

Instance
Predicted Label
Actual Label
notablity proven, the references do mention the sub- ject of the article and only one, at best, is reliable source and redirect, fails, and The term is neologism and is not used manner denote fiscal policy, either contemporar- ily or in archaic terminology.
delete
comment
delete
keep
fails on all counts. Also potential
delete
comment
The sources added by voceditenore seem to adequately establish the notability requirements
delete
keep
Seems like notable article. It needs Ireliable source though
delete
comment
as it appears to exist for the purpose of a lsynthetic associa- tion between this scene and the TV pilot
keep
merge
"Falls no sources discuss this particular data set. Only references GRG lists (note that none of these sources discuss ""Caribbean supercentenarians" and are just big table of names) which do nothing establish notability. All names in this article are available in other Longevity articles"
delete
keep
Per all arguments for deletion are based on subjective reasons. All the information comes from Time magazine New York Times, and others prime sources
delete
keep
"as not-notable under guidelines, as well as and concerns. If author(s) can cite Iverifiable lsources that this, ""Has received major award for excellence in some aspect of filmmaking," "" or ""Has been the subject of multiple, non- trivial news stories describing its artistic or societal impact"" I would consider it notable"
delete
comment
Speculative and highly inaccurate cherry-picked. No place for this ruminative dissertation Wikipedia
delete
keep
"the coverage is routine and / or wholly unremarkable, minor chain. If promotionalism is removed (""...with vi- sion of spreeding the vegetarian lifestyle worldwide...!") directory of
delete
comment
there won't be anything left. Wikipedia nn businesses, nor an opportunity to place franchisee ads"
Table 9: Most confident erroneous predictions by the English BERT model for stance detection


Reviews:
[{"review_id": "yRPN9ulwLi", "reviewer": "Reviewer_6EKo", "summary": "", "questions": "It would be cool to include some error analysis, e.g. from the best baseline system, so that people who work on your task in the future would understand where to make improvements.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback. \n\n### Human Annotation\n\nPer the reviewers’ suggestion, we sampled a subset of 100 randomly selected comments from the test dataset and did a preliminary human annotation study on this subset, for both stance and policy prediction. The accuracy scores between our annotation and the labels (which were self annotated by the Wikipedia editors) are 88% and 55% for stance and policy prediction respectively. We attribute the relatively lower score for policy prediction to the large number of labels (94), i.e., policies, with a random prediction baseline of 1%. These policies also at times overlap with each other e.g. *Wikipedia:Lists* and *Wikipedia:Categories, lists, and navigation templates*. While we merged some of the policies with the highest overlap (e.g. notability criteria for specific article types were merged into the *Wikipedia:Notability* policy), for a realistic setup we kept the policies with lesser overlap separate. We plan to extend the human annotation experiment to a larger sample, given the time constraint we wanted to give an insight into the possible outcome of the human annotation study already.\n\n### Error Analysis\n\nRegarding the proposed error analysis, we performed a brief error analysis of the best model for the English stance detection task i.e. English BERT. Below, we provide the F1 scores for each label, the confusion matrix, as well as examples of comments that the model got wrong with the highest probability. Clearly, the *comment* category is the one the model has the hardest time with, often being mixed up with *keep* or *delete*. This makes intuitive sense as comments conceptually capture a broader range of statements than the other categories. The model has a tendency to over-predict *delete* due to its overrepresentation in the dataset, leaving scope for future studies to address this skewness. Looking at the examples (“Erroneous predictions with highest confidence” below), we can see instances of the most frequent errors (*delete-keep, delete-comment*) made by the model.\n\n~ F1 score per label ~ \n|         |   F1 score |\n|:--------|-----------:|\n| comment |   0.474584 |\n| delete  |   0.944729 |\n| keep    |   0.885506 |\n| merge   |   0.763986 |\n\n\n\n~ Confusion Matrix ~\n \n|         |   comment |   delete |   keep |   merge |\n|:--------|----------:|---------:|-------:|--------:|\n| comment |      1083 |     1121 |    866 |      22 |\n| delete  |       188 |    27092 |    498 |     112 |\n| keep    |       182 |      795 |   9830 |      51 |\n| merge   |        19 |      456 |    150 |    1311 |\n\n~ Erroneous predictions with the highest confidence ~\n\n*Please note that policies have been removed, which leads to incomplete sentences such as “fails <space>, and <space>.”, where editors refer to policies in the <space>.*\n\n* Instance 4911: no notablity proven, the references do not mention the subject of the article and only one, at best, is a |reliable source\n  * Predicted Label: delete\n  * Actual Label: comment\n* Instance 11806: and redirect, fails ,  and . The term is a neologism and is not used in a manner to denote fiscal policy, either contemporarily or in archaic terminology. &mdash\n  * Predicted Label: delete\n  * Actual Label: keep\n* Instance 37411: as it appears to exist for the purpose of a |synthetic association between this scene and the TV pilot\n  * Predicted Label: keep\n  * Actual Label: merge\n* Instance 39079: The sources added by voceditenore seem to adequately establish the notability requirements at\n  * Predicted Label: delete\n  * Actual Label: keep\n--------------------------------------\n\nWe will add all the points discussed above to the final manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The task of policy and stance detection is well motivated and has real world implications. \nThe dataset + task design is well-aligned with the real world use case of making editorial decisions on Wikipedia.  \nThe proposed multilingual dataset could facilitate future research on the direction, and will be interesting across research community of different languages.", "reasons_to_reject": "IMHO Ideally the paper should feature some analysis on the quality of the test set in the dataset. e.g. Check human / expert agreement compared to the labels in the test/dev set on a small sample. This can give us a sense of what \"oracle\" or upper-bound performance on your test set, which will help contextualize the performance numbers you get out of your baselines. \n\nHappy to raise my ratings if the authors can address my concerns.", "questions_to_authors": "It would be cool to include some error analysis, e.g. from the best baseline system, so that people who work on your task in the future would understand where to make improvements.", "paper_topic_and_main_contributions": "The paper proposes a dataset for action/policy detection and post stance detection during wikipedia editor discussions. The study is motivated by the fact that during editor discussions, the suggested action/policy is rarely explicitly mentioned, especially in non-English wikipedia. The paper introduces two different tasks, policy detection and stance detection, as well as a joint setup, namely transparent stance detection, where the policy and stance are predicted jointly.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Who Did What: Editor Role Identification in Wikipedia, Yang et. al. in ICWSM 2016. \nAnnotating Social Acts: Authority Claims and Alignment Moves in Wikipedia Talk Pages, Bender et. al. 2011", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 908, 990, 1061, etc -- In appendix: Remove all mentions of \"Any other comments?\" :P?"}, "extra_text": {}}, {"review_id": "NcZ2XB7jvD", "reviewer": "Reviewer_N2ar", "summary": "", "questions": "How do you define stance? What expressions of stance are the most significant features distinguishing moderation statements within your dataset?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for raising the important point of discussing the linguistic perspective on stance. We define stance along the commonly adopted definition in the field of natural language processing, as the expression of a speaker’s standpoint and judgment towards a given proposition (Biber & Finegan (1988)). In our case, the proposition is the deletion of a given article, and the standpoint is expressed through one of *keep, delete, merge*, or *comment* labels that the author articulates at the beginning of their comment. We will include the definition of stance as well as the relevant citations in the related work section to provide more context on the area as a whole.\n\nTo measure common features in the comments expressing stance, we trained a TF-IDF based linear classifier to understand the words most associated with each of the four stances expressed in the comments. We will add a visualisation of our findings to the paper. Below, we provide a textual version of the bi-grams with the highest and lowest weights for the above classifier. We can see that negative expressions like “fails and” and “not enough” are very highly weighted with the *delete* class and negatively weighted for the *keep* class, which makes intuitive sense. Similarly, \"clearly passes\" and \"easily passes\", which are presumably highlighting a policy that the article fulfills are associated with the *keep* class, further demonstrating the intertwined nature of editor stance and policy references. The bi-gram “per nom”, an abbreviation for \"per nominator\" or “per nomination”, has positive weights for the *delete* and *merge* classes while negative ones for the other two, indicating that when the article’s nomination for deletion is referenced, the article tends to be voted for deletion or merging. “can you” being the highest weighted bi-gram for *comment* is intuitive as well since a request for a change is being made.\n| target   | feature                |    weight |\n|:---------|:-----------------------|----------:|\n| delete   | fails and              |   9.08073 |\n| delete   | per nom                |   7.95096 |\n| delete   | non notable            |   6.68485 |\n| delete   | no reliable            |   6.26482 |\n| delete   | not enough             |   6.09139 |\n| delete   | no sources             |   5.32827 |\n| delete   | fails the              |   5.23198 |\n| delete   | not notable            |   5.13093 |\n| keep     | clearly passes         |   6.90419 |\n| keep     | easily passes          |   6.32113 |\n| keep     | clearly notable        |   6.09736 |\n| keep     | not notable            |  -6.11826 |\n| keep     | non notable            |  -6.23591 |\n| keep     | no reliable            |  -6.27908 |\n| keep     | per nom                |  -7.2516  |\n| keep     | not enough             |  -9.10034 |\n| keep     | fails and              | -10.0099  |\n| merge    | to list                |   9.22478 ||\n| merge    | search term            |   6.79893 |\n| merge    | per nom                |   5.69042 |\n| merge    | standalone article     |   5.48076 |\n| merge    | as suggested           |   5.23025 |\n| merge    | to the                 |   5.03847 |\n| comment  | can you                |   4.74349 |\n| comment  | my nomination          |   4.4577  |\n| comment  | this discussion        |   3.98882 |\n| comment  | not sure               |   3.9844  |\n| comment  | this afd               |   3.97706 |\n| comment  | per nom                |  -6.38978 |\n\n--------------------------------------------------------------------------------\nBiber, D., & Finegan, E. (1988). Adverbial stance types in English. *Discourse Processes, 11*, 1-34.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The research addresses an important issue, i.e. contributing to the transparency of moderation discussions on online content platforms such as Wikipedia based on the stance of expressed in moderation statements as well we policies issued by the platforms. It takes into consideration data from English as well as German and Turkish, thus addressing the issue for multiple languages even though for the latter two languages there is a data sparsity problem as well as differences between the language versions of Wikipedia as far as explicit policy statements are concerned.", "reasons_to_reject": "The authors do not address the linguistic aspects of expressions of stance. There is a rich literature on stance in linguistics by scholars like Geoff Thompson, Susan Hunston and Douglas Biber. This literature is not addressed here at all and would potentially contribute to a better understanding of expressions of stance and the respective feature engineering.", "questions_to_authors": "How do you define stance? What expressions of stance are the most significant features distinguishing moderation statements within your dataset?", "paper_topic_and_main_contributions": "The paper is concerned with the analysis and transparency of moderation process discussions in multilingual  online media, in this case Wikipedia which makes these discussions publicly available, albeit not for all such discussions and not always in their full breadth. The aim is an approach for analyzing not only the  moderation itself, but also the stated reasoning behind e.g. suggestions for deletion of content. In presenting an approach to help contributors better understand and transparently explain moderation decisions the approach presented contributes to making moderation decisions more transparent to the community thus potentially boosting user engagement in content creation and moderation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Hunston, Susan  & Thompson, Geoffrey. 2000. Evaluation in Text : Authorial Stance and the Construction of Discourse: Authorial Stance and the Construction of Discourse. Oxford University Press."}, "extra_text": {}}, {"review_id": "1Wu7nLDj8S", "reviewer": "Reviewer_Xa7D", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their appreciation of transparent content moderation as a valuable and important topic. \n\nMany online platforms, such as Reddit or Facebook, set overall content policies [1][2] that communication on the platform has to adhere to. Communities on those platforms further have norms and policies of their own. For members of such communities, it is imperative to be able to comprehend why their content was deleted based on these policies. For the platforms themselves, highlighting such policies are crucial for maintaining the trust and safety of the platform and is also required by upcoming regulation. Finally, for the content moderators on those platforms, navigating through a large number of legalese heavy policies can be challenging and time consuming. Hence, joint moderation and linking the relevant policies has demonstrable impact and can make the process tenable for the moderators on all platforms.\n\nHowever, to the best of our knowledge, joint prediction of the moderation decision and the corresponding policy has not been explored in prior research. As policies for content moderation differ across platforms, our model wouldn’t be able to predict other platforms’ policies in a zero-shot setting. However:\n* The model could be fine-tuned on another platforms’ policies and potentially leverage the policy-content relationship observed on Wikipedia\n* We believe that the general framework that we propose as well as the challenges faced in defining the problem as a classification task (for instance, overlap of policies) has implications beyond Wikipedia, to most platforms where norms or policies are explicitly defined.\n\nTo address the reviewer’s point of F1 scores of 0.6-0.8 not having real world impact: the purpose of our study is not to provide a cross-domain tool for transparent content moderation. Rather, it is to present research on a first attempt to conceptualise and address this novel task, with Wikipedia as an example, that we hope the research community can improve upon and the larger community can benefit from. Hence, the scores we provide are baselines, demonstrating evidence of the feasibility and difficulty of the task.\n\nWe will add the discussion around the real-world application of our approach to the camera-ready version of the paper.\n\n-------------------------------------------------\n[1] https://www.redditinc.com/policies/content-policy\n\n[2] https://transparency.fb.com/policies/community-standards/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Addresses the important issue of content moderation (a very important problem), with a focus on transparency.\n- Shows successful application to Wikipedia editor discussions in three languages, highlighting the potential for multilingual applicability.\n- The release of the multilingual dataset and joint prediction models could foster more research and development in content moderation and transparency.", "reasons_to_reject": "- Limited applicability due to the focus on Wikipedia, which might make the method too niche. It would help if the authors point out other applicable areas.\n- Related with the above, It is uncertain whether the method can be generalized to other out-of-distribution problems or online platforms other than Wikipedia. Further exploration and validation on other platforms would strengthen the paper.\n- Unclear if the F1 scores between 0.6 - 0.8 have a significant real-world impact.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper focuses on developing models for predicting the stance and policy (reason) for Wikipedia content moderation in three languages.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a dataset of Wikipedia editor discussions for policy detection and stance detection in three languages, along with models for the two tasks and a joint task. 

Reviewers appreciated addressing the important issue of content moderation with a focus on transparency. Also, the multilingual dataset and models released will be useful for future research. 

The reviewers share a sentiment that a deeper analysis of the dataset from a linguistic perspective can be beneficial, shedding light on linguistic characteristics of different classes and confirming the quality of the dataset in the process. Also, the focus on the Wikipedia editor discussions can be a weakness as it is hard to confirm generalizability of the findings.
Decision: Accept-Main