DOI: 10.7910/dvn/e0ryj4
title: Replication data for: Effects of algorithmic flagging on fairness: Quasi-experimental evidence from Wikipedia

description: &lt;h1 id="code-overview"&gt;Code overview&lt;/h1&gt;
&lt;p&gt;The code is all in &lt;code&gt;code.tar.gz&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="identifying-thresholds-and-cutoffs-over-time"&gt;Identifying thresholds and cutoffs over time&lt;/h2&gt;
&lt;p&gt;Pretty much all in &lt;code&gt;identify_cutoffs.py&lt;/code&gt;. Iterates git repository, parses wmf-config/InitializeSettings.php. Interprets historical json versions. Builds a pandas table of events with threshold configuration settings and some other configuration settings like when different UI elements were enabled. There are some traces of the first attempt at a project, an attempted time series analysis that failed due to high noise.&lt;/p&gt;
&lt;p&gt;In cases where thresholds are not configured, default thresholds are configured in the ORES service repository &lt;code&gt;mediawiki-extentions-ORES/extension.json&lt;/code&gt; (a copy of the git repository is in &lt;code&gt;mediawiki-extensions-ORES.tar.gz&lt;/code&gt;. &lt;code&gt;get_default_threshold_strings.py&lt;/code&gt; scripts this git repository to get the history of the default thresholds. They don’t change much.&lt;/p&gt;
&lt;h2 id="reading-server-admin-log"&gt;Reading server admin log&lt;/h2&gt;
&lt;p&gt;Right now, the code to get the history of deployments is in a chunk at the &lt;code&gt;identify_cutoffs.py.&lt;/code&gt; I think I will refactor this to its own file. The precise timing of changes to the models does not come from the source code repository but rather the live deployments. The SAL (server admin log) publishes a history of live deployments.&lt;/p&gt;
&lt;h2 id="converting-ores-configuration-strings-to-prediction-score-cutoffs"&gt;Converting ORES configuration strings to prediction score cutoffs&lt;/h2&gt;
&lt;p&gt;This is done by &lt;code&gt;ores_archaeologist.py&lt;/code&gt;. This is by far the most complex complex script and it wraps functionality from the &lt;code&gt;revscoring&lt;/code&gt; package (a copy of this repository is in &lt;code&gt;revscoring.tar.gz&lt;/code&gt;) to load different versions of models and analyze them. It checks out git commits corresponding to changes in InitializeSettings.php or SAL, installs the correct python dependencies in a helper repository to make sure the models run in as close as possible to the correct environment to ensure the thresholds are correct. &lt;code&gt;helper.py&lt;/code&gt; has functions used by &lt;code&gt;ores_archaeologist.py&lt;/code&gt;. Sometimes there are errors and we start analyzing data starting after the last error to give a continuous period.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;get_model_threshold.py&lt;/code&gt; is a simple script that is run by &lt;code&gt;ores_archeologist.py&lt;/code&gt; and actually loads the &lt;code&gt;revscoring&lt;/code&gt; code.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;ores_archeologist.py&lt;/code&gt; script can also attempt to find historical revision scores. This was not actually used in the paper because these historical scores may not be reliable. &lt;code&gt;revscoring_score_shim.py&lt;/code&gt; is analogous to &lt;code&gt;get_model_threshold.py&lt;/code&gt;, but for scoring edits.&lt;/p&gt;
&lt;h2 id="sampling-from-wikimedia-history-and-event-table"&gt;Sampling from Wikimedia history and event table&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;sample_edits_near_thresholds.py&lt;/code&gt; is a spark script that runs on the Wikimedia Foundation datalake nad builds the revision dataset. Much of the logic is in&lt;code&gt;spark_functions.py&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="fitting-models."&gt;Fitting models.&lt;/h2&gt;
&lt;p&gt;The master file is &lt;code&gt;fit_10_rdds.R&lt;/code&gt; and &lt;code&gt;fit_vlb_rdds.R&lt;/code&gt;. During the review cycle we found a bug in the ‘very likely bad’ data and I refit only those models to save time. &lt;code&gt;fit_10_rdds.R&lt;/code&gt; just fits the models asynchronously. The main logic is in &lt;code&gt;fit_base_rdds.R&lt;/code&gt; and &lt;code&gt;modeling_init.R&lt;/code&gt;. The dataset is put together in &lt;code&gt;modeling_init.R&lt;/code&gt;. &lt;code&gt;ob_util.R&lt;/code&gt; and &lt;code&gt;helper.R&lt;/code&gt; have a few miscellanious functions. &lt;code&gt;rdd_defaults.R&lt;/code&gt; has the formulas and sets stan modeling parameters.&lt;/p&gt;
&lt;p&gt;Fit models are available in &lt;code&gt;models.tar.gz&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="interpreting-models."&gt;Interpreting models.&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;analyze_threshold_models.R&lt;/code&gt; builds smaller dataframes and variables that will be used by the Knitr Latex system to build the paper. &lt;code&gt;analyze_vlb_models.R&lt;/code&gt; does the same, but just for the ‘very likely bad’ data. Code shared by both scripts are in &lt;code&gt;analyze_main_models.R&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="dataset-summary-statistics"&gt;Dataset summary statistics&lt;/h2&gt;
&lt;p&gt;Some additional statistics reported in the paper are calculated in &lt;code&gt;summary_stats.R&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="evaluating-encoded-bias"&gt;Evaluating encoded bias&lt;/h2&gt;
&lt;p&gt;The &lt;code&gt;bias_analysis.tar.gz&lt;/code&gt; archive has code and data used for evaluating the bias of the ORES models including a copy of the &lt;code&gt;editquality&lt;/code&gt; git repository. A copy of the repository is in &lt;code&gt;editquality.tar.gz&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="building-the-paper-and-appendix"&gt;Building the paper and appendix&lt;/h2&gt;
&lt;p&gt;This is in the &lt;code&gt;paper.tar.gz&lt;/code&gt; and &lt;code&gt;appendix.tar.gz&lt;/code&gt; archives.&lt;/p&gt;
&lt;h1 id="data-files-overview"&gt;Data files overview&lt;/h1&gt;
&lt;p&gt;The following data files are published at the top level of the dataverse. Copy them into a &lt;code&gt;data&lt;/code&gt; subdirectory to use them with the code.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;cutoff_revisions_2periods.csv.gz.part1&lt;/code&gt; and &lt;code&gt;cutoff_revisions_2periods.csv.gz.part2&lt;/code&gt; have the full dataset of edits within the neighborhood.You should do &lt;code&gt;cat cutoff_revisions_2periods.csv.gz.part1 cutoff_revisions_2periods.csv.gz.part2 &gt; cutoff_revisions_2periods.csv.gz&lt;/code&gt; and then decompress the output to get the full csv.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;cutoff_revisions_sample.csv&lt;/code&gt; and &lt;code&gt;cutoff_revisions_sample_vlbfix.csv&lt;/code&gt; have the sampled datasets on which the models are fit.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;threshold_strata_counts.csv&lt;/code&gt; and &lt;code&gt;threshold_strata_counts_vlbfix.csv&lt;/code&gt; have the counts from statified sampling which are used to calculate modeling weights.&lt;/p&gt;
&lt;h1 id="what-does-vlbfix-mean"&gt;What does vlb_fix mean?&lt;/h1&gt;
&lt;p&gt;The original submission of the paper contained a bug that affected the sample at the verylikelybad RCFilters threshold. The bug was on line 241 of &lt;code&gt;sample_edits_near_threshold.py&lt;/code&gt; and lead to NA values in the sample which would have affected the sample weights. During the revise-and-resubmit process we found and fixed the bug and fit new models at the verylikelybad threshold.&lt;/p&gt;
&lt;h1 id="license"&gt;LICENSE&lt;/h1&gt;
&lt;p&gt;The data in this repository is released under a CC0 license. The original code is released under an MIT permissive license. Code included through external git repositories is repackaged here and released under the applicable open source licenses. See the pacakges for details.&lt;/p&gt;