DOI: 10.7910/dvn/u5v0g1
title: Replication Data for: Measuring Wikipedia Article Quality in One Dimension by Extending ORES with Ordinal Regression

description: &lt;p&gt;This dataset provides code, data, and instructions for replicating the analysis of &lt;em&gt;Measuring Wikipedia Article Quality in One Dimension by Extending ORES with Ordinal Regression&lt;/em&gt; published in OpenSym 2021 (link to come). The paper introduces a method for transforming scores from the &lt;a href="https://www.mediawiki.org/wiki/ORES#Article_quality" title="mediawiki.org"&gt;ORES quality models&lt;/a&gt; into a single dimensional measure of quality amenable for statistical analysis that is well-calibrated to a dataset. The purpose is to improve the validity of research into article quality through more precise measurement.&lt;/p&gt;
&lt;p&gt;The code and data for replicating the paper are found in this dataverse repository. If you wish to use method on a new dataset, you should obtain the actively maintaned version of the code from this &lt;a href="https://code.communitydata.science/articlequality_ordinal.git" title="code.communitydata.science"&gt;git repository&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;If you attempt to replicate part of this repository please let me know via an email to nathante@uw.edu.&lt;/p&gt;
&lt;h1 id="replicating-the-analysis-from-the-opensym-paper"&gt;Replicating the Analysis from the OpenSym Paper&lt;/h1&gt;
&lt;p&gt;This project analyzes a sample of articles with quality labels from the &lt;a href="https://dumps.wikimedia.org/" title="dumps.wikimedia.org"&gt;English Wikipedia XML dumps&lt;/a&gt; from March 2020. Copies of the dumps are not provided in this dataset. They can be obtained via &lt;a href="https://dumps.wikimedia.org/" class="uri"&gt;https://dumps.wikimedia.org/&lt;/a&gt;. Everything else you need to replicate the project (other than a sufficiently powerful computer) should be available here.&lt;/p&gt;
&lt;p&gt;The project is organized into stages. The prerequisite data files are provided at each stage so you do not need to rerun the entire pipeline from the beginning, which is not easily done without a high-performance computer. If you start replicating at an intermediate stage, this should overwrite the inputs to the downstream stages. This should make it easier to verify a partial replication.&lt;/p&gt;
&lt;p&gt;To help manage the size of the dataverse, all code files are included in &lt;code&gt;code.tar.gz&lt;/code&gt;. Extracting this with &lt;code&gt;tar xzvf code.tar.gz&lt;/code&gt; is the first step.&lt;/p&gt;
&lt;h2 id="getting-set-up"&gt;Getting Set Up&lt;/h2&gt;
&lt;p&gt;You need a version of &lt;code&gt;R&lt;/code&gt; &gt;= 4.0 and a version of &lt;code&gt;Python&lt;/code&gt; &gt;= 3.7.8. You also need a &lt;code&gt;bash&lt;/code&gt; shell, &lt;code&gt;tar&lt;/code&gt;, &lt;code&gt;gzip&lt;/code&gt;, and &lt;code&gt;make&lt;/code&gt; installed as they should be installed on any Unix system. To install &lt;code&gt;brms&lt;/code&gt; you need a working C++ compiler. If you run into trouble see the &lt;a href="https://github.com/stan-dev/rstan/wiki/RStan-Getting-Started" title="github.com"&gt;instructions for installing Rstan&lt;/a&gt;. The datasets were built on &lt;code&gt;CentOS 7&lt;/code&gt;, except for the ORES scoring which was done on Ubuntu 18.04.5 and building which was done on Debian 9.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;RemembR&lt;/code&gt; and &lt;code&gt;pyRembr&lt;/code&gt; projects provide simple tools for saving intermediate variables for building papers with LaTex. First, extract the &lt;code&gt;articlequality.tar.gz&lt;/code&gt;, &lt;code&gt;RemembR.tar.gz&lt;/code&gt; and &lt;code&gt;pyRembr.tar.gz&lt;/code&gt; archives.&lt;/p&gt;
&lt;p&gt;Then, install the following:&lt;/p&gt;
&lt;h3 id="python-packages"&gt;Python Packages&lt;/h3&gt;
&lt;p&gt;Running the following steps in a new &lt;a href="https://docs.python.org/3/library/venv.html" title="docs.python.org"&gt;Python virtual environment&lt;/a&gt; is strongly recommended. Run &lt;code&gt;pip3 install -r requirements.txt&lt;/code&gt; to install the Python dependencies. Then navigate into the &lt;code&gt;pyRembr&lt;/code&gt; directory and run &lt;code&gt;python3 setup.py install.&lt;/code&gt;&lt;/p&gt;
&lt;h3 id="r-packages"&gt;R Packages&lt;/h3&gt;
&lt;p&gt;Run &lt;code&gt;Rscript install_requirements.R&lt;/code&gt; to install the necessary R libraries. If you run into trouble installing &lt;code&gt;brms&lt;/code&gt; see the instructions on&lt;/p&gt;
&lt;h2 id="drawing-a-sample-of-labeled-articles"&gt;Drawing a Sample of Labeled Articles&lt;/h2&gt;
&lt;p&gt;I provide steps and intermediate data files for replicating the sampling of labeled articles. The steps in this section are quite computationally intensive. Those only interested in replicating the models and analyses should skip this section.&lt;/p&gt;
&lt;h3 id="extracting-metadata-from-wikipedia-dumps"&gt;Extracting Metadata from Wikipedia Dumps&lt;/h3&gt;
&lt;p&gt;Metadata from the Wikipedia dumps is required for calibrating models to the revision and article levels of analysis. You can use the &lt;code&gt;wikiq&lt;/code&gt; Python script from the &lt;a href="https://code.communitydata.science/mediawiki_dump_tools.git" title="code.communitydata.science"&gt;mediawiki dump tools&lt;/a&gt; git repository to extract metadata from the XML dumps as TSV files. The version of &lt;code&gt;wikiq&lt;/code&gt; that was used is provided here.&lt;/p&gt;
&lt;p&gt;Running Wikiq on a full dump of English Wikipedia in a reasonable amount of requires considerable computing resources. For this project, Wikiq was run on &lt;a href="https://itconnect.uw.edu/research/hpc/hyak-next-generation-supercomputer/"&gt;Hyak&lt;/a&gt; a high performance computer at the University of Washington. The code for doing so is highly speicific to Hyak. For transparency and in case it helps others using similar academic computers this code is included in &lt;code&gt;WikiqRunning.tar.gz&lt;/code&gt;. A copy of the wikiq output is included in this dataset in the multi-part archive &lt;code&gt;enwiki202003-wikiq.tar.gz&lt;/code&gt;. To extract this archive, download all the parts and then run &lt;code&gt;cat enwiki202003-wikiq.tar.gz* &gt; enwiki202003-wikiq.tar.gz &amp;&amp; tar xvzf enwiki202003-wikiq.tar.gz&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="obtaining-quality-labels-for-articles"&gt;Obtaining Quality Labels for Articles&lt;/h3&gt;
&lt;p&gt;We obtain up-to-date labels for each article using the &lt;code&gt;articlequality&lt;/code&gt; python package included in &lt;code&gt;articlequality.tar.gz&lt;/code&gt;.&lt;br /&gt;
The XML dumps are also the input to this step, and while it does not require a great deal of memory, a powerful computer (we used 28 cores) is helpful so that it completes in a reasonable amount of time. &lt;code&gt;extract_quality_labels.sh&lt;/code&gt; runs the command to extract the labels from the xml dumps. The resulting files have the format &lt;code&gt;data/enwiki-20200301-pages-meta-history*.xml-p*.7z_article_labelings.json&lt;/code&gt; and are included in this dataset in the archive &lt;code&gt;enwiki202003-article_labelings-json.tar.gz&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="taking-a-sample-of-quality-labels"&gt;Taking a Sample of Quality Labels&lt;/h3&gt;
&lt;p&gt;I used Apache Spark to merge the metadata from Wikiq with the quality labels and to draw a sample of articles where each quality class is equally represented. To improve the efficiency of this merge, I first converted the &lt;code&gt;wikiq&lt;/code&gt; tsv outputs into the &lt;code&gt;parquet&lt;/code&gt; columnar format using the script &lt;code&gt;wikiq_to_parquet.py&lt;/code&gt;. The parquet files are in this dataset in the multi-part archive &lt;code&gt;enwiki202003-wikiq-parquet.tar.gz&lt;/code&gt;. To extract this archive, download all the parts and then run &lt;code&gt;cat enwiki202003-wikiq-parquet.tar.gz* &gt; enwiki202003-wikiq-parquet.tar.gz &amp;&amp; tar xvzf enwiki202003-wikiq-parquet.tar.gz&lt;/code&gt;. It also saves the counts of revisions and articles by quality class to &lt;code&gt;remember_sample_quality_labels.RDS&lt;/code&gt; for computing the weights.&lt;/p&gt;
&lt;p&gt;Run &lt;code&gt;python3 sample_training_labels.py&lt;/code&gt; to draw the sample which is written to &lt;code&gt;20200301_article_labelings_sample.json&lt;/code&gt; and additional metadata is saved to &lt;code&gt;training_quality_labels.RDS&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="obtaining-ores-scores"&gt;Obtaining ORES Scores&lt;/h2&gt;
&lt;p&gt;Scoring the sample of labeled articles requires an internet connection to Wikipedia. The Python script &lt;code&gt;ores_scores_sample.py&lt;/code&gt; uses the &lt;code&gt;revscoring&lt;/code&gt; package to obtain quality scores for the sample. The bash script &lt;code&gt;ores_score_sample.sh&lt;/code&gt; runs the script. The scores are output to &lt;code&gt;scored_article_sample.feather&lt;/code&gt;, the same data is provided in &lt;code&gt;scored_article_sample.csv&lt;/code&gt;. This is the dataset used in the remaining analysis.&lt;/p&gt;
&lt;p&gt;If you run this step you should edit the &lt;code&gt;user_agent&lt;/code&gt; variable in &lt;code&gt;ores_score_sample.py&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="computing-weighted-datasets"&gt;Computing Weighted Datasets&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;load_data.R&lt;/code&gt; reads &lt;code&gt;remember_sample_quality_labels.RDS&lt;/code&gt; and &lt;code&gt;scored_article_sample.feather&lt;/code&gt; to finialize the weighted dataset. Each of the R scripts source this file to load the dataset.&lt;/p&gt;
&lt;h2 id="fitting-ordinal-regression-models"&gt;Fitting Ordinal Regression Models&lt;/h2&gt;
&lt;p&gt;All three bayesian regression models are fit in &lt;code&gt;ordinal_quality_models.R&lt;/code&gt;. The models are saved in the &lt;code&gt;models.tar.gz&lt;/code&gt; archive. The scripts fit the models in parallel and will use 12 cores.&lt;/p&gt;
&lt;h2 id="interpreting-the-models"&gt;Interpreting the Models&lt;/h2&gt;
&lt;p&gt;There are 3 separate scripts for analyzing each of the three models and the ORES model: &lt;code&gt;analyze_quality_models_articles.R&lt;/code&gt;, &lt;code&gt;analyze_quality_models_unweighted.R&lt;/code&gt; and &lt;code&gt;analyze_quality_models_revisions.R&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;For convenience, the script &lt;code&gt;run_ordinal_quality.sh&lt;/code&gt; runs the R scripts for fitting and interpreting all the models. These scripts output 4 &lt;code&gt;.RDS&lt;/code&gt; files that contain data for use by &lt;code&gt;knitr&lt;/code&gt; to insert the numbers, graphs, and tables in to the paper.&lt;/p&gt;
&lt;h2 id="building-the-paper"&gt;Building the Paper&lt;/h2&gt;
&lt;p&gt;Extract &lt;code&gt;paper.tar.gz&lt;/code&gt; and run &lt;code&gt;make&lt;/code&gt; in the resulting directory to build the paper using the data stored in the provided &lt;code&gt;.RDS&lt;/code&gt; files.&lt;/p&gt;
&lt;h1 id="calibrating-a-model-for-a-new-sample"&gt;Calibrating a Model for a New Sample&lt;/h1&gt;
TK