Repository: dissemin/wikiciteparser (language: python)

Issue:
Support for bulk processing of XML dumps
Hi! Thank you for creating and maintaining this helpful python library.

I am interested in generating snapshot wikipedia reference dumps, using the period bulk XML dumps and wikiciteparser. I am wondering if you know of an existing script, and whether you would accept a patch to this repository to support a helper script. If not, I can just create a separate small repository.

Another, already existing project, builds on wikictieparser to achieve the same goal using PySpark and generating parquet files, and was re-used as part of a recent "wcw" grant project: https://github.com/Harshdeep1996/cite-classifications-wiki

I'm thinking on doing something more like https://github.com/mediawiki-utilities/python-mwcites, which uses https://github.com/mediawiki-utilities/python-mwxml to iterate over the bulk dumps. I would then generate JSON lines files.


Discussion / hints:
Here is a quick example script. On my laptop this does only 20 revisions/second, but I think that is fine:

https://gist.github.com/bnewbold/c5ca0630b42ee3e17947221e015e421c
Hi! Sure, why not, we could also add this in this library :)

Fix patch:
diff --git a/requirements.txt b/requirements.txt
index 76329ec..022632f 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,2 +1,4 @@
 lupa
 mwparserfromhell
+mwxml
+mwtypes
diff --git a/setup.py b/setup.py
index a6e70da..dc1cb5f 100644
--- a/setup.py
+++ b/setup.py
@@ -70,7 +70,7 @@
     # your project is installed. For an analysis of "install_requires" vs pip's
     # requirements files see:
     # https://packaging.python.org/en/latest/requirements.html
-    install_requires=['lupa','mwparserfromhell'],
+    install_requires=['lupa','mwparserfromhell', 'mwxml', 'mwtypes'],
 
     # List additional groups of dependencies here (e.g. development
     # dependencies). You can install these using the following syntax,
diff --git a/wikiciteparser/bulk.py b/wikiciteparser/bulk.py
new file mode 100644
index 0000000..51bd340
--- /dev/null
+++ b/wikiciteparser/bulk.py
@@ -0,0 +1,91 @@
+"""
+Helper script for extracting references from bulk mediawiki XML dumps (eg,
+those from https://dumps.wikimedia.org/)
+
+With wikiciteparser installed, run this script like::
+
+    python3 -m wikiciteparser.bulk enwiki-20210801-pages-meta-current.xml.bz2 | pv -l | gzip > enwiki-20210801-pages-meta-current.refs.json.gz
+
+Depending on the size of the dump, it can take a long time to process all
+articles (eg, 10 revisions/second, days of processing for millions of
+articles).
+
+Output is JSON lines on stdout, with one line per page revision. JSON keys:
+
+    site_name (eg, 'enwiki')
+    page_title
+    revsion_id
+    refs (array of parsed references)
+"""
+
+import json
+import argparse
+from typing import Iterator
+
+import mwxml
+import mwtypes
+import mwtypes.files
+import mwparserfromhell
+
+from .parser import parse_citation_template
+
+
+def extract_revision(revision):
+    # type: (mwtypes.Revision) -> dict
+    meta = {}
+    meta["revision_id"] = revision.id
+    refs = []
+    wikicode = mwparserfromhell.parse(revision.text)
+    for tmpl in wikicode.filter_templates():
+        parsed = parse_citation_template(tmpl)
+        if parsed:
+            refs.append(parsed)
+    meta["refs"] = refs
+    return meta
+
+
+def extract_file(xml_file_path):
+    # type: (str) -> Iterator[dict]
+    """
+    Iterate over all revisions of all pages not in a namespace from the dump.
+    Yields dicts.
+
+    If we are processing one of the 'current' dumps, there will be only one
+    Revision per Page.
+    """
+
+    dump = mwxml.Dump.from_file(mwtypes.files.reader(xml_file_path))
+    site_name = dump.site_info.dbname
+
+    for page in dump.pages:
+        if (page.namespace not in [0, "0"]) or page.redirect:
+            # print(f"SKIPPED: [{page.namespace}] {page.title} redirect={page.redirect}", file=sys.stderr)
+            continue
+        for revision in page:
+            if revision.deleted.text or not revision.text:
+                continue
+            meta = extract_revision(revision)
+            meta["site_name"] = site_name
+            meta["page_title"] = page.title
+            yield meta
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter
+    )
+    parser.add_argument(
+        "xml_files",
+        help="input XML files to extract from. may be compessed (eg, .bz2)",
+        nargs="+",
+        type=str,
+    )
+    args = parser.parse_args()
+
+    for xml_path in args.xml_files:
+        for obj in extract_file(xml_path):
+            print(json.dumps(obj))
+
+
+if __name__ == "__main__":
+    main()


Test patch:
diff --git a/wikiciteparser/test_files/bulk_dump_sample.xml.bz2 b/wikiciteparser/test_files/bulk_dump_sample.xml.bz2
new file mode 100644
index 0000000..a1d45d2
Binary files /dev/null and b/wikiciteparser/test_files/bulk_dump_sample.xml.bz2 differ
diff --git a/wikiciteparser/tests.py b/wikiciteparser/tests.py
index 4404b33..c628cb5 100644
--- a/wikiciteparser/tests.py
+++ b/wikiciteparser/tests.py
@@ -3,6 +3,7 @@
 
 import unittest
 from wikiciteparser.parser import *
+from wikiciteparser.bulk import extract_file
 
 
 class ParsingTests(unittest.TestCase):
@@ -172,5 +173,12 @@ def test_mwtext(self):
         })
 
 
+class BulkXMLTests(unittest.TestCase):
+    def test_basic_xml_parse(self):
+        revs = list(extract_file('wikiciteparser/test_files/bulk_dump_sample.xml.bz2'))
+        self.assertEqual(len(revs), 3)
+        self.assertEqual(len(revs[0]['refs']), 3)
+
+
 if __name__ == '__main__':
         unittest.main()