Package: wikitextprocessor
wikitextprocessorThis is a Python package for processingWikiMedia dump
filesforWiktionary,Wikipedia, etc., for data extraction,
error checking, offline conversion into HTML or other formats, and
other uses.  Key features include:Parsing dump files, including built-in support for processing pages
in parallelWikitextsyntax
parser that converts the whole page into a parse treeExtracting template definitions andScribuntoLua module definitions from dump filesExpanding selected templates or all templates, and
heuristically identifying templates that need to be expanded before
parsing is reasonably possible (e.g., templates that emit table
start and end tags)Processing and expanding wikitext parser functionsProcessing, executing, and expanding Scribunto Lua modules (they are
very widely used in, e.g., Wiktionary, for example for generatingIPAstrings for many languages)Controlled expansion of parts of pages for applications that parse
overall page structure before parsing but then expand templates on
certain sections of the pageCapturing information from template arguments while expanding them,
as template arguments often contain useful information not available
in the expanded content.This module is primarily intended as a building block for other
packages that process Wikitionary or Wikipedia data, particularly for
data extraction.  You will need to write code to use this.For pre-existing extraction modules that use this package, please see:Wiktextractfor
extracting rich machine-readable dictionaries from Wiktionary.  You can also
find pre-extracted machine-readable Wiktionary data in JSON format atkaikki.org.Getting startedInstallingThe best way to install this package is frompypi:pip3 install wikitextprocessorAlternatively, you can install the master branch from github:git clone https://github.com/tatuylonen/wikitextprocessor
cd wikitextprocessor
pip3 install -e .Running testsThis package includes tests written using theunittestframework.
They can be run using, for example,nose, which can be installed
usingpip3 install nose.To run the tests, use the following command in the top-level directory:nosetestsObtaining WikiMedia dump filesThis package is primarily intended for processing Wiktionary and
Wikipedia dump files (though you can also use it for processing
individual pages or other files that are in wikitext format).  To
download WikiMedia dump files, go to thedump download
page.  We recommend
using the <name>-<date>-pages-articles.xml.bz2 files.API documentationUsage example:from wikitextprocessor import Wtp, WikiNode, NodeKind
   ctx = Wtp()

   def page_handler(model, title, text):
       if model != "wikitext" or title.startswith("Template:"):
           return None
       tree = ctx.parse(text, pre_expand=True)
       ... process parse tree
         ... value = ctx.node_to_wikitext(node)

   ctx.process("enwiktionary-20201201-pages-articles.xml.bz2", page_handler)The basic operation ofWtp.process()is as follows:Extract templates, modules, and other pages from the dump file and save
them in a temporary fileHeuristically analyze which templates need to be pre-expanded before
parsing to make sense of the page structure (this cannot detect templates
that call Lua code that outputs wikitext that affects parsed structure).
These first steps together are called the "first phase".Process the pages again, calling a page handler function for each page.
The page handler can extract, parse, and otherwise process the page, and
has full access to templates and Lua macros defined in the dump.  This may
call the page handler in multiple processes in parallel.  Return values
from the page handler calls are returned to the caller (this function acts
as an iterator).  This is called the second phase.Optionally, theWtp.reprocess()function may be used for processing the
same data several times (it basically repeats the second phase).Most of the functionality is hidden behind theWtpobject.WikiNodeobjects are used for representing the parse
tree that is returned by theWtp.parse()function.NodeKindis an enumeration type used to encode the type of aWikiNode.
Additionally,ALL_LANGUAGESis exported and is a list that
describes all languages (language codes, names, and other data) used
in Wiktionary.class Wtp(object)def __init__(self, num_threads=None, cache_file=None, quiet=False)The initializer can usually be called without arguments, but recognizes
the following arguments:num_threads- if set to an integer, use that many parallel processes
for processing the dump.  The default is to use as many processors as there
are available cores/hyperthreads.  You may need to limit the number of
parallel processes if you are limited by available memory; we have found
that processing Wiktionary (including templates and Lua macros)
requires 3-4GB of memory per process.  This MUST be set to 1 on Windows.cache_filecan normally beNone, in which case a temporary file will
be created under/tmp, or a path (str) for the cache file(s).
There are two reasons why you might want to
set this: 1) you don't have enough space on/tmp(the whole uncompressed
dump must fit there, which can easily be 10-20GB), or 2) for testing.
If you specify the cache file, if an existing cache file exists, that will be
loaded and used, eliminating the time needed for Phase 1 (this is very
important for testing, allowing processing single pages reasonably fast).
In this case, you should not callWtp.process()but instead useWtp.reprocess()or just callWtp.expand()orWtp.parse()on
wikitext that you have obtained otherwise (e.g., from some file).
If the cache file doesn't exist, you will need to callWtp.process()to parse a dump file, which will initialize the cache file during the
first phase.  If you wish to re-create cache file, you should remove
the old cache file first.  The cache file path is actually a prefix for
multiple individual files.quiet- if set to True, suppress progress messages during processingWindows and MacOS note:Settingnum_threadsto a value other than 1
probably doesn't currently work on Windows and MacOS.  It now defaults to 1
on these platforms.  This is because these platforms don't usefork()in
the Python multiprocessing package, and the current parallelization
implementation depends on this.def process(self, path, page_handler, phase1_only=False)This function processes a WikiMedia dump, uncompressing and extracing pages
(including templates and Lua modules), and callingWtp.add_page()for
each page (phase 1).  This then callsWtp.reprocess()to execute the
second phase.This takes the following arguments:path(str) - path to the WikiMedia dump file to be processed
(e.g., "enwiktionary-20201201-pages-articles.xml.bz2").  Note that the
compressed file can be used.  Dump files can be
downloadedhere.page_handler(function) - this function will be called for each page
in phase 2 (unlessphase1_onlyis set to True).  The call takes the formpage_handler(model, title, data), wheremodelis themodelvalue
for the page in the dump (wikitextfor normal wikitext pages and
templates,Scribuntofor Lua modules; other values are also possible),titleis page title (e.g.,sampleorTemplate:foobarorModule:mystic), anddatais the contents of the page (usually
wikitext).phase1_only(boolean) - if set to True, prevents phase 2
processing and thepage_handlerfunction will not be called.  TheWtp.reprocess()function can be used to run the second phase separately,
orWtp.expand(),Wtp.parse()and other functions can be used.This function returns an iterator over the values returned by thepage_handlerfunction (ifpage_handlerreturnsNoneor no
value, the iterator does not return those values).  Note thatpage_handlerwill usually be run in a separate process, and cannot
pass any values back in global variables.  It can, however, access
global variables assigned before callingWtp.process()(in Linux
only).def reprocess(self, page_handler, autoload=True)Iterates over all pages in the cache file and callspage_handlerfor each page.  This basically implements phase 2 of processing a dump
file (seeWtp.process()).  This can be called more than once if desired.The arguments are:page_handler(function) - this function will be called for every page.
The call takes the formpage_handler(model, title, data),
wheremodelis themodelvalue
for the page in the dump (wikitextfor normal wikitext pages and
templates,Scribuntofor Lua modules; other values are also possible),titleis page title (e.g.,sampleorTemplate:foobarorModule:mystic), anddatais the contents of the page (usually
wikitext).  Ifautoloadis set toFalse, thendatawill beNone.autoload(boolean) - Normally this function loads the page contents
automatically before calling the page handler.  Ifautoloadis set toFalse, then this will not automatically load the contents.  This will
make the iteration over the pages much faster, and is useful for scanning
all pages when only a small fraction of them are likely to be of interest.
TheWtp.read_by_title()function can then be used to load the page
contents.This function returns an iterator that iterates over the return values
ofpage_handler.  If the return value isNoneorpage_handlerreturns no value, no value is returned by the iterator for such calls.This calls thepage_handlerusing subprocesses (unlessnum_threadswas set to 1 in the initializer).  It may be necessary to set it to 1
on Windows and MacOS due to operating system/python limitations on those
platforms.def read_by_title(self, title):Reads the contents of the page with the specified title from the cache
file.  There is usually no need to call this function explicitly, asWtp.process()andWtp.reprocess()normally load the page
automatically.  However, this can be useful if callingWtp.reprocess()withautoloadset toFalse.  This function
does not automatically callWtp.start_page().Arguments are:title(str) - the title of the page to readThis returns the page contents as a string, orNoneif the page
does not exist.  If a transient page has been added with that title
(seeWtp.add_page()), then this returns the transient page.def parse(self, text, pre_expand=False, expand_all=False,
          additional_expand=None)Parses wikitext into a parse tree (WikiNode), optionally expanding
some or all the templates and Lua macros in the wikitext (using the definitions
for the templates and macros in the cache files, as added byWtp.process()or calls toWtp.add_page().TheWtp.start_page()function must be called before this function
to set the page title (which may be used by templates, Lua macros, and
error messages).  TheWtp.process()andWtp.reprocess()functions will call it automatically.This accepts the following arguments:text(str) - the wikitext to be parsedpre_expand(boolean) - if set toTrue, the templates that were
heuristically detected as affecting parsing (e.g., expanding to table start
or end tags or list items) will be automatically expanded before parsing.
Any Lua macros those templates use may also be called.expand_all- if set toTrue, expands all templates and Lua
macros in the wikitext before parsing.additional_expand(set orNone) - if this argument is provided, it
should be a set of template names that should be expanded in addition to
those specified by the other options (i.e., in addition to to the
heuristically detected templates ifpre_expandisTrueor just these
if it is false; this option is meaningless ifexpand_allis set toTrue).This returns the parse tree.  See below for a documentation of theWikiNodeclass used for representing the parse tree.def node_to_wikitext(self, node)Converts a part of a parse tree back to wikitext.node(WikiNode, str, list/tuple of these) - This is the part of the
parse tree that is to be converted back to wikitext.  We also allow
strings and lists, so thatnode.childrencan be used directly as
the argument.def expand(self, text, template_fn=None, post_template_fn=None,
           pre_expand=False, templates_to_expand=None,
           expand_parserfns=True, expand_invoke=True)Expands the selected templates, parser functions and Lua macros in the
given Wikitext.  This can selectively expand some or all templates.  This can
also capture the arguments and/or the expansion of any template as well as
substitute custom expansions instead of the default expansions.TheWtp.start_page()function must be called before this function to
set the page title (which may be used by templates and Lua macros).  TheWtp.process()andWtp.reprocess()will call it automatically.  The
page title is also used in error messages.The arguments are as follows:text(str) - the wikitext to be expandedtemplate_fn(function) - if set, this will be called astemplate_fn(name, args), wherename(str) is the name of the
template andargsis a dictionary containing arguments to the
template.  Positional arguments (and named arguments with numeric
names) will have integer keys in the dictionary, whereas other named
arguments will have their names as keys.  All values corresponding
to arguments are strings (after they have been expanded).  This
function can returnNoneto cause the template to be expanded in
the normal way, or a string that will be used instead of the
expansion of the template.  This can return""(empty string) to
expand the template to nothing.  This can also capture the template name
and its arguments.post_template_fn(function) - if set, this will be called
aspost_template_fn(name, ht, expansion)after the template has
been expanded in the normal way.  This can returnNoneto use the
default expansion, or a string to use a that string as the expansion.
This can also be used to capture the template, its arguments, and/or its
expansion.pre_expand(boolean) - if set toTrue, all templates that were
heuristically determined as needing to be expanded before parsing will be
expanded.templates_to_expand(Noneor set or dictionary) - if this is set,
these templates will be expanded in addition to any other templates that
have been specified to be expanded.  If a dictionary is provided, its keys
will be taken as the names of the templates to be expanded.  If this has not
been set or isNone, all templates will be expanded.expand_parserfns(boolean) - Normally, wikitext parser functions will
be expanded.  This can be set toFalseto prevent parser function
expansion.expand_invoke(boolean) - Normally, the#invokeparser function
(which calls a Lua module) will be expanded along with other parser
functions.  This can be set toFalseto prevent expansion of the#invokeparser function.def start_page(self, title)This function should be called before starting the processing of a new page
or file.  This saves the page title (which is frequently accessed by
templates, parser functions, and Lua macros).  The page title is also
used in error messages.TheWtp.process()andWtp.reprocess()functions will automatically
call this before calling the page handler for each page.  This needs to be
called manually when processing wikitext obtained from other sources.The arguments are as follows:title(str) - The page title.  For normal pages, there is usually no
prefix.  Templates typically haveTemplate:prefix and Lua modulesModule:prefix, and other prefixes are also used (e.g.,Thesaurus:).
This does not care about the form of the name, but some parser functions do.def start_section(self, title)Sets the title of the current section on the page.  This is
automatically reset toNonebyWtp.start_page().  The section
title is only used in error, warning, and debug messages.The arguments are:title(str) - the title of the section, orNoneto clear it.def start_subsection(self, title)Sets the title of the current subsection of the current section on the
page.  This is autimatically reset toNonebyWtp.start_page()andWtp.start_section().  The subsection title is only used in error,
warning, and debug messages.The arguments are:title(str) - the title of the subsection, orNoneto clear it.def add_page(self, model, title, text, transient=False)This function is used to add pages, templates, and modules for
processing.  There is usually no need to use this ifWtp.process()is used; however, this can be used to add templates and pages for
testing or other special processing needs.  This can also be used for
adding transient pages that are not stored in the cache file but could
be used for debugging or information extraction.  An example would be
overriding some Lua module from the dump file for debugging the Lua
code, or adding a new Lua module that can import other modules and
then dump data from them, for example to extract the category
hierarchy of Wiktionary pages.The arguments are:model(str) - the model value for the page (usuallywikitextfor normal pages and templates andScribuntofor Lua modules)title(str) - the title of the page to be added (normal pages typically
have no prefix in the title, templates begin withTemplate:, and Lua
modules begin withModule:)text(str) - the content of the page, template, or moduletransient(boolean) - normally, the added pages will be stored in
the cache file.  If this is set toTrue, the page will not be stored
in the cache file and will only affect the current run.  Transient
pages will override any pages in the cache file, and are useful for
debugging and data extraction.TheWtp.analyze_templates()function needs to be called after
callingWtp.add_page()before pages can be expanded or parsed (it should
preferably only be called once after adding all pages and templates).def analyze_templates(self)Analyzes the template definitions in the cache file and determines which
of them should be pre-expanded before parsing because they affect the
document structure significantly.  Some templates in, e.g., Wiktionary
expand to table start tags, table end tags, or list items, and parsing
results are generally much better if they are expanded before parsing.
The actual expansion only happens ifpre_expandor some other argument
toWtp.expand()orWtp.parse()tells them to do so.The analysis is heuristic and is not guaranteed to find every such template.
In particular, it cannot detect templates that call Lua modules that output
Wikitext control structures (there are several templates in Wiktionary that
call Lua code that outputs list items, for example).  Such templates may need
to be identified manually and specified as additional templates to expand.
Luckily, there seem to be relatively few such templates, at least in
Wiktionary.This function is automatically called byWtp.process()at the end of
phase 1.  An explicit call is only necessary ifWtp.add_page()has been
used by the application.Error handlingVarious functions in this module, includingWtp.parse()andWtp.expand()may generate errors and warnings.  Those will be displayed
onstdoutas well as collected inWtp.errors,Wtp.warnings, andWtp.debugs.  These fields will contain lists of dictionaries, where
each dictionary describes an error/warning/debug message.  The dictionary can
have the following keys (not all of them are always present):msg(str) - the error messagetrace(str orNone) - optional stacktrace where the error occurredtitle(str) - the page title on which the error occurredsection(str orNone) - the section where the error occurredsubsection(str orNone) - the subsection where the error occurredpath(tuple of str) - a path of title, template names, parser function
names, or Lua module/function names, giving information about where the
error occurred during expansion or parsing.The fields containing the error messages will be cleared by every call
toWtp.start_page()(including the implicit calls duringWtp.process()andWtp.reprocess()).  Thus, thepage_handlerfunction often returns these lists together with any
information extracted from the page, and they can be collected
together from the values returned by the iterators returned by these
functions.  TheWtp.to_return()function maybe useful for this.The following functions can be used for reporting errors.  These can
also be called by application code from within thepage_handlerfunction as well astemplate_fnandpost_template_fnfunctions
to report errors, warnings, and debug messages in a uniform way.def error(self, msg, trace=None)Reports an error message.  The error will be added toWtp.errorslist and
printed to stdout.  The arguments are:msg (str) - the error message (need not include page title or section)trace (str orNone) - an optional stack trace giving more information
about where the error occurreddef warning(self, msg, trace=None)Reports a warning message.  The warning will be added toWtp.warningslist
and printed to stdout.  The arguments are the same as forWtp.error().def debug(self, msg, trace=None)Reports a debug message.  The message will be added toWtp.debugslist
and printed to stdout.  The arguments are the same as forWtp.error().def to_return(self)Produces a dictionary containing the error, warning, and debug
messages fromWtp.  This would typically be called at the end of apage_handlerfunction and the value returned along with whatever
data was extracted from that page.  The error lists are reset byWtp.start_page()(including the implicit calls fromWtp.process()andWtp.reprocess()), so they should be saved
(e.g., by this call) for each page.  (Given the parallelism in
the processing of the pages, they cannot just be accumulated in the
subprocesses.)The returned dictionary contains the following keys:errors- a list of dictionaries describing any error messageswarnings- a list of dictionaries describing any warning messagesdebugs- a list of dictionaries describing any debug messages.class WikiNode(object)TheWikiNodeclass represents a parse tree node and is returned byWtp.parse().  This object can be printed or converted to a string
and will display a human-readable format that is suitable for
debugging purposes (at least for small parse trees).TheWikiNodeobjects have the following fields:kind(NodeKind, see below) - The type of the node.  This determines
how to interpret the other fields.children(list) - Contents of the node.  This is generally used when
the node has arbitrary size content, such as subsections, list items/sublists,
other HTML tags, etc.args(list or str, depending onkind) - Direct arguments to the
node.  This is used, for example, for templates, template arguments, parser
function arguments, and link arguments, in which case this is a list.
For some node types (e.g., list, list item, and HTML tag), this is
directly a string.attrs- A dictionary containing HTML attributes or a definition list
definition (under thedefkey).class NodeKind(enum.Enum)TheNodeKindtype is an enumerated value for parse tree (WikiNode)
node types.  Currently the following values are used (typically these
need to be prefixed byNodekind., e.g.,NodeKind.LEVEL2):ROOT- The root node of the parse tree.LEVEL2- Level 2 subtitle (==).  Theargsfield contains the title
andchildrenfield contains any contents that are within this sectionLEVEL3- Level 3 subtitle (===)LEVEL4- Level 4 subtitle (====)LEVEL5- Level 5 subtitle (=====)LEVEL6- Level 6 subtitle (======)ITALIC- Italic, content is inchildrenBOLD- Bold, content is inchildrenHLINE- A horizontal line (no arguments or children)LIST- Indicates a list.  Each list and sublist will start with
this kind of node.argswill contain the prefix used to open the
list (e.g.,"##"- note this is stored directly as a string
inargs).  List items will be stored inchildren.LIST_ITEM- A list item in the children of aLISTnode.argsis the prefix used to open the list item (same as for theLISTnode).
The contents of the list item (including any possible sublists) are inchildren.  If the list is a definition list (i.e., the prefix ends
in";"), thenchildrencontains the item label to be defined
andattrs["def"]contains the definition.PREFORMATTED- Preformatted text where markup is interpreted.  Content
is inchildren.  This is used for lines starting with a space in
wikitext.PRE- Preformatted text where markup is not interpreted.  Content
is inchildren.  This is indicated in wikitext by
<pre>...</pre>.LINK- An internal wikimedia link ([[...]] in wikitext).  The link
arguments are inargs.  This tag is also used for media inclusion.
Links with a trailing word end immediately after the link have the trailing
part inchildren.TEMPLATE- A template call (transclusion).  Template name is in the
first argument and template arguments in subsequent arguments inargs.
Thechildrenfield is not used.  In wikitext templates are marked up
as {{name|arg1|arg2|...}}.TEMPLATE_ARG- A template argument.  The argument name is in the first
item inargsfollowed by any subsequet arguments (normally at most two
items, but I've seen arguments with more - probably an error in those
template definitions).  Thechildrenfield is not used.  In wikitext
template arguments are marked up as {{{name|defval}}}.PARSER_FN- A parser function invocation.  This is also used for built-in
variables such as {{PAGENAME}}.  The parser function name is in the
first element ofargsand parser function arguments in subsequent
elements.URL- An external URL. The first argument is the URL.  The second
optional argument (inargs) is the display text.  Thechildrenfield is not used.TABLE- A table.  Content is inchildren.  In wikitext, a table
is encoded as {| ... |}.TABLE_CAPTION- A table caption.  This can only occur underTABLE.  The content is inchildren.  Theattrsfield contains
a dictionary of any HTML attributes given to the table.TABLE_ROW- A table row.  This can only occur underTABLE.  The
content is inchildren(normally the content would beTABLE_CELLorTABLE_HEADER_CELLnodes).  Theattrsfield contains a dictionary
of any HTML attributes given to the table row.TABLE_HEADER_CELL- A table header cell.  This can only occur underTABLE_ROW.  Content is in children.  Theattrsfield contains
a dictionary of any HTML attributes given to the table row.TABLE_CELL- A table cell.  This can only occur underTABLE_ROW.
Content is inchildren.  Theattrsfield contains a dictionary
of any HTML attributes given to the table row.MAGIC_WORD- A MediaWiki magic word.  The magic word is assigned
directly toargsas a string (i.e., not in a list).childrenis
not used.  An example of a magic word would be__NOTOC__.HTML- A HTML tag (or a matched pair of HTML tags).argsis the
name of the HTML tag directly (not in a list and always without a slash).attrsis set to a dictionary of any HTML attributes from the tag.
The contents of the HTML tag is inchildren.Expected performanceThis can generally process a few Wiktionary pages second per processor
core, including expansion of all templates, Lua macros, parsing the
full page, and analyzing the parse.  On a multi-core machine, this can
generally process a few dozen to a few hundred pages per second,
depending on the speed and the number of the cores.Most of the processing effort goes to expanding Lua macros.  You can
elect not to expand Lua macros, but they are used extensively in
Wiktionary and for important information.  Expanding templates and Lua
macros allows much more robust and complete data extraction, but does
not come cheap.Contributing and bug reportsPlease create an issue on github to report bugs or to contribute!