Repository: piskvorky/gensim

Issue:
Add interlinks to segment_wiki
### Idea
[Users ask about this feature](https://twitter.com/TheShubhanshu/status/929493213689335808), this is really useful to have [interlinks](https://en.wikipedia.org/wiki/Help:Link) in the dump to construct the graph of articles or use relation between articles in any way.

### What's need to implement
Add field `"section_interlinks" (list of str)` that contains a list of article titles referenced by this section. 



Discussion:
Thanks for creating this issue. 
The suggested step is great, but to make it more consistent with the overall structure of the output, we should not only use the string of the link text, but also the Wikipage title it points to.

Another suggestion would be to include the span of the matched text with the begin offset and the end position. This will result in getting a segmented corpus for free based on your technique. It can be later used to tokenize the section text with link text items as a single unit. 

So the final format may look like:

```
"section_interlinks":  [
("link string", "link wikipage title", offset, end)
]
```
The corpora and code included with gensim are restricted to topic modelling and unsupervised text processing. We're not aiming to be "everything for everybody".

Including other types of information (supervised labels, graph structure) is possible but needs to be clearly motivated.

@napsternxg how would you use this extra information? What is the intended application?
@piskvorky I understand the requirement to for gensim being focused on topic modelling and unsupervised text processing. 

The major application area is utilizing multi-word units in Wikipedia which are usually linked to other wiki pages - as components of topic models and other text processing. E.g. simple tokenization will split words like "Barack Obama" or "Natural Language Processing". Although, there is support for extracting Ngrams using the Phrases module, a more principled approach when processing wiki pages would be to identify these phrases as a single concept (which is very easy to do for Wikipedia). Mapping the wiki link text to the wiki page would allow for normalizing these phrases to a common concept in Wikipedia. E.g. LDA is both "Latent Dirichlet allocation" and "Linear Discriminant Analysis". This will help in reducing the vocabulary size. 

Finally, the motivation to allow offset and end values in the json data, was to help in overriding tokenization flaws, especially with biomedical and chemical names. 

These were the use cases I had in mind. I would be happy to see this feature since I have been quite impressed with the processing speed of algorithms in gensim, and the wikipedia dump parser appears to be very fast. 

Another alternative would be to use the `segment_all_articles` generator to add this feature as a post processing step using the `article_sections` variable. However, this would require that `article_sections` contains the original text and not the `filter_wiki` plain text. 
https://github.com/RaRe-Technologies/gensim/blob/07c3130283a7512f74293a18eff4344cdbe85f94/gensim/scripts/segment_wiki.py#L83
Thank you @napsternxg, maybe you'll try to implement this feature, this will be great!

I can have a look at it after December 15th. Will send a PR then. 

Resolving patch:
diff --git a/gensim/corpora/wikicorpus.py b/gensim/corpora/wikicorpus.py
--- a/gensim/corpora/wikicorpus.py
+++ b/gensim/corpora/wikicorpus.py
@@ -22,7 +22,6 @@
 
 """
 
-
 import bz2
 import logging
 import multiprocessing
@@ -45,7 +44,6 @@
 TOKEN_MIN_LEN = 2
 TOKEN_MAX_LEN = 15
 
-
 RE_P0 = re.compile(r'<!--.*?-->', re.DOTALL | re.UNICODE)
 """Comments."""
 RE_P1 = re.compile(r'<ref([> ].*?)(</ref>|/>)', re.DOTALL | re.UNICODE)
@@ -78,6 +76,8 @@
 """Categories."""
 RE_P15 = re.compile(r'\[\[([fF]ile:|[iI]mage)[^]]*(\]\])', re.UNICODE)
 """Remove File and Image templates."""
+RE_P16 = re.compile(r'\[{2}(.*?)\]{2}', re.UNICODE)
+"""Capture interlinks text and article linked"""
 
 IGNORED_NAMESPACES = [
     'Wikipedia', 'Category', 'File', 'Portal', 'Template',
@@ -93,34 +93,70 @@
 """
 
 
-def filter_wiki(raw):
+def find_interlinks(raw):
+    """Find all interlinks to other articles in the dump.
+
+    Parameters
+    ----------
+    raw : str
+        Unicode or utf-8 encoded string.
+
+    Returns
+    -------
+    dict
+        Mapping from the linked article to the actual text found.
+    """
+    filtered = filter_wiki(raw, promote_remaining=False, simplify_links=False)
+    interlinks_raw = re.findall(RE_P16, filtered)
+
+    interlinks = {}
+    for parts in [i.split('|') for i in interlinks_raw]:
+        actual_title = parts[0]
+        try:
+            interlink_text = parts[1]
+            interlinks[actual_title] = interlink_text
+        except IndexError:
+            interlinks[actual_title] = actual_title
+
+    legit_interlinks = {i: j for i, j in interlinks.items() if '[' not in i and ']' not in i}
+    return legit_interlinks
+
+
+def filter_wiki(raw, promote_remaining=True, simplify_links=True):
     """Filter out wiki markup from `raw`, leaving only text.
 
     Parameters
     ----------
     raw : str
         Unicode or utf-8 encoded string.
+    promote_remaining : bool
+        Whether uncaught markup should be promoted to plain text.
+    simplify_links : bool
+        Whether links should be simplified keeping only their description text.
 
     Returns
     -------
     str
         `raw` without markup.
-
     """
     # parsing of the wiki markup is not perfect, but sufficient for our purposes
     # contributions to improving this code are welcome :)
     text = utils.to_unicode(raw, 'utf8', errors='ignore')
     text = utils.decode_htmlentities(text)  # '&amp;nbsp;' --> '\xa0'
-    return remove_markup(text)
+    return remove_markup(text, promote_remaining, simplify_links)
 
 
-def remove_markup(text):
+def remove_markup(text, promote_remaining=True, simplify_links=True):
     """Filter out wiki markup from `text`, leaving only text.
 
     Parameters
     ----------
     text : str
         String containing markup.
+    promote_remaining : bool
+        Whether uncaught markup should be promoted to plain text.
+    simplify_links : bool
+        Whether links should be simplified keeping only their description text.
 
     Returns
     -------
@@ -145,8 +181,11 @@ def remove_markup(text):
         text = re.sub(RE_P11, '', text)  # remove all remaining tags
         text = re.sub(RE_P14, '', text)  # remove categories
         text = re.sub(RE_P5, '\\3', text)  # remove urls, keep description
-        text = re.sub(RE_P6, '\\2', text)  # simplify links, keep description only
+
+        if simplify_links:
+            text = re.sub(RE_P6, '\\2', text)  # simplify links, keep description only
         # remove table markup
+
         text = text.replace('||', '\n|')  # each table cell on a separate line
         text = re.sub(RE_P12, '\n', text)  # remove formatting lines
         text = re.sub(RE_P13, '\n\\3', text)  # leave only cell content
@@ -156,9 +195,9 @@ def remove_markup(text):
         if old == text or iters > 2:
             break
 
-    # the following is needed to make the tokenizer see '[[socialist]]s' as a single word 'socialists'
-    # TODO is this really desirable?
-    text = text.replace('[', '').replace(']', '')  # promote all remaining markup to plain text
+    if promote_remaining:
+        text = text.replace('[', '').replace(']', '')  # promote all remaining markup to plain text
+
     return text
 
 
@@ -333,7 +372,7 @@ def extract_pages(f, filter_namespaces=False):
                     text = None
 
             pageid = elem.find(pageid_path).text
-            yield title, text or "", pageid     # empty page will yield None
+            yield title, text or "", pageid  # empty page will yield None
 
             # Prune the element tree, as per
             # http://www.ibm.com/developerworks/xml/library/x-hiperfparse/
@@ -461,6 +500,7 @@ class WikiCorpus(TextCorpus):
     >>> MmCorpus.serialize('wiki_en_vocab200k.mm', wiki) # another 8h, creates a file in MatrixMarket format and mapping
 
     """
+
     def __init__(self, fname, processes=None, lemmatize=utils.has_pattern(), dictionary=None,
                  filter_namespaces=('0',), tokenizer_func=tokenize, article_min_tokens=ARTICLE_MIN_WORDS,
                  token_min_len=TOKEN_MIN_LEN, token_max_len=TOKEN_MAX_LEN, lower=True):
diff --git a/gensim/scripts/segment_wiki.py b/gensim/scripts/segment_wiki.py
old mode 100755
new mode 100644
--- a/gensim/scripts/segment_wiki.py
+++ b/gensim/scripts/segment_wiki.py
@@ -1,323 +1,377 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-#
-# Author: Jayant Jain <jayant@rare-technologies.com>
-# Copyright (C) 2016 RaRe Technologies
-
-"""
-CLI script for extracting plain text out of a raw Wikipedia dump. Input is an xml.bz2 file provided by MediaWiki \
-that looks like <LANG>wiki-<YYYYMMDD>-pages-articles.xml.bz2 or <LANG>wiki-latest-pages-articles.xml.bz2 \
-(e.g. 14 GB of https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2).
-
-It streams through all the XML articles using multiple cores (#cores - 1, by default), \
-decompressing on the fly and extracting plain text from the articles and their sections.
-
-For each extracted article, it prints its title, section names and plain text section contents, in json-line format.
-
-Examples
---------
-
-  python -m gensim.scripts.segment_wiki -h
-
-  python -m gensim.scripts.segment_wiki -f enwiki-latest-pages-articles.xml.bz2 -o enwiki-latest.json.gz
-
-Processing the entire English Wikipedia dump takes 1.7 hours (about 3 million articles per hour, \
-or 10 MB of XML per second) on an 8 core Intel i7-7700 @3.60GHz.
-
-You can then read the created output (~6.1 GB gzipped) with:
-
->>> # iterate over the plain text data we just created
->>> for line in smart_open('enwiki-latest.json.gz'):
->>>    # decode each JSON line into a Python dictionary object
->>>    article = json.loads(line)
->>>
->>>    # each article has a "title" and a list of "section_titles" and "section_texts".
->>>    print("Article title: %s" % article['title'])
->>>    for section_title, section_text in zip(article['section_titles'], article['section_texts']):
->>>        print("Section title: %s" % section_title)
->>>        print("Section text: %s" % section_text)
-"""
-
-import argparse
-import json
-import logging
-import multiprocessing
-import re
-import sys
-from xml.etree import cElementTree
-
-from gensim.corpora.wikicorpus import IGNORED_NAMESPACES, WikiCorpus, filter_wiki, get_namespace, utils
-from smart_open import smart_open
-
-
-logger = logging.getLogger(__name__)
-
-
-def segment_all_articles(file_path, min_article_character=200, workers=None):
-    """Extract article titles and sections from a MediaWiki bz2 database dump.
-
-    Parameters
-    ----------
-    file_path : str
-        Path to MediaWiki dump, typical filename is <LANG>wiki-<YYYYMMDD>-pages-articles.xml.bz2
-        or <LANG>wiki-latest-pages-articles.xml.bz2.
-
-    min_article_character : int, optional
-        Minimal number of character for article (except titles and leading gaps).
-
-    workers: int or None
-        Number of parallel workers, max(1, multiprocessing.cpu_count() - 1) if None.
-
-    Yields
-    ------
-    (str, list of (str, str))
-        Structure contains (title, [(section_heading, section_content), ...]).
-
-    """
-    with smart_open(file_path, 'rb') as xml_fileobj:
-        wiki_sections_corpus = _WikiSectionsCorpus(
-            xml_fileobj, min_article_character=min_article_character, processes=workers)
-        wiki_sections_corpus.metadata = True
-        wiki_sections_text = wiki_sections_corpus.get_texts_with_sections()
-        for article_title, article_sections in wiki_sections_text:
-            yield article_title, article_sections
-
-
-def segment_and_write_all_articles(file_path, output_file, min_article_character=200, workers=None):
-    """Write article title and sections to `output_file` (or stdout, if output_file is None).
-
-    The output format is one article per line, in json-line format with 3 fields::
-
-        'title' - title of article,
-        'section_titles' - list of titles of sections,
-        'section_texts' - list of content from sections.
-
-    Parameters
-    ----------
-    file_path : str
-        Path to MediaWiki dump, typical filename is <LANG>wiki-<YYYYMMDD>-pages-articles.xml.bz2
-        or <LANG>wiki-latest-pages-articles.xml.bz2.
-
-    output_file : str or None
-        Path to output file in json-lines format, or None for printing to stdout.
-
-    min_article_character : int, optional
-        Minimal number of character for article (except titles and leading gaps).
-
-    workers: int or None
-        Number of parallel workers, max(1, multiprocessing.cpu_count() - 1) if None.
-
-    """
-    if output_file is None:
-        outfile = getattr(sys.stdout, 'buffer', sys.stdout)  # we want write bytes, so for py3 we used 'buffer'
-    else:
-        outfile = smart_open(output_file, 'wb')
-
-    try:
-        article_stream = segment_all_articles(file_path, min_article_character, workers=workers)
-        for idx, (article_title, article_sections) in enumerate(article_stream):
-            output_data = {"title": article_title, "section_titles": [], "section_texts": []}
-            for section_heading, section_content in article_sections:
-                output_data["section_titles"].append(section_heading)
-                output_data["section_texts"].append(section_content)
-            if (idx + 1) % 100000 == 0:
-                logger.info("processed #%d articles (at %r now)", idx + 1, article_title)
-            outfile.write((json.dumps(output_data) + "\n").encode('utf-8'))
-    finally:
-        if output_file is not None:
-            outfile.close()
-
-
-def extract_page_xmls(f):
-    """Extract pages from a MediaWiki database dump.
-
-    Parameters
-    ----------
-    f : file
-        File descriptor of MediaWiki dump.
-
-    Yields
-    ------
-    str
-        XML strings for page tags.
-
-    """
-    elems = (elem for _, elem in cElementTree.iterparse(f, events=("end",)))
-
-    elem = next(elems)
-    namespace = get_namespace(elem.tag)
-    ns_mapping = {"ns": namespace}
-    page_tag = "{%(ns)s}page" % ns_mapping
-
-    for elem in elems:
-        if elem.tag == page_tag:
-            yield cElementTree.tostring(elem)
-            # Prune the element tree, as per
-            # http://www.ibm.com/developerworks/xml/library/x-hiperfparse/
-            # except that we don't need to prune backlinks from the parent
-            # because we don't use LXML.
-            # We do this only for <page>s, since we need to inspect the
-            # ./revision/text element. The pages comprise the bulk of the
-            # file, so in practice we prune away enough.
-            elem.clear()
-
-
-def segment(page_xml):
-    """Parse the content inside a page tag
-
-    Parameters
-    ----------
-    page_xml : str
-        Content from page tag.
-
-    Returns
-    -------
-    (str, list of (str, str))
-        Structure contains (title, [(section_heading, section_content)]).
-
-    """
-    elem = cElementTree.fromstring(page_xml)
-    filter_namespaces = ('0',)
-    namespace = get_namespace(elem.tag)
-    ns_mapping = {"ns": namespace}
-    text_path = "./{%(ns)s}revision/{%(ns)s}text" % ns_mapping
-    title_path = "./{%(ns)s}title" % ns_mapping
-    ns_path = "./{%(ns)s}ns" % ns_mapping
-    lead_section_heading = "Introduction"
-    top_level_heading_regex = r"\n==[^=].*[^=]==\n"
-    top_level_heading_regex_capture = r"\n==([^=].*[^=])==\n"
-
-    title = elem.find(title_path).text
-    text = elem.find(text_path).text
-    ns = elem.find(ns_path).text
-    if ns not in filter_namespaces:
-        text = None
-
-    if text is not None:
-        section_contents = re.split(top_level_heading_regex, text)
-        section_headings = [lead_section_heading] + re.findall(top_level_heading_regex_capture, text)
-        assert len(section_contents) == len(section_headings)
-    else:
-        section_contents = []
-        section_headings = []
-
-    section_contents = [filter_wiki(section_content) for section_content in section_contents]
-    sections = list(zip(section_headings, section_contents))
-    return title, sections
-
-
-class _WikiSectionsCorpus(WikiCorpus):
-    """Treat a wikipedia articles dump (<LANG>wiki-<YYYYMMDD>-pages-articles.xml.bz2
-    or <LANG>wiki-latest-pages-articles.xml.bz2) as a (read-only) corpus.
-
-    The documents are extracted on-the-fly, so that the whole (massive) dump can stay compressed on disk.
-
-    """
-    def __init__(self, fileobj, min_article_character=200, processes=None,
-                 lemmatize=utils.has_pattern(), filter_namespaces=('0',)):
-        """
-        Parameters
-        ----------
-        fileobj : file
-            File descriptor of MediaWiki dump.
-        min_article_character : int, optional
-            Minimal number of character for article (except titles and leading gaps).
-        processes : int, optional
-            Number of processes, max(1, multiprocessing.cpu_count() - 1) if None.
-        lemmatize : bool, optional
-            If `pattern` package is installed, use fancier shallow parsing to get token lemmas.
-            Otherwise, use simple regexp tokenization.
-        filter_namespaces : tuple of int, optional
-            Enumeration of namespaces that will be ignored.
-
-        """
-        self.fileobj = fileobj
-        self.filter_namespaces = filter_namespaces
-        self.metadata = False
-        if processes is None:
-            processes = max(1, multiprocessing.cpu_count() - 1)
-        self.processes = processes
-        self.lemmatize = lemmatize
-        self.min_article_character = min_article_character
-
-    def get_texts_with_sections(self):
-        """Iterate over the dump, returning titles and text versions of all sections of articles.
-
-        Notes
-        -----
-        Only articles of sufficient length are returned (short articles & redirects
-        etc are ignored).
-
-        Note that this iterates over the **texts**; if you want vectors, just use
-        the standard corpus interface instead of this function::
-
-        >>> for vec in wiki_corpus:
-        >>>     print(vec)
-
-        Yields
-        ------
-        (str, list of (str, str))
-            Structure contains (title, [(section_heading, section_content), ...]).
-
-        """
-        skipped_namespace, skipped_length, skipped_redirect = 0, 0, 0
-        total_articles, total_sections = 0, 0
-        page_xmls = extract_page_xmls(self.fileobj)
-        pool = multiprocessing.Pool(self.processes)
-        # process the corpus in smaller chunks of docs, because multiprocessing.Pool
-        # is dumb and would load the entire input into RAM at once...
-        for group in utils.chunkize(page_xmls, chunksize=10 * self.processes, maxsize=1):
-            for article_title, sections in pool.imap(segment, group):  # chunksize=10):
-                # article redirects are pruned here
-                if any(article_title.startswith(ignore + ':') for ignore in IGNORED_NAMESPACES):  # filter non-articles
-                    skipped_namespace += 1
-                    continue
-                if not sections or sections[0][1].lstrip().lower().startswith("#redirect"):  # filter redirect
-                    skipped_redirect += 1
-                    continue
-                if sum(len(body.strip()) for (_, body) in sections) < self.min_article_character:
-                    # filter stubs (incomplete, very short articles)
-                    skipped_length += 1
-                    continue
-
-                total_articles += 1
-                total_sections += len(sections)
-                yield (article_title, sections)
-        logger.info(
-            "finished processing %i articles with %i sections (skipped %i redirects, %i stubs, %i ignored namespaces)",
-            total_articles, total_sections, skipped_redirect, skipped_length, skipped_namespace)
-        pool.terminate()
-        self.length = total_articles  # cache corpus length
-
-
-if __name__ == "__main__":
-    logging.basicConfig(format='%(asctime)s : %(processName)s : %(levelname)s : %(message)s', level=logging.INFO)
-    logger.info("running %s", " ".join(sys.argv))
-
-    parser = argparse.ArgumentParser(formatter_class=argparse.RawTextHelpFormatter, description=globals()['__doc__'])
-    default_workers = max(1, multiprocessing.cpu_count() - 1)
-    parser.add_argument('-f', '--file', help='Path to MediaWiki database dump (read-only).', required=True)
-    parser.add_argument(
-        '-o', '--output',
-        help='Path to output file (stdout if not specified). If ends in .gz or .bz2, '
-             'the output file will be automatically compressed (recommended!).')
-    parser.add_argument(
-        '-w', '--workers',
-        help='Number of parallel workers for multi-core systems. Default: %(default)s.',
-        type=int,
-        default=default_workers
-    )
-    parser.add_argument(
-        '-m', '--min-article-character',
-        help="Ignore articles with fewer characters than this (article stubs). Default: %(default)s.",
-        default=200
-    )
-    args = parser.parse_args()
-
-    segment_and_write_all_articles(
-        args.file, args.output,
-        min_article_character=args.min_article_character,
-        workers=args.workers
-    )
-
-    logger.info("finished running %s", sys.argv[0])
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Author: Jayant Jain <jayant@rare-technologies.com>
+# Copyright (C) 2016 RaRe Technologies
+
+"""
+CLI script for extracting plain text out of a raw Wikipedia dump. Input is an xml.bz2 file provided by MediaWiki \
+that looks like <LANG>wiki-<YYYYMMDD>-pages-articles.xml.bz2 or <LANG>wiki-latest-pages-articles.xml.bz2 \
+(e.g. 14 GB of https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2).
+
+It streams through all the XML articles using multiple cores (#cores - 1, by default), \
+decompressing on the fly and extracting plain text from the articles and their sections.
+
+For each extracted article, it prints its title, section names and plain text section contents, in json-line format.
+
+Examples
+--------
+
+  python -m gensim.scripts.segment_wiki -h
+
+  python -m gensim.scripts.segment_wiki -f enwiki-latest-pages-articles.xml.bz2 -o enwiki-latest.json.gz
+
+Processing the entire English Wikipedia dump takes 1.7 hours (about 3 million articles per hour, \
+or 10 MB of XML per second) on an 8 core Intel i7-7700 @3.60GHz.
+
+You can then read the created output (~6.1 GB gzipped) with:
+
+>>> # iterate over the plain text data we just created
+>>> for line in smart_open('enwiki-latest.json.gz'):
+>>>    # decode each JSON line into a Python dictionary object
+>>>    article = json.loads(line)
+>>>
+>>>    # each article has a "title", a mapping of interlinks and a list of "section_titles" and "section_texts".
+>>>    print("Article title: %s" % article['title'])
+>>>    print("Interlinks: %s" + article['interlinks'])
+>>>    for section_title, section_text in zip(article['section_titles'], article['section_texts']):
+>>>        print("Section title: %s" % section_title)
+>>>        print("Section text: %s" % section_text)
+"""
+
+import argparse
+import json
+import logging
+import multiprocessing
+import re
+import sys
+from xml.etree import cElementTree
+from functools import partial
+
+from gensim.corpora.wikicorpus import IGNORED_NAMESPACES, WikiCorpus, filter_wiki, find_interlinks, get_namespace, utils
+from smart_open import smart_open
+
+logger = logging.getLogger(__name__)
+
+
+def segment_all_articles(file_path, min_article_character=200, workers=None, include_interlinks=False):
+    """Extract article titles and sections from a MediaWiki bz2 database dump.
+
+    Parameters
+    ----------
+    file_path : str
+        Path to MediaWiki dump, typical filename is <LANG>wiki-<YYYYMMDD>-pages-articles.xml.bz2
+        or <LANG>wiki-latest-pages-articles.xml.bz2.
+
+    min_article_character : int, optional
+        Minimal number of character for article (except titles and leading gaps).
+
+    workers: int or None
+        Number of parallel workers, max(1, multiprocessing.cpu_count() - 1) if None.
+
+    include_interlinks: bool
+        Whether or not interlinks should be included in the output
+
+    Yields
+    ------
+    (str, list of (str, str), (Optionally) dict of str: str)
+        Structure contains (title, [(section_heading, section_content), ...], (Optionally) {interlinks}).
+
+    """
+    with smart_open(file_path, 'rb') as xml_fileobj:
+        wiki_sections_corpus = _WikiSectionsCorpus(
+            xml_fileobj, min_article_character=min_article_character, processes=workers,
+            include_interlinks=include_interlinks)
+        wiki_sections_corpus.metadata = True
+        wiki_sections_text = wiki_sections_corpus.get_texts_with_sections()
+
+        for article in wiki_sections_text:
+            yield article
+
+
+def segment_and_write_all_articles(file_path, output_file, min_article_character=200, workers=None,
+                                   include_interlinks=False):
+    """Write article title and sections to `output_file` (or stdout, if output_file is None).
+
+    The output format is one article per line, in json-line format with 4 fields::
+
+        'title' - title of article,
+        'section_titles' - list of titles of sections,
+        'section_texts' - list of content from sections,
+        (Optional) 'section_interlinks' - list of interlinks in the article.
+
+    Parameters
+    ----------
+    file_path : str
+        Path to MediaWiki dump, typical filename is <LANG>wiki-<YYYYMMDD>-pages-articles.xml.bz2
+        or <LANG>wiki-latest-pages-articles.xml.bz2.
+
+    output_file : str or None
+        Path to output file in json-lines format, or None for printing to stdout.
+
+    min_article_character : int, optional
+        Minimal number of character for article (except titles and leading gaps).
+
+    workers: int or None
+        Number of parallel workers, max(1, multiprocessing.cpu_count() - 1) if None.
+
+    include_interlinks: bool
+        Whether or not interlinks should be included in the output
+    """
+    if output_file is None:
+        outfile = getattr(sys.stdout, 'buffer', sys.stdout)  # we want write bytes, so for py3 we used 'buffer'
+    else:
+        outfile = smart_open(output_file, 'wb')
+
+    try:
+        article_stream = segment_all_articles(file_path, min_article_character, workers=workers,
+                                              include_interlinks=include_interlinks)
+        for idx, article in enumerate(article_stream):
+            article_title, article_sections = article[0], article[1]
+            if include_interlinks:
+                interlinks = article[2]
+
+            output_data = {
+                "title": article_title,
+                "section_titles": [],
+                "section_texts": [],
+            }
+            if include_interlinks:
+                output_data["interlinks"] = interlinks
+
+            for section_heading, section_content in article_sections:
+                output_data["section_titles"].append(section_heading)
+                output_data["section_texts"].append(section_content)
+
+            if (idx + 1) % 100000 == 0:
+                logger.info("processed #%d articles (at %r now)", idx + 1, article_title)
+            outfile.write((json.dumps(output_data) + "\n").encode('utf-8'))
+
+    finally:
+        if output_file is not None:
+            outfile.close()
+
+
+def extract_page_xmls(f):
+    """Extract pages from a MediaWiki database dump.
+
+    Parameters
+    ----------
+    f : file
+        File descriptor of MediaWiki dump.
+
+    Yields
+    ------
+    str
+        XML strings for page tags.
+
+    """
+    elems = (elem for _, elem in cElementTree.iterparse(f, events=("end",)))
+
+    elem = next(elems)
+    namespace = get_namespace(elem.tag)
+    ns_mapping = {"ns": namespace}
+    page_tag = "{%(ns)s}page" % ns_mapping
+
+    for elem in elems:
+        if elem.tag == page_tag:
+            yield cElementTree.tostring(elem)
+            # Prune the element tree, as per
+            # http://www.ibm.com/developerworks/xml/library/x-hiperfparse/
+            # except that we don't need to prune backlinks from the parent
+            # because we don't use LXML.
+            # We do this only for <page>s, since we need to inspect the
+            # ./revision/text element. The pages comprise the bulk of the
+            # file, so in practice we prune away enough.
+            elem.clear()
+
+
+def segment(page_xml, include_interlinks=False):
+    """Parse the content inside a page tag
+
+    Parameters
+    ----------
+    page_xml : str
+        Content from page tag.
+
+    include_interlinks : bool
+        Whether or not interlinks should be parsed.
+
+    Returns
+    -------
+    (str, list of (str, str), (Optionally) dict of (str: str))
+        Structure contains (title, [(section_heading, section_content), ...], (Optionally) {interlinks}).
+
+    """
+    elem = cElementTree.fromstring(page_xml)
+    filter_namespaces = ('0',)
+    namespace = get_namespace(elem.tag)
+    ns_mapping = {"ns": namespace}
+    text_path = "./{%(ns)s}revision/{%(ns)s}text" % ns_mapping
+    title_path = "./{%(ns)s}title" % ns_mapping
+    ns_path = "./{%(ns)s}ns" % ns_mapping
+    lead_section_heading = "Introduction"
+    top_level_heading_regex = r"\n==[^=].*[^=]==\n"
+    top_level_heading_regex_capture = r"\n==([^=].*[^=])==\n"
+
+    title = elem.find(title_path).text
+    text = elem.find(text_path).text
+    ns = elem.find(ns_path).text
+    if ns not in filter_namespaces:
+        text = None
+
+    if text is not None:
+        if include_interlinks:
+            interlinks = find_interlinks(text)
+        section_contents = re.split(top_level_heading_regex, text)
+        section_headings = [lead_section_heading] + re.findall(top_level_heading_regex_capture, text)
+        section_headings = [heading.strip() for heading in section_headings]
+        assert len(section_contents) == len(section_headings)
+    else:
+        interlinks = []
+        section_contents = []
+        section_headings = []
+
+    section_contents = [filter_wiki(section_content) for section_content in section_contents]
+    sections = list(zip(section_headings, section_contents))
+
+    if include_interlinks:
+        return title, sections, interlinks
+    else:
+        return title, sections
+
+
+class _WikiSectionsCorpus(WikiCorpus):
+    """Treat a wikipedia articles dump (<LANG>wiki-<YYYYMMDD>-pages-articles.xml.bz2
+    or <LANG>wiki-latest-pages-articles.xml.bz2) as a (read-only) corpus.
+
+    The documents are extracted on-the-fly, so that the whole (massive) dump can stay compressed on disk.
+
+    """
+
+    def __init__(self, fileobj, min_article_character=200, processes=None,
+                 lemmatize=utils.has_pattern(), filter_namespaces=('0',), include_interlinks=False):
+        """
+        Parameters
+        ----------
+        fileobj : file
+            File descriptor of MediaWiki dump.
+        min_article_character : int, optional
+            Minimal number of character for article (except titles and leading gaps).
+        processes : int, optional
+            Number of processes, max(1, multiprocessing.cpu_count() - 1) if None.
+        lemmatize : bool, optional
+            If `pattern` package is installed, use fancier shallow parsing to get token lemmas.
+            Otherwise, use simple regexp tokenization.
+        filter_namespaces : tuple of int, optional
+            Enumeration of namespaces that will be ignored.
+        include_interlinks: bool
+            Whether or not interlinks should be included in the output
+
+        """
+        self.fileobj = fileobj
+        self.filter_namespaces = filter_namespaces
+        self.metadata = False
+        if processes is None:
+            processes = max(1, multiprocessing.cpu_count() - 1)
+        self.processes = processes
+        self.lemmatize = lemmatize
+        self.min_article_character = min_article_character
+        self.include_interlinks = include_interlinks
+
+    def get_texts_with_sections(self):
+        """Iterate over the dump, returning titles and text versions of all sections of articles.
+
+        Notes
+        -----
+        Only articles of sufficient length are returned (short articles & redirects
+        etc are ignored).
+
+        Note that this iterates over the **texts**; if you want vectors, just use
+        the standard corpus interface instead of this function::
+
+        >>> for vec in wiki_corpus:
+        >>>     print(vec)
+
+        Yields
+        ------
+        (str, list of (str, str), dict of (str: str))
+            Structure contains (title, [(section_heading, section_content), ...], (Optionally){interlinks}).
+
+        """
+        skipped_namespace, skipped_length, skipped_redirect = 0, 0, 0
+        total_articles, total_sections = 0, 0
+        page_xmls = extract_page_xmls(self.fileobj)
+        pool = multiprocessing.Pool(self.processes)
+        # process the corpus in smaller chunks of docs, because multiprocessing.Pool
+        # is dumb and would load the entire input into RAM at once...
+        for group in utils.chunkize(page_xmls, chunksize=10 * self.processes, maxsize=1):
+            for article in pool.imap(partial(segment, include_interlinks=self.include_interlinks),
+                                     group):  # chunksize=10): partial(merge_names, b='Sons')
+                article_title, sections = article[0], article[1]
+
+                # article redirects are pruned here
+                if any(article_title.startswith(ignore + ':') for ignore in IGNORED_NAMESPACES):  # filter non-articles
+                    skipped_namespace += 1
+                    continue
+                if not sections or sections[0][1].lstrip().lower().startswith("#redirect"):  # filter redirect
+                    skipped_redirect += 1
+                    continue
+                if sum(len(body.strip()) for (_, body) in sections) < self.min_article_character:
+                    # filter stubs (incomplete, very short articles)
+                    skipped_length += 1
+                    continue
+                total_articles += 1
+                total_sections += len(sections)
+
+                if self.include_interlinks:
+                    interlinks = article[2]
+                    yield (article_title, sections, interlinks)
+                else:
+                    yield (article_title, sections)
+
+        logger.info(
+            "finished processing %i articles with %i sections (skipped %i redirects, %i stubs, %i ignored namespaces)",
+            total_articles, total_sections, skipped_redirect, skipped_length, skipped_namespace)
+        pool.terminate()
+        self.length = total_articles  # cache corpus length
+
+
+if __name__ == "__main__":
+    logging.basicConfig(format='%(asctime)s : %(processName)s : %(levelname)s : %(message)s', level=logging.INFO)
+    logger.info("running %s", " ".join(sys.argv))
+
+    parser = argparse.ArgumentParser(formatter_class=argparse.RawTextHelpFormatter, description=globals()['__doc__'])
+    default_workers = max(1, multiprocessing.cpu_count() - 1)
+    parser.add_argument('-f', '--file', help='Path to MediaWiki database dump (read-only).', required=True)
+    parser.add_argument(
+        '-o', '--output',
+        help='Path to output file (stdout if not specified). If ends in .gz or .bz2, '
+             'the output file will be automatically compressed (recommended!).')
+    parser.add_argument(
+        '-w', '--workers',
+        help='Number of parallel workers for multi-core systems. Default: %(default)s.',
+        type=int,
+        default=default_workers
+    )
+    parser.add_argument(
+        '-m', '--min-article-character',
+        help="Ignore articles with fewer characters than this (article stubs). Default: %(default)s.",
+        default=200
+    )
+    parser.add_argument(
+        '-i', '--include-interlinks',
+        help='Include a mapping for interlinks to other articles in the dump. The mappings format is: '
+             '"interlinks": {"article_title_1": "interlink_text_1", "article_title_2": "interlink_text_2", ...}',
+        action='store_true'
+    )
+    args = parser.parse_args()
+
+    segment_and_write_all_articles(
+        args.file, args.output,
+        min_article_character=args.min_article_character,
+        workers=args.workers,
+        include_interlinks=args.include_interlinks
+    )
+
+    logger.info("finished running %s", sys.argv[0])


Test patch:
diff --git a/gensim/test/test_scripts.py b/gensim/test/test_scripts.py
new file mode 100644
--- /dev/null
+++ b/gensim/test/test_scripts.py
@@ -0,0 +1,108 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2018 Manos Stergiadis <em.stergiadis@gmail.com>
+# Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html
+
+"""
+Automated tests for checking the output of gensim.scripts.
+"""
+
+from __future__ import unicode_literals
+
+import json
+import logging
+import os.path
+import unittest
+
+from gensim.scripts.segment_wiki import segment_all_articles, segment_and_write_all_articles
+from smart_open import smart_open
+from gensim.test.utils import datapath, get_tmpfile
+
+
+class TestSegmentWiki(unittest.TestCase):
+
+    def setUp(self):
+        self.fname = datapath('enwiki-latest-pages-articles1.xml-p000000010p000030302-shortened.bz2')
+        self.expected_title = 'Anarchism'
+        self.expected_section_titles = [
+            'Introduction',
+            'Etymology and terminology',
+            'History',
+            'Anarchist schools of thought',
+            'Internal issues and debates',
+            'Topics of interest',
+            'Criticisms',
+            'References',
+            'Further reading',
+            'External links'
+        ]
+
+    def tearDown(self):
+        # remove all temporary test files
+        fname = get_tmpfile('script.tst')
+        extensions = ['', '.json']
+        for ext in extensions:
+            try:
+                os.remove(fname + ext)
+            except OSError:
+                pass
+
+    def test_segment_all_articles(self):
+        title, sections, interlinks = next(segment_all_articles(self.fname, include_interlinks=True))
+
+        # Check title
+        self.assertEqual(title, self.expected_title)
+
+        # Check section titles
+        section_titles = [s[0] for s in sections]
+        self.assertEqual(section_titles, self.expected_section_titles)
+
+        # Check text
+        first_section_text = sections[0][1]
+        first_sentence = "'''Anarchism''' is a political philosophy that advocates self-governed societies"
+        self.assertTrue(first_sentence in first_section_text)
+
+        # Check interlinks
+        self.assertTrue(interlinks['self-governance'] == 'self-governed')
+        self.assertTrue(interlinks['Hierarchy'] == 'hierarchical')
+        self.assertTrue(interlinks['Pierre-Joseph Proudhon'] == 'Proudhon')
+
+    def test_generator_len(self):
+        expected_num_articles = 106
+        num_articles = sum(1 for x in segment_all_articles(self.fname))
+
+        self.assertEqual(num_articles, expected_num_articles)
+
+    def test_json_len(self):
+        tmpf = get_tmpfile('script.tst.json')
+        segment_and_write_all_articles(self.fname, tmpf, workers=1)
+
+        expected_num_articles = 106
+        num_articles = sum(1 for line in smart_open(tmpf))
+        self.assertEqual(num_articles, expected_num_articles)
+
+    def test_segment_and_write_all_articles(self):
+        tmpf = get_tmpfile('script.tst.json')
+        segment_and_write_all_articles(self.fname, tmpf, workers=1, include_interlinks=True)
+
+        # Get the first line from the text file we created.
+        with open(tmpf) as f:
+            first = next(f)
+
+        # decode JSON line into a Python dictionary object
+        article = json.loads(first)
+        title, section_titles, interlinks = article['title'], article['section_titles'], article['interlinks']
+
+        self.assertEqual(title, self.expected_title)
+        self.assertEqual(section_titles, self.expected_section_titles)
+
+        # Check interlinks
+        self.assertTrue(interlinks['self-governance'] == 'self-governed')
+        self.assertTrue(interlinks['Hierarchy'] == 'hierarchical')
+        self.assertTrue(interlinks['Pierre-Joseph Proudhon'] == 'Proudhon')
+
+
+if __name__ == '__main__':
+    logging.basicConfig(level=logging.DEBUG)
+    unittest.main()