Repository: piskvorky/gensim

Issue:
WikiCorpus Tokenization issue
<!--
If your issue is a usage or a general question, please submit it here instead:
- Mailing List: https://groups.google.com/forum/#!forum/gensim
For more information, see Recipes&FAQ: https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ
-->

<!-- Instructions For Filing a Bug: https://github.com/RaRe-Technologies/gensim/blob/develop/CONTRIBUTING.md -->

#### Description
Currently the **wikicorpus.py** file houses the  class **WikiCorpus** and the **tokenize** function used while processing a wikipedia dump.

```
def tokenize(content):
    """
    Tokenize a piece of text from wikipedia. The input string `content` is assumed
    to be mark-up free (see `filter_wiki()`).

    Return list of tokens as utf8 bytestrings. Ignore words shorted than 2 or longer
    that 15 characters (not bytes!).
    """
    # TODO maybe ignore tokens with non-latin characters? (no chinese, arabic, russian etc.)
    return [
        utils.to_unicode(token) for token in utils.tokenize(content, lower=True, errors='ignore')
        if 2 <= len(token) <= 15 and not token.startswith('_')
    ]

```
The **tokenize** function in **wikicorpus.py** imposes certain hard coded filters which there doesnt seem to be a way to control from the outside.

This **tokenize** function in turn calls a **tokenize** function from **utils.py** which uses a regex to tokenize text.

```
PAT_ALPHABETIC = re.compile('(((?![\d])\w)+)', re.UNICODE)

def tokenize(text, lowercase=False, deacc=False, errors="strict", to_lower=False, lower=False):
    """
    Iteratively yield tokens as unicode strings, removing accent marks
    and optionally lowercasing the unidoce string by assigning True
    to one of the parameters, lowercase, to_lower, or lower.

    Input text may be either unicode or utf8-encoded byte string.

    The tokens on output are maximal contiguous sequences of alphabetic
    characters (no digits!).

    >>> list(tokenize('Nic nemůže letět rychlostí vyšší, než 300 tisíc kilometrů za sekundu!', deacc = True))
    [u'Nic', u'nemuze', u'letet', u'rychlosti', u'vyssi', u'nez', u'tisic', u'kilometru', u'za', u'sekundu']

    """
    lowercase = lowercase or to_lower or lower
    text = to_unicode(text, errors=errors)
    if lowercase:
        text = text.lower()
    if deacc:
        text = deaccent(text)
    for match in PAT_ALPHABETIC.finditer(text):
        yield match.group()

```


Now this regex will not work for languages like Japanese which need a tokenizer like **mecab** since word boundaries are not explicitly defined.

This causes issues when trying to use the text for certain use cases where unaltered text is needed.

#### Steps/Code/Corpus to Reproduce

```
import logging
from gensim.corpora import WikiCorpus

def get_text_from_wiki_bz(wiki_bz_path, wiki_text_path, log_status_every=10000):
    """
    convert the bz wiki dump to text where each line is one article
    :param wiki_bz_path: path to the wiki bz dump file
    :param wiki_text_path: path to the output file
    :param log_status_every: log the status after this count. set to None if no logs are required
    :return: 
    """
    wiki = WikiCorpus(wiki_bz_path, lemmatize=False, dictionary={})

    i = 0
    with open(wiki_text_path, 'w') as fp:
        for text in wiki.get_texts():
                fp.write(" ".join(text) + "\n")
                i = i + 1
                if log_status_every and i % log_status_every == 0:
                    logging.info("Saved " + str(i) + " articles")
        logging.info("Finished Saved " + str(i) + " articles")

    return True
```

#### Expected Results
Should be able to control tokenization by passing a reference to a custom function as an optional parameter to the **WikiCorpus** object

#### Actual Results
Tokenization is broken for languages like Japanese and there is no control  over the hard coded rules.


#### Versions
Linux-4.10.0-32-generic-x86_64-with-Ubuntu-16.04-xenial
('Python', '2.7.12 (default, Nov 19 2016, 06:48:10) \n[GCC 5.4.0 20160609]')
('NumPy', '1.13.1')
('SciPy', '0.19.1')
('gensim', '2.2.0')
('FAST_VERSION', 1)



Discussion:
Agreed, we need more flexibility there! Related to #1489 (same thing for Thai).

Can you open a PR with a fix @roopalgarg ?
Sounds good. Will do.

Resolving patch:
diff --git a/gensim/corpora/wikicorpus.py b/gensim/corpora/wikicorpus.py
--- a/gensim/corpora/wikicorpus.py
+++ b/gensim/corpora/wikicorpus.py
@@ -36,6 +36,10 @@
 # ignore articles shorter than ARTICLE_MIN_WORDS characters (after full preprocessing)
 ARTICLE_MIN_WORDS = 50
 
+# default thresholds for lengths of individual tokens
+TOKEN_MIN_LEN = 2
+TOKEN_MAX_LEN = 15
+
 
 RE_P0 = re.compile('<!--.*?-->', re.DOTALL | re.UNICODE)  # comments
 RE_P1 = re.compile('<ref([> ].*?)(</ref>|/>)', re.DOTALL | re.UNICODE)  # footnotes
@@ -163,18 +167,19 @@ def remove_file(s):
     return s
 
 
-def tokenize(content):
+def tokenize(content, token_min_len=TOKEN_MIN_LEN, token_max_len=TOKEN_MAX_LEN, lower=True):
     """
     Tokenize a piece of text from wikipedia. The input string `content` is assumed
     to be mark-up free (see `filter_wiki()`).
 
-    Return list of tokens as utf8 bytestrings. Ignore words shorted than 2 or longer
-    that 15 characters (not bytes!).
+    Set `token_min_len`, `token_max_len` as character length (not bytes!) thresholds for individual tokens.
+
+    Return list of tokens as utf8 bytestrings.
     """
     # TODO maybe ignore tokens with non-latin characters? (no chinese, arabic, russian etc.)
     return [
-        utils.to_unicode(token) for token in utils.tokenize(content, lower=True, errors='ignore')
-        if 2 <= len(token) <= 15 and not token.startswith('_')
+        utils.to_unicode(token) for token in utils.tokenize(content, lower=lower, errors='ignore')
+        if token_min_len <= len(token) <= token_max_len and not token.startswith('_')
     ]
 
 
@@ -238,17 +243,20 @@ def extract_pages(f, filter_namespaces=False):
 _extract_pages = extract_pages  # for backward compatibility
 
 
-def process_article(args):
+def process_article(args, tokenizer_func=tokenize, token_min_len=TOKEN_MIN_LEN, token_max_len=TOKEN_MAX_LEN, lower=True):
     """
     Parse a wikipedia article, returning its content as a list of tokens
     (utf8-encoded strings).
+
+    Set `tokenizer_func` (defaults to `tokenize`) parameter for languages like japanese or thai to perform better
+    tokenization. The `tokenizer_func` needs to take 4 parameters: (text, token_min_len, token_max_len, lower).
     """
     text, lemmatize, title, pageid = args
     text = filter_wiki(text)
     if lemmatize:
         result = utils.lemmatize(text)
     else:
-        result = tokenize(text)
+        result = tokenizer_func(text, token_min_len, token_max_len, lower)
     return result, title, pageid
 
 
@@ -257,6 +265,18 @@ def init_to_ignore_interrupt():
     signal.signal(signal.SIGINT, signal.SIG_IGN)
 
 
+def _process_article(args):
+    """Should not be called explicitly. Use `process_article` instead."""
+
+    tokenizer_func, token_min_len, token_max_len, lower = args[-1]
+    args = args[:-1]
+
+    return process_article(
+        args, tokenizer_func=tokenizer_func, token_min_len=token_min_len,
+        token_max_len=token_max_len, lower=lower
+    )
+
+
 class WikiCorpus(TextCorpus):
     """
     Treat a wikipedia articles dump (<LANG>wiki-<YYYYMMDD>-pages-articles.xml.bz2 or <LANG>wiki-latest-pages-articles.xml.bz2) as a (read-only) corpus.
@@ -272,8 +292,9 @@ class WikiCorpus(TextCorpus):
     >>> MmCorpus.serialize('wiki_en_vocab200k.mm', wiki) # another 8h, creates a file in MatrixMarket format plus file with id->word
 
     """
-
-    def __init__(self, fname, processes=None, lemmatize=utils.has_pattern(), dictionary=None, filter_namespaces=('0',)):
+    def __init__(self, fname, processes=None, lemmatize=utils.has_pattern(), dictionary=None,
+                 filter_namespaces=('0',), tokenizer_func=tokenize, article_min_tokens=ARTICLE_MIN_WORDS,
+                 token_min_len=TOKEN_MIN_LEN, token_max_len=TOKEN_MAX_LEN, lower=True):
         """
         Initialize the corpus. Unless a dictionary is provided, this scans the
         corpus once, to determine its vocabulary.
@@ -283,6 +304,18 @@ def __init__(self, fname, processes=None, lemmatize=utils.has_pattern(), diction
         this automatic logic by forcing the `lemmatize` parameter explicitly.
         self.metadata if set to true will ensure that serialize will write out article titles to a pickle file.
 
+        Set `article_min_tokens` as a min threshold for article token count (defaults to 50). Any article below this is
+        ignored.
+
+        Set `tokenizer_func` (defaults to `tokenize`) with a custom function reference to control tokenization else use
+        the default regexp tokenization. Set this parameter for languages like japanese or thai to perform better
+        tokenization. The `tokenizer_func` needs to take 4 parameters: (text, token_min_len, token_max_len, lower). The
+        parameter values are as configured on the class instance by default.
+
+        Set `lower` to control if everything should be converted to lowercase or not (default True).
+
+        Set `token_min_len`, `token_max_len` as thresholds for token lengths that are returned (default to 2 and 15).
+
         """
         self.fname = fname
         self.filter_namespaces = filter_namespaces
@@ -291,6 +324,12 @@ def __init__(self, fname, processes=None, lemmatize=utils.has_pattern(), diction
             processes = max(1, multiprocessing.cpu_count() - 1)
         self.processes = processes
         self.lemmatize = lemmatize
+        self.tokenizer_func = tokenizer_func
+        self.article_min_tokens = article_min_tokens
+        self.token_min_len = token_min_len
+        self.token_max_len = token_max_len
+        self.lower = lower
+
         if dictionary is None:
             self.dictionary = Dictionary(self.get_texts())
         else:
@@ -302,7 +341,7 @@ def get_texts(self):
         of tokens.
 
         Only articles of sufficient length are returned (short articles & redirects
-        etc are ignored).
+        etc are ignored). This is control by `article_min_tokens` on the class instance.
 
         Note that this iterates over the **texts**; if you want vectors, just use
         the standard corpus interface instead of this function::
@@ -310,23 +349,27 @@ def get_texts(self):
         >>> for vec in wiki_corpus:
         >>>     print(vec)
         """
+
         articles, articles_all = 0, 0
         positions, positions_all = 0, 0
-        texts = (
-            (text, self.lemmatize, title, pageid) for title, text, pageid
-            in extract_pages(bz2.BZ2File(self.fname), self.filter_namespaces)
-        )
+
+        tokenization_params = (self.tokenizer_func, self.token_min_len, self.token_max_len, self.lower)
+        texts = \
+            ((text, self.lemmatize, title, pageid, tokenization_params)
+             for title, text, pageid
+             in extract_pages(bz2.BZ2File(self.fname), self.filter_namespaces))
         pool = multiprocessing.Pool(self.processes, init_to_ignore_interrupt)
 
         try:
             # process the corpus in smaller chunks of docs, because multiprocessing.Pool
             # is dumb and would load the entire input into RAM at once...
             for group in utils.chunkize(texts, chunksize=10 * self.processes, maxsize=1):
-                for tokens, title, pageid in pool.imap(process_article, group):
+                for tokens, title, pageid in pool.imap(_process_article, group):
                     articles_all += 1
                     positions_all += len(tokens)
                     # article redirects and short stubs are pruned here
-                    if len(tokens) < ARTICLE_MIN_WORDS or any(title.startswith(ignore + ':') for ignore in IGNORED_NAMESPACES):
+                    if len(tokens) < self.article_min_tokens or \
+                            any(title.startswith(ignore + ':') for ignore in IGNORED_NAMESPACES):
                         continue
                     articles += 1
                     positions += len(tokens)


Test patch:
diff --git a/gensim/test/test_wikicorpus.py b/gensim/test/test_wikicorpus.py
--- a/gensim/test/test_wikicorpus.py
+++ b/gensim/test/test_wikicorpus.py
@@ -14,7 +14,7 @@
 import unittest
 
 from gensim.corpora.wikicorpus import WikiCorpus
-
+from gensim import utils
 
 module_path = os.path.dirname(__file__)  # needed because sample data files are located in the same folder
 datapath = lambda fname: os.path.join(module_path, 'test_data', fname)
@@ -24,6 +24,13 @@
 logger = logging.getLogger(__name__)
 
 
+def custom_tokeiner(content, token_min_len=2, token_max_len=15, lower=True):
+    return [
+        utils.to_unicode(token.lower()) if lower else utils.to_unicode(token) for token in content.split()
+        if token_min_len <= len(token) <= token_max_len and not token.startswith('_')
+    ]
+
+
 class TestWikiCorpus(unittest.TestCase):
 
     # #TODO: sporadic failure to be investigated
@@ -61,6 +68,73 @@ def test_unicode_element(self):
         texts = wc.get_texts()
         self.assertTrue(u'папа' in next(texts))
 
+    def test_lower_case_set_true(self):
+        """
+        set the parameter lower to True and check that upper case 'Anarchism' token doesnt exist
+        """
+        wc = WikiCorpus(datapath(FILENAME), processes=1, lower=True, lemmatize=False)
+        l = wc.get_texts()
+        list_tokens = next(l)
+        self.assertTrue(u'Anarchism' not in list_tokens)
+        self.assertTrue(u'anarchism' in list_tokens)
+
+    def test_lower_case_set_false(self):
+        """
+        set the parameter lower to False and check that upper case Anarchism' token exist
+        """
+        wc = WikiCorpus(datapath(FILENAME), processes=1, lower=False, lemmatize=False)
+        l = wc.get_texts()
+        list_tokens = next(l)
+        self.assertTrue(u'Anarchism' in list_tokens)
+        self.assertTrue(u'anarchism' in list_tokens)
+
+    def test_min_token_len_not_set(self):
+        """
+        don't set the parameter token_min_len and check that 'a' as a token doesn't exists
+        default token_min_len=2
+        """
+        wc = WikiCorpus(datapath(FILENAME), processes=1, lemmatize=False)
+        l = wc.get_texts()
+        self.assertTrue(u'a' not in next(l))
+
+    def test_min_token_len_set(self):
+        """
+        set the parameter token_min_len to 1 and check that 'a' as a token exists
+        """
+        wc = WikiCorpus(datapath(FILENAME), processes=1, token_min_len=1, lemmatize=False)
+        l = wc.get_texts()
+        self.assertTrue(u'a' in next(l))
+
+    def test_max_token_len_not_set(self):
+        """
+        don't set the parameter token_max_len and check that 'collectivisation' as a token doesn't exists
+        default token_max_len=15
+        """
+        wc = WikiCorpus(datapath(FILENAME), processes=1, lemmatize=False)
+        l = wc.get_texts()
+        self.assertTrue(u'collectivization' not in next(l))
+
+    def test_max_token_len_set(self):
+        """
+        set the parameter token_max_len to 16 and check that 'collectivisation' as a token exists
+        """
+        wc = WikiCorpus(datapath(FILENAME), processes=1, token_max_len=16, lemmatize=False)
+        l = wc.get_texts()
+        self.assertTrue(u'collectivization' in next(l))
+
+    def test_custom_tokenizer(self):
+        """
+        define a custom tokenizer function and use it
+        """
+        wc = WikiCorpus(datapath(FILENAME), processes=1, lemmatize=False, tokenizer_func=custom_tokeiner,
+                        token_max_len=16, token_min_len=1, lower=False)
+        l = wc.get_texts()
+        list_tokens = next(l)
+        self.assertTrue(u'Anarchism' in list_tokens)
+        self.assertTrue(u'collectivization' in list_tokens)
+        self.assertTrue(u'a' in list_tokens)
+        self.assertTrue(u'i.e.' in list_tokens)
+
 
 if __name__ == '__main__':
     logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.DEBUG)