Repository: piskvorky/gensim
Pull request: Remove `pattern` dependency
- Fix #2697 



Problem statement:
Cannot use gensim 3.8.x when `nltk` package is installed
#### Problem description

> What are you trying to achieve? What is the expected result? What are you seeing instead?

In my script i'm trying to `import gensim.models.keyedvectors` and also import another package, that requires `nltk` package internally. Whenever i have NLTK installed in the same virtualenv (i'm not using virtualenv, but a docker image actually) - the gensim model fails to import.

#### Steps/code/corpus to reproduce

```
# pip list | grep -E 'gensim|nltk'
gensim                        3.8.1

# pip install nltk
Processing /root/.cache/pip/wheels/96/86/f6/68ab24c23f207c0077381a5e3904b2815136b879538a24b483/nltk-3.4.5-cp36-none-any.whl
Requirement already satisfied: six in /usr/local/lib/python3.6/site-packages (from nltk) (1.13.0)
Installing collected packages: nltk
Successfully installed nltk-3.4.5

# pip list | grep -E 'gensim|nltk'
gensim                        3.8.1
nltk                          3.4.5

# python
Python 3.6.8 (default, Jun 11 2019, 01:16:11)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/gensim/__init__.py", line 5, in <module>
    from gensim import parsing, corpora, matutils, interfaces, models, similarities, summarization, utils  # noqa:F401
  File "/usr/local/lib/python3.6/site-packages/gensim/corpora/__init__.py", line 14, in <module>
    from .wikicorpus import WikiCorpus  # noqa:F401
  File "/usr/local/lib/python3.6/site-packages/gensim/corpora/wikicorpus.py", line 539, in <module>
    class WikiCorpus(TextCorpus):
  File "/usr/local/lib/python3.6/site-packages/gensim/corpora/wikicorpus.py", line 577, in WikiCorpus
    def __init__(self, fname, processes=None, lemmatize=utils.has_pattern(), dictionary=None,
  File "/usr/local/lib/python3.6/site-packages/gensim/utils.py", line 1614, in has_pattern
    from pattern.en import parse  # noqa:F401
  File "/usr/local/lib/python3.6/site-packages/pattern/text/en/__init__.py", line 61, in <module>
    from pattern.text.en.inflect import (
  File "/usr/local/lib/python3.6/site-packages/pattern/text/en/__init__.py", line 80, in <module>
    from pattern.text.en import wordnet
  File "/usr/local/lib/python3.6/site-packages/pattern/text/en/wordnet/__init__.py", line 57, in <module>
    nltk.data.find("corpora/" + token)
  File "/usr/local/lib/python3.6/site-packages/nltk/data.py", line 673, in find
    return find(modified_name, paths)
  File "/usr/local/lib/python3.6/site-packages/nltk/data.py", line 660, in find
    return ZipFilePathPointer(p, zipentry)
  File "/usr/local/lib/python3.6/site-packages/nltk/compat.py", line 228, in _decorator
    return init_func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/nltk/data.py", line 506, in __init__
    zipfile = OpenOnDemandZipFile(os.path.abspath(zipfile))
  File "/usr/local/lib/python3.6/site-packages/nltk/compat.py", line 228, in _decorator
    return init_func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/nltk/data.py", line 1055, in __init__
    zipfile.ZipFile.__init__(self, filename)
  File "/usr/local/lib/python3.6/zipfile.py", line 1131, in __init__
    self._RealGetContents()
  File "/usr/local/lib/python3.6/zipfile.py", line 1198, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
```

#### Versions

```python
 python
Python 3.6.8 (default, Jun 11 2019, 01:16:11)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform; print(platform.platform())
Linux-5.0.0-050000rc8-generic-x86_64-with-debian-9.11
>>> import sys; print("Python", sys.version)
Python 3.6.8 (default, Jun 11 2019, 01:16:11)
[GCC 6.3.0 20170516]
>>> import numpy; print("NumPy", numpy.__version__)
NumPy 1.17.4
>>> import scipy; print("SciPy", scipy.__version__)
SciPy 1.3.3
>>> import gensim; print("gensim", gensim.__version__)
gensim 3.8.1
>>> from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
FAST_VERSION 1
```

Hints from issue discussion:
Looks like an issue `pattern` (a 3rd party optional dependency) again. Thanks for reporting.

@mpenkov how about we either A) drop `pattern` altogether, or B) import it only in functions where it's really needed? (as opposed to "import at module-level scope").
Looking at the traceback more closely, B) doesn't seem possible, because of that top-level `lemmatize=utils.has_pattern()` call. So I'm in favour of A) then, or setting a `lemmatize=False` default.

And I guess there's also C) "wait for `pattern` maintainers to fix their stuff", but that sounds like a continued headache for us…
@piskvorky so is this actually an `nltk` problem not `gensim`? should i report to NLTK authors?
According to your traceback, it is some problem inside the `pattern` library trying to use NLTK's wordnet:

```
File "/usr/local/lib/python3.6/site-packages/pattern/text/en/__init__.py", line 80, in <module>
    from pattern.text.en import wordnet
File "/usr/local/lib/python3.6/site-packages/pattern/text/en/wordnet/__init__.py", line 57, in <module>
    nltk.data.find("corpora/" + token)
```

I have no idea what that is, it's something internal to `pattern`.
ok, looks like it's actually already fixed in clips/pattern@master, so i'm good for now, thanks. as far as i got it - it's somehow related to Python 3.7
Your traceback shows Python 3.6, not 3.7.

Anyway thanks for confirming. I'll leave this ticket open because we probably still want to avoid using `pattern`. Or at least defer importing it until absolutely the last moment possible (definitely not at Gensim import time, when people might not need it at all).
Hey, have you resolved this error? I have the same error and have no idea how to fix it...
@GaoFansakura i just used `clips/pattern@master` module (not `clips/pattern` from pip)

Commit message: rm removed file from docs

Patch to review:
diff --git a/docs/src/apiref.rst b/docs/src/apiref.rst
index 8f5e8fc61e..d6cdeeaf52 100644
--- a/docs/src/apiref.rst
+++ b/docs/src/apiref.rst
@@ -89,7 +89,6 @@ Modules:
     scripts/make_wikicorpus
     scripts/word2vec_standalone
     scripts/make_wiki_online
-    scripts/make_wiki_online_lemma
     scripts/make_wiki_online_nodebug
     scripts/word2vec2tensor
     scripts/segment_wiki
diff --git a/docs/src/scripts/make_wiki_online.rst b/docs/src/scripts/make_wiki_online.rst
deleted file mode 100644
index fc4e99c839..0000000000
--- a/docs/src/scripts/make_wiki_online.rst
+++ /dev/null
@@ -1,9 +0,0 @@
-:mod:`scripts.make_wiki_online` -- Convert articles from a Wikipedia dump
-=========================================================================
-
-.. automodule:: gensim.scripts.make_wiki_online
-    :synopsis: Convert articles from a Wikipedia dump
-    :members:
-    :inherited-members:
-    :undoc-members:
-    :show-inheritance:
diff --git a/gensim/corpora/_mmreader.pyx b/gensim/corpora/_mmreader.pyx
index 60cc4378e7..37ea774512 100644
--- a/gensim/corpora/_mmreader.pyx
+++ b/gensim/corpora/_mmreader.pyx
@@ -7,8 +7,6 @@ from __future__ import with_statement
 
 from gensim import utils
 
-from six import string_types
-from six.moves import range
 import logging
 
 cimport cython
@@ -187,7 +185,7 @@ cdef class MmReader():
 
         if offset == -1:
             return []
-        if isinstance(self.input, string_types):
+        if isinstance(self.input, str):
             fin, close_fin = utils.open(self.input, 'rb'), True
         else:
             fin, close_fin = self.input, False
diff --git a/gensim/corpora/wikicorpus.py b/gensim/corpora/wikicorpus.py
index 8c3c94b5ff..02db4e075b 100644
--- a/gensim/corpora/wikicorpus.py
+++ b/gensim/corpora/wikicorpus.py
@@ -12,11 +12,8 @@
 
 Notes
 -----
-If you have the `pattern <https://github.com/clips/pattern>`_ package installed,
-this module will use a fancy lemmatization to get a lemma of each token (instead of plain alphabetic tokenizer).
 
 See :mod:`gensim.scripts.make_wiki` for a canned (example) command-line script based on this module.
-
 """
 
 import bz2
@@ -468,8 +465,7 @@ def process_article(args, tokenizer_func=tokenize, token_min_len=TOKEN_MIN_LEN,
     Parameters
     ----------
     args : (str, bool, str, int)
-        Article text, lemmatize flag (if True, :func:`~gensim.utils.lemmatize` will be used), article title,
-        page identificator.
+        Article text, article title, page identificator.
     tokenizer_func : function
         Function for tokenization (defaults is :func:`~gensim.corpora.wikicorpus.tokenize`).
         Needs to have interface:
@@ -487,12 +483,9 @@ def process_article(args, tokenizer_func=tokenize, token_min_len=TOKEN_MIN_LEN,
         List of tokens from article, title and page id.
 
     """
-    text, lemmatize, title, pageid = args
+    text, title, pageid = args
     text = filter_wiki(text)
-    if lemmatize:
-        result = utils.lemmatize(text)
-    else:
-        result = tokenizer_func(text, token_min_len, token_max_len, lower)
+    result = tokenizer_func(text, token_min_len, token_max_len, lower)
     return result, title, pageid
 
 
@@ -574,7 +567,7 @@ class WikiCorpus(TextCorpus):
         >>> MmCorpus.serialize(corpus_path, wiki)  # another 8h, creates a file in MatrixMarket format and mapping
 
     """
-    def __init__(self, fname, processes=None, lemmatize=utils.has_pattern(), dictionary=None,
+    def __init__(self, fname, processes=None, dictionary=None,
                  filter_namespaces=('0',), tokenizer_func=tokenize, article_min_tokens=ARTICLE_MIN_WORDS,
                  token_min_len=TOKEN_MIN_LEN, token_max_len=TOKEN_MAX_LEN, lower=True, filter_articles=None):
         """Initialize the corpus.
@@ -588,9 +581,6 @@ def __init__(self, fname, processes=None, lemmatize=utils.has_pattern(), diction
             Path to the Wikipedia dump file.
         processes : int, optional
             Number of processes to run, defaults to `max(1, number of cpu - 1)`.
-        lemmatize : bool
-            Use lemmatization instead of simple regexp tokenization.
-            Defaults to `True` if you have the `pattern <https://github.com/clips/pattern>`_ package installed.
         dictionary : :class:`~gensim.corpora.dictionary.Dictionary`, optional
             Dictionary, if not provided,  this scans the corpus once, to determine its vocabulary
             **IMPORTANT: this needs a really long time**.
@@ -625,7 +615,6 @@ def __init__(self, fname, processes=None, lemmatize=utils.has_pattern(), diction
         if processes is None:
             processes = max(1, multiprocessing.cpu_count() - 1)
         self.processes = processes
-        self.lemmatize = lemmatize
         self.tokenizer_func = tokenizer_func
         self.article_min_tokens = article_min_tokens
         self.token_min_len = token_min_len
@@ -677,7 +666,7 @@ def get_texts(self):
 
         tokenization_params = (self.tokenizer_func, self.token_min_len, self.token_max_len, self.lower)
         texts = (
-            (text, self.lemmatize, title, pageid, tokenization_params)
+            (text, title, pageid, tokenization_params)
             for title, text, pageid
             in extract_pages(bz2.BZ2File(self.fname), self.filter_namespaces, self.filter_articles)
         )
diff --git a/gensim/scripts/make_wiki_lemma.py b/gensim/scripts/make_wiki_lemma.py
deleted file mode 120000
index 85ddf6cc4f..0000000000
--- a/gensim/scripts/make_wiki_lemma.py
+++ /dev/null
@@ -1 +0,0 @@
-make_wikicorpus.py
\ No newline at end of file
diff --git a/gensim/scripts/make_wiki_online_lemma.py b/gensim/scripts/make_wiki_online_lemma.py
deleted file mode 100755
index 0ec9704724..0000000000
--- a/gensim/scripts/make_wiki_online_lemma.py
+++ /dev/null
@@ -1,112 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-#
-# Copyright (C) 2010 Radim Rehurek <radimrehurek@seznam.cz>
-# Copyright (C) 2012 Lars Buitinck <larsmans@gmail.com>
-# Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html
-
-
-"""
-USAGE: %(program)s WIKI_XML_DUMP OUTPUT_PREFIX [VOCABULARY_SIZE]
-
-Convert articles from a Wikipedia dump to (sparse) vectors. The input is a
-bz2-compressed dump of Wikipedia articles, in XML format.
-
-This actually creates three files:
-
-* `OUTPUT_PREFIX_wordids.txt`: mapping between words and their integer ids
-* `OUTPUT_PREFIX_bow.mm`: bag-of-words (word counts) representation, in
-  Matrix Matrix format
-* `OUTPUT_PREFIX_tfidf.mm`: TF-IDF representation
-* `OUTPUT_PREFIX.tfidf_model`: TF-IDF model dump
-
-The output Matrix Market files can then be compressed (e.g., by bzip2) to save
-disk space; gensim's corpus iterators can work with compressed input, too.
-
-`VOCABULARY_SIZE` controls how many of the most frequent words to keep (after
-removing tokens that appear in more than 10%% of all documents). Defaults to
-100,000.
-
-If you have the `pattern` package installed, this script will use a fancy
-lemmatization to get a lemma of each token (instead of plain alphabetic
-tokenizer). The package is available at https://github.com/clips/pattern .
-
-Example:
-  python -m gensim.scripts.make_wikicorpus ~/gensim/results/enwiki-latest-pages-articles.xml.bz2 ~/gensim/results/wiki
-"""
-
-
-import logging
-import os.path
-import sys
-
-from gensim.corpora import Dictionary, HashDictionary, MmCorpus, WikiCorpus
-from gensim.models import TfidfModel
-
-
-# Wiki is first scanned for all distinct word types (~7M). The types that
-# appear in more than 10% of articles are removed and from the rest, the
-# DEFAULT_DICT_SIZE most frequent types are kept.
-DEFAULT_DICT_SIZE = 100000
-
-
-if __name__ == '__main__':
-    program = os.path.basename(sys.argv[0])
-    logger = logging.getLogger(program)
-
-    logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s')
-    logging.root.setLevel(level=logging.INFO)
-    logger.info("running %s", ' '.join(sys.argv))
-
-    # check and process input arguments
-    if len(sys.argv) < 3:
-        print(globals()['__doc__'] % locals())
-        sys.exit(1)
-    inp, outp = sys.argv[1:3]
-
-    if not os.path.isdir(os.path.dirname(outp)):
-        raise SystemExit("Error: The output directory does not exist. Create the directory and try again.")
-
-    if len(sys.argv) > 3:
-        keep_words = int(sys.argv[3])
-    else:
-        keep_words = DEFAULT_DICT_SIZE
-    online = 'online' in program
-    lemmatize = 'lemma' in program
-    debug = 'nodebug' not in program
-
-    if online:
-        dictionary = HashDictionary(id_range=keep_words, debug=debug)
-        dictionary.allow_update = True  # start collecting document frequencies
-        wiki = WikiCorpus(inp, lemmatize=lemmatize, dictionary=dictionary)
-        # ~4h on my macbook pro without lemmatization, 3.1m articles (august 2012)
-        MmCorpus.serialize(outp + '_bow.mm', wiki, progress_cnt=10000)
-        # with HashDictionary, the token->id mapping is only fully instantiated now, after `serialize`
-        dictionary.filter_extremes(no_below=20, no_above=0.1, keep_n=DEFAULT_DICT_SIZE)
-        dictionary.save_as_text(outp + '_wordids.txt.bz2')
-        wiki.save(outp + '_corpus.pkl.bz2')
-        dictionary.allow_update = False
-    else:
-        wiki = WikiCorpus(inp, lemmatize=lemmatize)  # takes about 9h on a macbook pro, for 3.5m articles (june 2011)
-        # only keep the most frequent words (out of total ~8.2m unique tokens)
-        wiki.dictionary.filter_extremes(no_below=20, no_above=0.1, keep_n=DEFAULT_DICT_SIZE)
-        # save dictionary and bag-of-words (term-document frequency matrix)
-        MmCorpus.serialize(outp + '_bow.mm', wiki, progress_cnt=10000)  # another ~9h
-        wiki.dictionary.save_as_text(outp + '_wordids.txt.bz2')
-        # load back the id->word mapping directly from file
-        # this seems to save more memory, compared to keeping the wiki.dictionary object from above
-        dictionary = Dictionary.load_from_text(outp + '_wordids.txt.bz2')
-    del wiki
-
-    # initialize corpus reader and word->id mapping
-    mm = MmCorpus(outp + '_bow.mm')
-
-    # build tfidf, ~50min
-    tfidf = TfidfModel(mm, id2word=dictionary, normalize=True)
-    tfidf.save(outp + '.tfidf_model')
-
-    # save tfidf vectors in matrix market format
-    # ~4h; result file is 15GB! bzip2'ed down to 4.5GB
-    MmCorpus.serialize(outp + '_tfidf.mm', tfidf[mm], progress_cnt=10000)
-
-    logger.info("finished running %s", program)
diff --git a/gensim/scripts/make_wikicorpus.py b/gensim/scripts/make_wikicorpus.py
index 49c249a707..66056cf10b 100755
--- a/gensim/scripts/make_wikicorpus.py
+++ b/gensim/scripts/make_wikicorpus.py
@@ -29,10 +29,6 @@
 removing tokens that appear in more than 10%% of all documents). Defaults to
 100,000.
 
-If you have the `pattern` package installed, this script will use a fancy
-lemmatization to get a lemma of each token (instead of plain alphabetic
-tokenizer). The package is available at https://github.com/clips/pattern .
-
 Example:
   python -m gensim.scripts.make_wikicorpus ~/gensim/results/enwiki-latest-pages-articles.xml.bz2 ~/gensim/results/wiki
 """
@@ -74,13 +70,12 @@
     else:
         keep_words = DEFAULT_DICT_SIZE
     online = 'online' in program
-    lemmatize = 'lemma' in program
     debug = 'nodebug' not in program
 
     if online:
         dictionary = HashDictionary(id_range=keep_words, debug=debug)
         dictionary.allow_update = True  # start collecting document frequencies
-        wiki = WikiCorpus(inp, lemmatize=lemmatize, dictionary=dictionary)
+        wiki = WikiCorpus(inp, dictionary=dictionary)
         # ~4h on my macbook pro without lemmatization, 3.1m articles (august 2012)
         MmCorpus.serialize(outp + '_bow.mm', wiki, progress_cnt=10000, metadata=True)
         # with HashDictionary, the token->id mapping is only fully instantiated now, after `serialize`
@@ -89,7 +84,7 @@
         wiki.save(outp + '_corpus.pkl.bz2')
         dictionary.allow_update = False
     else:
-        wiki = WikiCorpus(inp, lemmatize=lemmatize)  # takes about 9h on a macbook pro, for 3.5m articles (june 2011)
+        wiki = WikiCorpus(inp)  # takes about 9h on a macbook pro, for 3.5m articles (june 2011)
         # only keep the most frequent words (out of total ~8.2m unique tokens)
         wiki.dictionary.filter_extremes(no_below=20, no_above=0.1, keep_n=DEFAULT_DICT_SIZE)
         # save dictionary and bag-of-words (term-document frequency matrix)
diff --git a/gensim/scripts/segment_wiki.py b/gensim/scripts/segment_wiki.py
index 925c8877a0..8cfbc44760 100644
--- a/gensim/scripts/segment_wiki.py
+++ b/gensim/scripts/segment_wiki.py
@@ -268,7 +268,7 @@ class _WikiSectionsCorpus(WikiCorpus):
     """
 
     def __init__(self, fileobj, min_article_character=200, processes=None,
-                 lemmatize=utils.has_pattern(), filter_namespaces=('0',), include_interlinks=False):
+                 filter_namespaces=('0',), include_interlinks=False):
         """
         Parameters
         ----------
@@ -278,9 +278,6 @@ def __init__(self, fileobj, min_article_character=200, processes=None,
             Minimal number of character for article (except titles and leading gaps).
         processes : int, optional
             Number of processes, max(1, multiprocessing.cpu_count() - 1) if None.
-        lemmatize : bool, optional
-            If `pattern` package is installed, use fancier shallow parsing to get token lemmas.
-            Otherwise, use simple regexp tokenization.
         filter_namespaces : tuple of int, optional
             Enumeration of namespaces that will be ignored.
         include_interlinks: bool
@@ -293,7 +290,6 @@ def __init__(self, fileobj, min_article_character=200, processes=None,
         if processes is None:
             processes = max(1, multiprocessing.cpu_count() - 1)
         self.processes = processes
-        self.lemmatize = lemmatize
         self.min_article_character = min_article_character
         self.include_interlinks = include_interlinks
 
diff --git a/gensim/test/test_corpora.py b/gensim/test/test_corpora.py
index 611cc875eb..6c09ea2d1f 100644
--- a/gensim/test/test_corpora.py
+++ b/gensim/test/test_corpora.py
@@ -698,7 +698,7 @@ def test_custom_tokenizer(self):
         """
         define a custom tokenizer function and use it
         """
-        wc = self.corpus_class(self.enwiki, processes=1, lemmatize=False, tokenizer_func=custom_tokenizer,
+        wc = self.corpus_class(self.enwiki, processes=1, tokenizer_func=custom_tokenizer,
                         token_max_len=16, token_min_len=1, lower=False)
         row = wc.get_texts()
         list_tokens = next(row)
@@ -711,7 +711,7 @@ def test_lower_case_set_true(self):
         """
         Set the parameter lower to True and check that upper case 'Anarchism' token doesnt exist
         """
-        corpus = self.corpus_class(self.enwiki, processes=1, lower=True, lemmatize=False)
+        corpus = self.corpus_class(self.enwiki, processes=1, lower=True)
         row = corpus.get_texts()
         list_tokens = next(row)
         self.assertTrue(u'Anarchism' not in list_tokens)
@@ -721,7 +721,7 @@ def test_lower_case_set_false(self):
         """
         Set the parameter lower to False and check that upper case Anarchism' token exists
         """
-        corpus = self.corpus_class(self.enwiki, processes=1, lower=False, lemmatize=False)
+        corpus = self.corpus_class(self.enwiki, processes=1, lower=False)
         row = corpus.get_texts()
         list_tokens = next(row)
         self.assertTrue(u'Anarchism' in list_tokens)
@@ -732,14 +732,14 @@ def test_min_token_len_not_set(self):
         Don't set the parameter token_min_len and check that 'a' as a token doesn't exist
         Default token_min_len=2
         """
-        corpus = self.corpus_class(self.enwiki, processes=1, lemmatize=False)
+        corpus = self.corpus_class(self.enwiki, processes=1)
         self.assertTrue(u'a' not in next(corpus.get_texts()))
 
     def test_min_token_len_set(self):
         """
         Set the parameter token_min_len to 1 and check that 'a' as a token exists
         """
-        corpus = self.corpus_class(self.enwiki, processes=1, token_min_len=1, lemmatize=False)
+        corpus = self.corpus_class(self.enwiki, processes=1, token_min_len=1)
         self.assertTrue(u'a' in next(corpus.get_texts()))
 
     def test_max_token_len_not_set(self):
@@ -747,14 +747,14 @@ def test_max_token_len_not_set(self):
         Don't set the parameter token_max_len and check that 'collectivisation' as a token doesn't exist
         Default token_max_len=15
         """
-        corpus = self.corpus_class(self.enwiki, processes=1, lemmatize=False)
+        corpus = self.corpus_class(self.enwiki, processes=1)
         self.assertTrue(u'collectivization' not in next(corpus.get_texts()))
 
     def test_max_token_len_set(self):
         """
         Set the parameter token_max_len to 16 and check that 'collectivisation' as a token exists
         """
-        corpus = self.corpus_class(self.enwiki, processes=1, token_max_len=16, lemmatize=False)
+        corpus = self.corpus_class(self.enwiki, processes=1, token_max_len=16)
         self.assertTrue(u'collectivization' in next(corpus.get_texts()))
 
     def test_removed_table_markup(self):
diff --git a/gensim/utils.py b/gensim/utils.py
index ba6171f109..fd524d2194 100644
--- a/gensim/utils.py
+++ b/gensim/utils.py
@@ -1597,105 +1597,6 @@ def pyro_daemon(name, obj, random_suffix=False, ip=None, port=None, ns_conf=None
             daemon.requestLoop()
 
 
-def has_pattern():
-    """Check whether the `pattern <https://github.com/clips/pattern>`_ package is installed.
-
-    Returns
-    -------
-    bool
-        Is `pattern` installed?
-
-    """
-    try:
-        from pattern.en import parse  # noqa:F401
-        return True
-    except ImportError:
-        return False
-
-
-def lemmatize(content, allowed_tags=re.compile(r'(NN|VB|JJ|RB)'), light=False,
-              stopwords=frozenset(), min_length=2, max_length=15):
-    """Use the English lemmatizer from `pattern <https://github.com/clips/pattern>`_ to extract UTF8-encoded tokens in
-    their base form aka lemma, e.g. "are, is, being" becomes "be" etc.
-
-    This is a smarter version of stemming, taking word context into account.
-
-    Parameters
-    ----------
-    content : str
-        Input string
-    allowed_tags : :class:`_sre.SRE_Pattern`, optional
-        Compiled regexp to select POS that will be used.
-        Only considers nouns, verbs, adjectives and adverbs by default (=all other lemmas are discarded).
-    light : bool, optional
-        DEPRECATED FLAG, DOESN'T SUPPORT BY `pattern`.
-    stopwords : frozenset, optional
-        Set of words that will be removed from output.
-    min_length : int, optional
-        Minimal token length in output (inclusive).
-    max_length : int, optional
-        Maximal token length in output (inclusive).
-
-    Returns
-    -------
-    list of str
-        List with tokens with POS tags.
-
-    Warnings
-    --------
-    This function is only available when the optional `pattern <https://github.com/clips/pattern>`_ is installed.
-
-    Raises
-    ------
-    ImportError
-        If `pattern <https://github.com/clips/pattern>`_ not installed.
-
-    Examples
-    --------
-    .. sourcecode:: pycon
-
-        >>> from gensim.utils import lemmatize
-        >>> lemmatize('Hello World! How is it going?! Nonexistentword, 21')
-        ['world/NN', 'be/VB', 'go/VB', 'nonexistentword/NN']
-
-    Note the context-dependent part-of-speech tags between these two examples:
-
-    .. sourcecode:: pycon
-
-        >>> lemmatize('The study ranks high.')
-        ['study/NN', 'rank/VB', 'high/JJ']
-
-        >>> lemmatize('The ranks study hard.')
-        ['rank/NN', 'study/VB', 'hard/RB']
-
-    """
-    if not has_pattern():
-        raise ImportError(
-            "Pattern library is not installed. Pattern library is needed in order to use lemmatize function"
-        )
-    from pattern.en import parse
-
-    if light:
-        import warnings
-        warnings.warn("The light flag is no longer supported by pattern.")
-
-    # Tokenization in `pattern` is weird; it gets thrown off by non-letters,
-    # producing '==relate/VBN' or '**/NN'... try to preprocess the text a little.
-    # XXX: this throws away all fancy parsing cues, including sentence structure,
-    # abbreviations etc.
-    content = ' '.join(tokenize(content, lower=True, errors='ignore'))
-
-    parsed = parse(content, lemmata=True, collapse=False)
-    result = []
-    for sentence in parsed:
-        for token, tag, _, _, lemma in sentence:
-            if min_length <= len(lemma) <= max_length and not lemma.startswith('_') and lemma not in stopwords:
-                if allowed_tags.match(tag):
-                    lemma += "/" + tag[:2]
-                    result.append(lemma.encode('utf8'))
-    return result
-
-
 def mock_data_row(dim=1000, prob_nnz=0.5, lam=1.0):
     """Create a random gensim BoW vector, with the feature counts following the Poisson distribution.
 
diff --git a/setup.py b/setup.py
index b3a2b61cf4..22835407b4 100644
--- a/setup.py
+++ b/setup.py
@@ -304,12 +304,6 @@ def run(self):
     'sphinxcontrib-napoleon',
     'matplotlib',  # expected by sphinx-gallery
     'plotly',
-    #
-    # Pattern is a PITA to install, it requires mysqlclient, which in turn
-    # requires MySQL dev tools be installed. We don't need it for building
-    # documentation.
-    #
-    # 'Pattern==3.6',  # Need 3.6 or later for Py3 support
     'memory_profiler',
     'annoy',
     'Pyro4',
@@ -327,13 +321,14 @@ def run(self):
 # to build with any sane version of Cython, so we should update this pin
 # periodically.
 #
-CYTHON_STR = 'Cython==0.29.14'
+CYTHON_STR = 'Cython==0.29.21'
 
 install_requires = [
     NUMPY_STR,
     'scipy >= 0.18.1',
     'smart_open >= 1.8.1',
     "dataclasses; python_version < '3.7'",  # pre-py3.7 needs `dataclasses` backport for use of `dataclass` in doc2vec.py
+    'six',
 ]
 
 setup_requires = [NUMPY_STR]


Code review comments:
diff_hunk: @@ -327,13 +321,14 @@ def run(self):
 # to build with any sane version of Cython, so we should update this pin
 # periodically.
 #
-CYTHON_STR = 'Cython==0.29.14'
+CYTHON_STR = 'Cython==0.29.21'
 
 install_requires = [
     NUMPY_STR,
     'scipy >= 0.18.1',
     'smart_open >= 1.8.1',
     "dataclasses; python_version < '3.7'",  # pre-py3.7 needs `dataclasses` backport for use of `dataclass` in doc2vec.py
+    'six',
path: setup.py
text: @user1:
What for? Looks like a regression, we already got rid of six in previous PRs IIRC.

@author:
gensim wouldn't build without it, so I added it as a temporary workaround.

Upon further digging, we haven't removed it completely:

```
$ grep -R "from six" gensim
gensim/models/word2vec_corpusfile.pyx:from six import iteritems
gensim/models/word2vec_corpusfile.cpp: * from six import iteritems
gensim/models/word2vec_corpusfile.cpp: * from six import iteritems
```

plus the one occurrence I removed in this PR.

@user1:
Aha, I guess I only searched in `*.py` when I did my purging.

I'll add a commit here for the `.pyx` and `.cpp` too, so that we're really six-free.

@author:
I handled the pyx already.

The cpp is autogenerated, so we shouldn't touch it.