Repository: piskvorky/gensim

Issue:
Do test instructions still work?
In our [README](https://github.com/RaRe-Technologies/gensim#installation), we tell users to run `python setup.py test`.

One user [reported on our mailing list](https://groups.google.com/forum/#!topic/gensim/JfeV2phNTjk) that this doesn't actually work.

We should fix the docs or the tests – whichever is at fault here.
Use python_requires in setup.py
https://packaging.python.org/guides/distributing-packages-using-setuptools/#python-requires, please :)

_Originally posted by @menshikh-iv in https://github.com/RaRe-Technologies/gensim/pull/2812_


Discussion:
@mpenkov apparently Gensim in `develop` requires py3.5+ already:
https://github.com/RaRe-Technologies/gensim/blob/develop/setup.py#L23

That means we cannot work off `develop` for 3.8.3, because that must still support py2.7. Which branch should I use?
Worse: does 3.8.2 have the same problem too? I don't see any py2.7 wheels on PyPI:
https://github.com/RaRe-Technologies/gensim/blob/develop/setup.py#L23
I tracked the problem down to commit https://github.com/RaRe-Technologies/gensim/commit/e859c11f6f57bf3c883a718a9ab7067ac0c2d4cf (part of PR https://github.com/RaRe-Technologies/gensim/pull/2630, merged 6 months ago). It claims to remove native Python implementations of Cython extensions, but also silently drops support for Python2 (!).

@mpenkov I'm not sure how to proceed, so I'll pause work on 3.8.3. Silently dropping Python2 support in a minor bugfix release is terrible, we cannot do that.

3.8.2 seems thoroughly broken, let's retract that release / supercede it ASAP.
Created #2811 to deal with the Py2 issue.
> @mpenkov I'm not sure how to proceed, so I'll pause work on 3.8.3.

@piskvorky We've successfully patched our develop HEAD to work on Py2 (https://github.com/RaRe-Technologies/gensim/pull/2812), so please proceed as previously discussed.
Great, will do. Thanks @mpenkov @menshikh-iv 
> We've successfully patched our develop HEAD

@mpenkov I don't see that in `develop`. Should I work off the `release-3.8.3` branch?
Because the changes for Py2.7 are essentially only 3.8.3, we won't be merging them into develop (there is no point, because we'd be removing them as soon as we ditch Py2.7).

Please work off `develop`. We will merge that into 3.8.3 before the actual release (as opposed to the other way around).
btw about original topic: `python setup.py test` no relevant for last years, should be `tox -e py<version>-<os>`, like `tox -e py37-linux`


Resolving patch:
diff --git a/gensim/similarities/nmslib.py b/gensim/similarities/nmslib.py
--- a/gensim/similarities/nmslib.py
+++ b/gensim/similarities/nmslib.py
@@ -8,10 +8,10 @@
 Intro
 -----
 
-This module contains integration Nmslib with :class:`~gensim.models.word2vec.Word2Vec`,
+This module contains integration NMSLIB with :class:`~gensim.models.word2vec.Word2Vec`,
 :class:`~gensim.models.doc2vec.Doc2Vec`, :class:`~gensim.models.fasttext.FastText` and
 :class:`~gensim.models.keyedvectors.KeyedVectors`.
-To use nmslib, instantiate a :class:`~gensim.similarities.nmslib.NmslibIndexer` class
+To use NMSLIB, instantiate a :class:`~gensim.similarities.nmslib.NmslibIndexer` class
 and pass the instance as the indexer parameter to your model's most_similar method
 (e.g. :py:func:`~gensim.models.doc2vec.most_similar`).
 
@@ -50,23 +50,23 @@
     >>> model.most_similar("cat", topn=2, indexer=new_indexer)
     [('cat', 1.0), ('meow', 0.5595494508743286)]
 
-What is Nmslib
--------------
+What is NMSLIB
+--------------
 
 Non-Metric Space Library (NMSLIB) is an efficient cross-platform similarity search library and a toolkit
 for evaluation of similarity search methods. The core-library does not have any third-party dependencies.
-More information about Nmslib: `github repository <https://github.com/nmslib/nmslib>`_.
+More information about NMSLIB: `github repository <https://github.com/nmslib/nmslib>`_.
 
-Why use Nmslib?
--------------
+Why use NMSIB?
+--------------
 
 The current implementation for finding k nearest neighbors in a vector space in gensim has linear complexity
 via brute force in the number of indexed documents, although with extremely low constant factors.
 The retrieved results are exact, which is an overkill in many applications:
 approximate results retrieved in sub-linear time may be enough.
-Nmslib can find approximate nearest neighbors much faster.
-Compared to annoy, nmslib has more parameters to control the build and query time and accuracy.
-Nmslib can achieve faster and more accurate nearest neighbors search than annoy.
+NMSLIB can find approximate nearest neighbors much faster.
+Compared to Annoy, NMSLIB has more parameters to control the build and query time and accuracy.
+NMSLIB can achieve faster and more accurate nearest neighbors search than annoy.
 """
 
 from smart_open import open
@@ -84,12 +84,12 @@
     import nmslib
 except ImportError:
     raise ImportError(
-        "Nmslib has not been installed, if you wish to use the nmslib indexer, please run `pip install nmslib`"
+        "NMSLIB not installed. To use the NMSLIB indexer, please run `pip install nmslib`."
     )
 
 
 class NmslibIndexer(object):
-    """This class allows to use `Nmslib <https://github.com/nmslib/nmslib>`_ as indexer for `most_similar` method
+    """This class allows to use `NMSLIB <https://github.com/nmslib/nmslib>`_ as indexer for `most_similar` method
     from :class:`~gensim.models.word2vec.Word2Vec`, :class:`~gensim.models.doc2vec.Doc2Vec`,
     :class:`~gensim.models.fasttext.FastText` and :class:`~gensim.models.keyedvectors.Word2VecKeyedVectors` classes.
 
@@ -102,9 +102,9 @@ def __init__(self, model, index_params=None, query_time_params=None):
         model : :class:`~gensim.models.base_any2vec.BaseWordEmbeddingsModel`
             Model, that will be used as source for index.
         index_params : dict, optional
-            index_params for Nmslib indexer.
+            index_params for NMSLIB indexer.
         query_time_params : dict, optional
-            query_time_params for Nmslib indexer.
+            query_time_params for NMSLIB indexer.
 
         """
         if index_params is None:
@@ -179,13 +179,13 @@ def load(cls, fname):
         return nmslib_instance
 
     def _build_from_word2vec(self):
-        """Build an Nmslib index using word vectors from a Word2Vec model."""
+        """Build an NMSLIB index using word vectors from a Word2Vec model."""
 
         self.model.init_sims()
         self._build_from_model(self.model.wv.vectors_norm, self.model.wv.index2word)
 
     def _build_from_doc2vec(self):
-        """Build an Nmslib index using document vectors from a Doc2Vec model."""
+        """Build an NMSLIB index using document vectors from a Doc2Vec model."""
 
         docvecs = self.model.docvecs
         docvecs.init_sims()
@@ -193,7 +193,7 @@ def _build_from_doc2vec(self):
         self._build_from_model(docvecs.vectors_docs_norm, labels)
 
     def _build_from_keyedvectors(self):
-        """Build an Nmslib index using word vectors from a KeyedVectors model."""
+        """Build an NMSLIB index using word vectors from a KeyedVectors model."""
 
         self.model.init_sims()
         self._build_from_model(self.model.vectors_norm, self.model.index2word)
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -20,8 +20,6 @@
 from setuptools import setup, find_packages, Extension
 from setuptools.command.build_ext import build_ext
 
-if sys.version_info[:2] < (3, 5):
-    raise Exception('This version of gensim needs 3.5 or later.')
 
 c_extensions = {
     'gensim.models.word2vec_inner': 'gensim/models/word2vec_inner.c',
@@ -169,11 +167,11 @@ def run(self):
 * All algorithms are **memory-independent** w.r.t. the corpus size (can process input larger than RAM, streamed, out-of-core),
 * **Intuitive interfaces**
 
-  * easy to plug in your own input corpus/datastream (trivial streaming API)
-  * easy to extend with other Vector Space algorithms (trivial transformation API)
+  * easy to plug in your own input corpus/datastream (simple streaming API)
+  * easy to extend with other Vector Space algorithms (simple transformation API)
 
 * Efficient multicore implementations of popular algorithms, such as online **Latent Semantic Analysis (LSA/LSI/SVD)**,
-  **Latent Dirichlet Allocation (LDA)**, **Random Projections (RP)**, **Hierarchical Dirichlet Process (HDP)**  or **word2vec deep learning**.
+  **Latent Dirichlet Allocation (LDA)**, **Random Projections (RP)**, **Hierarchical Dirichlet Process (HDP)** or **word2vec deep learning**.
 * **Distributed computing**: can run *Latent Semantic Analysis* and *Latent Dirichlet Allocation* on a cluster of computers.
 * Extensive `documentation and Jupyter Notebook tutorials <https://github.com/RaRe-Technologies/gensim/#documentation>`_.
 
@@ -190,21 +188,20 @@ def run(self):
 
 It is also recommended you install a fast BLAS library before installing NumPy. This is optional, but using an optimized BLAS such as `ATLAS <http://math-atlas.sourceforge.net/>`_ or `OpenBLAS <http://xianyi.github.io/OpenBLAS/>`_ is known to improve performance by as much as an order of magnitude. On OS X, NumPy picks up the BLAS that comes with it automatically, so you don't need to do anything special.
 
-The simple way to install `gensim` is::
+Install the latest version of gensim::
 
-    pip install -U gensim
+    pip install --upgrade gensim
 
-Or, if you have instead downloaded and unzipped the `source tar.gz <http://pypi.python.org/pypi/gensim>`_ package,
-you'd run::
+Or, if you have instead downloaded and unzipped the `source tar.gz <http://pypi.python.org/pypi/gensim>`_ package::
 
-    python setup.py test
     python setup.py install
 
 
-For alternative modes of installation (without root privileges, development
-installation, optional install features), see the `install documentation <http://radimrehurek.com/gensim/install.html>`_.
+For alternative modes of installation, see the `documentation <http://radimrehurek.com/gensim/install.html>`_.
+
+Gensim is being `continuously tested <https://travis-ci.org/RaRe-Technologies/gensim>`_ under Python 3.5, 3.6, 3.7 and 3.8.
+Support for Python 2.7 was dropped in gensim 4.0.0 – install gensim 3.8.3 if you must use Python 2.7.
 
-This version has been tested under Python 2.7, 3.5 and 3.6. Support for Python 2.6, 3.3 and 3.4 was dropped in gensim 1.0.0. Install gensim 0.13.4 if you *must* use Python 2.6, 3.3 or 3.4. Support for Python 2.5 was dropped in gensim 0.10.0; install gensim 0.9.1 if you *must* use Python 2.5). Gensim's github repo is hooked against `Travis CI for automated testing <https://travis-ci.org/RaRe-Technologies/gensim>`_ on every commit push and pull request.
 
 How come gensim is so fast and memory efficient? Isn't it pure Python, and isn't Python slow and greedy?
 --------------------------------------------------------------------------------------------------------
@@ -262,18 +259,20 @@ def run(self):
     'pytest-rerunfailures',
     'mock',
     'cython',
-    # temporarily remove pyemd to work around appveyor issues
-    # 'pyemd',
+    'nmslib',
+    'pyemd',
     'testfixtures',
     'Morfessor==2.0.2a4',
     'python-Levenshtein >= 0.10.2',
     'visdom >= 0.1.8, != 0.1.8.7',
     'scikit-learn',
+    # The following packages are commented out because they don't install on Windows. So skip the
+    # related tests in AppVeyor. We still test them in Linux via Travis, see linux_testenv below.
+    # See https://github.com/RaRe-Technologies/gensim/pull/2814
+    # 'tensorflow',
+    # 'keras',
 ]
 
-linux_testenv = win_testenv[:]
-
-#
 # This list partially duplicates requirements_docs.txt.
 # The main difference is that we don't include version pins here unless
 # absolutely necessary, whereas requirements_docs.txt includes pins for
@@ -283,18 +282,19 @@ def run(self):
 #
 #   https://packaging.python.org/discussions/install-requires-vs-requirements/
 #
-docs_testenv = linux_testenv + distributed_env + [
-    'sphinx <= 2.4.4',  # avoid `sphinx >= 3.0` that breaks build
+docs_testenv = win_testenv + distributed_env + [
+    'sphinx <= 2.4.4',  # avoid `sphinx >= 3.0` that breaks the build
+    'sphinx-gallery',
+    'sphinxcontrib.programoutput',
     'sphinxcontrib-napoleon',
+    'matplotlib',  # expected by sphinx-gallery
     'plotly',
     #
     # Pattern is a PITA to install, it requires mysqlclient, which in turn
-    # requires MySQL dev tools be installed.  We don't need it for building
+    # requires MySQL dev tools be installed. We don't need it for building
     # documentation.
     #
     # 'Pattern==3.6',  # Need 3.6 or later for Py3 support
-    'sphinxcontrib.programoutput',
-    'sphinx-gallery',
     'memory_profiler',
     'annoy',
     'Pyro4',
@@ -304,17 +304,18 @@ def run(self):
     'statsmodels',
     'pyemd',
     'pandas',
-    'matplotlib',  # sphinx-gallery expects this dep
 ]
 
-if sys.version_info < (3, 7):
-    linux_testenv.extend([
-        'tensorflow <= 1.3.0',
-        'keras >= 2.0.4, <= 2.1.4',
-    ])
-
-if (3, 0) < sys.version_info < (3, 7):
-    linux_testenv.extend(['nmslib'])
+# Add additional requirements for testing on Linux. We skip some tests on Windows,
+# because the libraries below are too tricky to install there.
+linux_testenv = win_testenv[:]
+if sys.version_info >= (3, 7):
+    # HACK: Installing tensorflow causes a segfault in Travis on py3.6. Other Pythons work – a mystery.
+    # See https://github.com/RaRe-Technologies/gensim/pull/2814#issuecomment-621477948
+    linux_testenv += [
+        'tensorflow',
+        'keras',
+    ]
 
 NUMPY_STR = 'numpy >= 1.11.3'
 #
@@ -374,12 +375,14 @@ def run(self):
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',
+        'Programming Language :: Python :: 3.8',
         'Topic :: Scientific/Engineering :: Artificial Intelligence',
         'Topic :: Scientific/Engineering :: Information Analysis',
         'Topic :: Text Processing :: Linguistic',
     ],
 
     test_suite="gensim.test",
+    python_requires='>=3.5',
     setup_requires=setup_requires,
     install_requires=install_requires,
     tests_require=linux_testenv,


Test patch:
diff --git a/gensim/test/test_keyedvectors.py b/gensim/test/test_keyedvectors.py
--- a/gensim/test/test_keyedvectors.py
+++ b/gensim/test/test_keyedvectors.py
@@ -11,15 +11,15 @@
 
 import logging
 import unittest
-from mock import patch
 
+from mock import patch
 import numpy as np
 
 from gensim.corpora import Dictionary
-from gensim.models.keyedvectors import KeyedVectors, WordEmbeddingSimilarityIndex, \
-    FastTextKeyedVectors, REAL
+from gensim.models.keyedvectors import (
+    KeyedVectors, WordEmbeddingSimilarityIndex, FastTextKeyedVectors, REAL,
+)
 from gensim.test.utils import datapath
-
 import gensim.models.keyedvectors
 
 logger = logging.getLogger(__name__)
diff --git a/gensim/test/test_similarities.py b/gensim/test/test_similarities.py
--- a/gensim/test/test_similarities.py
+++ b/gensim/test/test_similarities.py
@@ -544,8 +544,8 @@ class TestWord2VecAnnoyIndexer(unittest.TestCase):
     def setUp(self):
         try:
             import annoy  # noqa:F401
-        except ImportError:
-            raise unittest.SkipTest("Annoy library is not available")
+        except ImportError as e:
+            raise unittest.SkipTest("Annoy library is not available: %s" % e)
 
         from gensim.similarities.index import AnnoyIndexer
         self.indexer = AnnoyIndexer
@@ -648,8 +648,8 @@ class TestDoc2VecAnnoyIndexer(unittest.TestCase):
     def setUp(self):
         try:
             import annoy  # noqa:F401
-        except ImportError:
-            raise unittest.SkipTest("Annoy library is not available")
+        except ImportError as e:
+            raise unittest.SkipTest("Annoy library is not available: %s" % e)
 
         from gensim.similarities.index import AnnoyIndexer
 
@@ -707,8 +707,8 @@ class TestWord2VecNmslibIndexer(unittest.TestCase):
     def setUp(self):
         try:
             import nmslib  # noqa:F401
-        except ImportError:
-            raise unittest.SkipTest("Nmslib library is not available")
+        except ImportError as e:
+            raise unittest.SkipTest("NMSLIB library is not available: %s" % e)
 
         from gensim.similarities.nmslib import NmslibIndexer
         self.indexer = NmslibIndexer
@@ -800,8 +800,8 @@ class TestDoc2VecNmslibIndexer(unittest.TestCase):
     def setUp(self):
         try:
             import nmslib  # noqa:F401
-        except ImportError:
-            raise unittest.SkipTest("Nmslib library is not available")
+        except ImportError as e:
+            raise unittest.SkipTest("NMSLIB library is not available: %s" % e)
 
         from gensim.similarities.nmslib import NmslibIndexer