Repository: piskvorky/gensim

Issue:
Recent versions of smart-open (1.11.0 and 1.11.1)  break gensim in Python 2.7
### Problem description

What are you trying to achieve? What is the expected result? What are you seeing instead?

Using gensim with Python2.7

#### Steps/code/corpus to reproduce
```
pip install gensim
...
python
>>> import gensim
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/brukau/tmp/gensim/local/lib/python2.7/site-packages/gensim/__init__.py", line 5, in <module>
    from gensim import parsing, corpora, matutils, interfaces, models, similarities, summarization, utils  # noqa:F401
  File "/home/brukau/tmp/gensim/local/lib/python2.7/site-packages/gensim/parsing/__init__.py", line 4, in <module>
    from .preprocessing import (remove_stopwords, strip_punctuation, strip_punctuation2,  # noqa:F401
  File "/home/brukau/tmp/gensim/local/lib/python2.7/site-packages/gensim/parsing/preprocessing.py", line 42, in <module>
    from gensim import utils
  File "/home/brukau/tmp/gensim/local/lib/python2.7/site-packages/gensim/utils.py", line 45, in <module>
    from smart_open import open
  File "/home/brukau/tmp/gensim/local/lib/python2.7/site-packages/smart_open/__init__.py", line 28, in <module>
    from .smart_open_lib import open, parse_uri, smart_open, register_compressor
  File "/home/brukau/tmp/gensim/local/lib/python2.7/site-packages/smart_open/smart_open_lib.py", line 23, in <module>
    import pathlib
ImportError: No module named pathlib
>>> 
```

#### Versions

Please provide the output of:

```python
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import gensim; print("gensim", gensim.__version__)
from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
```

Linux-5.3.0-45-generic-x86_64-with-Ubuntu-19.10-eoan
('Python', '2.7.17 (default, Nov  7 2019, 10:07:09) \n[GCC 9.2.1 20191008]')
('NumPy', '1.16.1')
('SciPy', '1.2.3')
gensim import fails




Discussion:
@mpenkov we'll have to pin an older version of smart_open for Gensim. Gensim supports 2.7, and 2.7 is still widely used in the industry, but the latest smart_open releases dropped 2.7 support.

@brukau @bavaria95 why do you use Python2.7?
@mpenkov I also wonder how come CI didn't catch this? A critical blocking issue (for 2.7 users).
@piskvorky we are moving to Py3 only, but it seems not fast enough 

Resolving patch:
diff --git a/docs/src/conf.py b/docs/src/conf.py
--- a/docs/src/conf.py
+++ b/docs/src/conf.py
@@ -57,7 +57,7 @@
 # The short X.Y version.
 version = '3.8'
 # The full version, including alpha/beta/rc tags.
-release = '3.8.1'
+release = '3.8.2'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -276,8 +276,8 @@ def finalize_options(self):
     ])
 
 if (3, 0) < sys.version_info < (3, 7):
-    linux_testenv.extend(['nmslib'])    
-    
+    linux_testenv.extend(['nmslib'])
+
 docs_testenv = linux_testenv + distributed_env + [
     'sphinx',
     'sphinxcontrib-napoleon',
@@ -345,9 +345,24 @@ def finalize_options(self):
                   extra_link_args=extra_args)
     )
 
+install_requires = [
+    NUMPY_STR,
+    'scipy >= 1.0.0',
+    'six >= 1.5.0',
+]
+
+#
+# smart_open >= 1.11 is py3+ only.
+# TODO: Remove the pin once we drop py2.7 from gensim too.
+#
+if PY2:
+    install_requires.append('smart_open >= 1.8.1, < 1.11')
+else:
+    install_requires.append('smart_open >= 1.8.1')
+
 setup(
     name='gensim',
-    version='3.8.1',
+    version='3.8.2',
     description='Python framework for fast Vector Space Modelling',
     long_description=LONG_DESCRIPTION,
 
@@ -360,7 +375,7 @@ def finalize_options(self):
 
     url='http://radimrehurek.com/gensim',
     download_url='http://pypi.python.org/pypi/gensim',
-    
+
     license='LGPLv2.1',
 
     keywords='Singular Value Decomposition, SVD, Latent Semantic Indexing, '
@@ -391,12 +406,7 @@ def finalize_options(self):
     setup_requires=[
         NUMPY_STR,
     ],
-    install_requires=[
-        NUMPY_STR,
-        'scipy >= 0.18.1',
-        'six >= 1.5.0',
-        'smart_open >= 1.8.1',
-    ],
+    install_requires=install_requires,
     tests_require=linux_testenv,
     extras_require={
         'distributed': distributed_env,


Test patch: