Repository: piskvorky/gensim

Issue:
Filename overwrite in `FastText.load_fasttext_format`
I download FB pretrained common-crawl model using https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.en.300.bin.gz from https://fasttext.cc/docs/en/crawl-vectors.html page.
When I try to load it, I get an error

**Code:**
```python
from gensim.models import FastText

ft_cc = FastText.load_fasttext_format("models/cc.en.300.bin.gz", full_model=False)
```

**Stacktrace:**
```
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<timed exec> in <module>

~/.local/lib/python3.6/site-packages/gensim/models/fasttext.py in load_fasttext_format(cls, model_file, encoding, full_model)
   1012 
   1013         """
-> 1014         return _load_fasttext_format(model_file, encoding=encoding, full_model=full_model)
   1015 
   1016     def load_binary_data(self, encoding='utf8'):

~/.local/lib/python3.6/site-packages/gensim/models/fasttext.py in _load_fasttext_format(model_file, encoding, full_model)
   1245     if not model_file.endswith('.bin'):
   1246         model_file += '.bin'
-> 1247     with smart_open(model_file, 'rb') as fin:
   1248         m = gensim.models._fasttext_bin.load(fin, encoding=encoding, full_model=full_model)
   1249 

~/.local/lib/python3.6/site-packages/smart_open/smart_open_lib.py in smart_open(uri, mode, **kw)
    179         raise TypeError('mode should be a string')
    180 
--> 181     fobj = _shortcut_open(uri, mode, **kw)
    182     if fobj is not None:
    183         return fobj

~/.local/lib/python3.6/site-packages/smart_open/smart_open_lib.py in _shortcut_open(uri, mode, **kw)
    299     #
    300     if six.PY3:
--> 301         return open(parsed_uri.uri_path, mode, buffering=buffering, **open_kwargs)
    302     elif not open_kwargs:
    303         return open(parsed_uri.uri_path, mode, buffering=buffering)

FileNotFoundError: [Errno 2] No such file or directory: 'models/cc.en.300.bin.gz.bin'
```

Problem in filename overwriting that happens
https://github.com/RaRe-Technologies/gensim/blob/cebc9dbeacddf7689c35aadb8854ae850d4561d2/gensim/models/fasttext.py#L1302-L1303

To avoid that, I should `gunzip` model file first (that's definitely not a thing that user expect).

**Versions:**
no matters, all python & gensim version affected, including last release.


Discussion:
Not necessarily a bug, but I do think we could support this. Especially since we're already using `smart_open` (I think), so .gz comes for free.

The other option is to inform users through documentation/tutorials that they need to unpack their models first.

@menshikh-iv can you open a PR?
> Not necessarily a bug,

from user PoV - definitely a bug (fasttext try to open a file that user don't specify, instead of passed file and failed on it).

> can you open a PR?

No time, sorry


@menshikh-iv what is the `if not model_file.endswith('.bin'): ` condition for? Looks weird to me, to be changing the user input like that.
@piskvorky ask an author of this code, not me (btw @mpenkov isn't an author too, they just refactored it, according to git blame).

I guess that this is for the case when user have 2 files `X.bin` and `X.vec` and allow user to specify `load_fasttext_format("X")` (without extension) and automatically load `X.bin` (instead of specified `X` that doesnt exist).
Well asking you, the project maintainer from when these changes happened and were merged, seemed like a good start to understand them :)

If the filename change is not needed (or its reason is undocumented/unknown/outdated), then I propose simply removing it. Silently modifying user input looks like an anti-pattern to me.

Feel free to open a PR when you have the time. We'll have to be careful with updating tutorials too (if they rely on this silent extension promotion).
I'm already not a maintainer @piskvorky, this change was introduced 2 years ago (when I only start, just merged PR by Jayant approve), I really remember nothing about it.

I found an "source" of this changes, this code firstly was introduced in #1341 (and after moves to other classes, like `FastText`).

@jayantj @prakhar2b can you comment that?
My understanding is this mirrors FB behavior. You specify model_name, and it creates model_name.bin and model_name.vec.

This feature allows the user to specify model_name only and not worry about the file extension. In my opinion, we’re better off without it.

Resolving patch:
diff --git a/gensim/models/fasttext.py b/gensim/models/fasttext.py
--- a/gensim/models/fasttext.py
+++ b/gensim/models/fasttext.py
@@ -1186,8 +1186,13 @@ def load_facebook_model(path, encoding='utf-8'):
     Facebook provides both `.vec` and `.bin` files with their modules.
     The former contains human-readable vectors.
     The latter contains machine-readable vectors along with other model parameters.
-    This function effectively ignores `.vec` output file, since that file is redundant.
-    It only needs the `.bin` file.
+    This function requires you to **provide the full path to the .bin file**.
+    It effectively ignores the `.vec` output file, since it is redundant.
+
+    This function uses the smart_open library to open the path.
+    The path may be on a remote host (e.g. HTTP, S3, etc).
+    It may also be gzip or bz2 compressed (i.e. end in `.bin.gz` or `.bin.bz2`).
+    For details, see `<https://github.com/RaRe-Technologies/smart_open>`__.
 
     Parameters
     ----------
@@ -1228,6 +1233,12 @@ def load_facebook_model(path, encoding='utf-8'):
     gensim.models.fasttext.FastText
         The loaded model.
 
+    See Also
+    --------
+    :func:`~gensim.models.fasttext.load_facebook_vectors` loads
+    the word embeddings only.  Its faster, but does not enable you to continue
+    training.
+
     """
     return _load_fasttext_format(path, encoding=encoding, full_model=True)
 
@@ -1235,6 +1246,19 @@ def load_facebook_model(path, encoding='utf-8'):
 def load_facebook_vectors(path, encoding='utf-8'):
     """Load word embeddings from a model saved in Facebook's native fasttext `.bin` format.
 
+    Notes
+    ------
+    Facebook provides both `.vec` and `.bin` files with their modules.
+    The former contains human-readable vectors.
+    The latter contains machine-readable vectors along with other model parameters.
+    This function requires you to **provide the full path to the .bin file**.
+    It effectively ignores the `.vec` output file, since it is redundant.
+
+    This function uses the smart_open library to open the path.
+    The path may be on a remote host (e.g. HTTP, S3, etc).
+    It may also be gzip or bz2 compressed.
+    For details, see `<https://github.com/RaRe-Technologies/smart_open>`__.
+
     Parameters
     ----------
     path : str
@@ -1267,8 +1291,7 @@ def load_facebook_vectors(path, encoding='utf-8'):
 
     See Also
     --------
-
-    :meth:`gensim.models.fasttext.FastText.load_facebook_model` loads
+    :func:`~gensim.models.fasttext.load_facebook_model` loads
     the full model, not just word embeddings, and enables you to continue
     model training.
 
@@ -1278,15 +1301,12 @@ def load_facebook_vectors(path, encoding='utf-8'):
 
 
 def _load_fasttext_format(model_file, encoding='utf-8', full_model=True):
-    """Load the input-hidden weight matrix from Facebook's native fasttext `.bin` and `.vec` output files.
+    """Load the input-hidden weight matrix from Facebook's native fasttext `.bin` output files.
 
     Parameters
     ----------
     model_file : str
-        Path to the FastText output files.
-        FastText outputs two model files - `/path/to/model.vec` and `/path/to/model.bin`
-        Expected value for this example: `/path/to/model` or `/path/to/model.bin`,
-        as Gensim requires only `.bin` file to the load entire fastText model.
+        Full path to the FastText model file.
     encoding : str, optional
         Specifies the file encoding.
     full_model : boolean, optional
@@ -1299,8 +1319,6 @@ def _load_fasttext_format(model_file, encoding='utf-8', full_model=True):
         The loaded model.
 
     """
-    if not model_file.endswith('.bin'):
-        model_file += '.bin'
     with smart_open(model_file, 'rb') as fin:
         m = gensim.models._fasttext_bin.load(fin, encoding=encoding, full_model=full_model)
 


Test patch:
diff --git a/gensim/test/test_fasttext.py b/gensim/test/test_fasttext.py
--- a/gensim/test/test_fasttext.py
+++ b/gensim/test/test_fasttext.py
@@ -60,9 +60,9 @@ class TestFastTextModel(unittest.TestCase):
     def setUp(self):
         ft_home = os.environ.get('FT_HOME', None)
         self.ft_path = os.path.join(ft_home, 'fasttext') if ft_home else None
-        self.test_model_file = datapath('lee_fasttext')
+        self.test_model_file = datapath('lee_fasttext.bin')
         self.test_model = gensim.models.fasttext.load_facebook_model(self.test_model_file)
-        self.test_new_model_file = datapath('lee_fasttext_new')
+        self.test_new_model_file = datapath('lee_fasttext_new.bin')
 
     def test_training(self):
         model = FT_gensim(size=10, min_count=1, hs=1, negative=0, seed=42, workers=1)
@@ -298,10 +298,10 @@ def test_load_fasttext_new_format(self):
 
     def test_load_model_supervised(self):
         with self.assertRaises(NotImplementedError):
-            gensim.models.fasttext.load_facebook_model(datapath('pang_lee_polarity_fasttext'))
+            gensim.models.fasttext.load_facebook_model(datapath('pang_lee_polarity_fasttext.bin'))
 
     def test_load_model_with_non_ascii_vocab(self):
-        model = gensim.models.fasttext.load_facebook_model(datapath('non_ascii_fasttext'))
+        model = gensim.models.fasttext.load_facebook_model(datapath('non_ascii_fasttext.bin'))
         self.assertTrue(u'který' in model.wv)
         try:
             model.wv[u'který']
@@ -309,7 +309,7 @@ def test_load_model_with_non_ascii_vocab(self):
             self.fail('Unable to access vector for utf8 encoded non-ascii word')
 
     def test_load_model_non_utf8_encoding(self):
-        model = gensim.models.fasttext.load_facebook_model(datapath('cp852_fasttext'), encoding='cp852')
+        model = gensim.models.fasttext.load_facebook_model(datapath('cp852_fasttext.bin'), encoding='cp852')
         self.assertTrue(u'který' in model.wv)
         try:
             model.wv[u'který']