Repository: piskvorky/gensim

Issue:
Gensim error when loading FastText
Hello,

I downloaded fasttext [English wikipedia](https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki.en.zip)

I simply executed `model = FastText.load_fasttext_format('/Users/smidj/Repositories/snippetbot/data/fasttext/wiki.en/wiki.en')`

I got an error:
```
OSError                                   Traceback (most recent call last)
<ipython-input-3-83dcc56dad43> in <module>()
----> 1 model = FastText.load_fasttext_format('/Users/smidj/Repositories/snippetbot/data/fasttext/wiki.en/wiki.en')

/Users/smidj/.virtualenvs/snippetbot/lib/python3.6/site-packages/gensim/models/wrappers/fasttext.py in load_fasttext_format(cls, model_file)
    236         model = cls()
    237         model.wv = cls.load_word2vec_format('%s.vec' % model_file)
--> 238         model.load_binary_data('%s.bin' % model_file)
    239         return model
    240 

/Users/smidj/.virtualenvs/snippetbot/lib/python3.6/site-packages/gensim/models/wrappers/fasttext.py in load_binary_data(self, model_binary_file)
    254             self.load_model_params(f)
    255             self.load_dict(f)
--> 256             self.load_vectors(f)
    257 
    258     def load_model_params(self, file_handle):

/Users/smidj/.virtualenvs/snippetbot/lib/python3.6/site-packages/gensim/models/wrappers/fasttext.py in load_vectors(self, file_handle)
    301 
    302         self.num_original_vectors = num_vectors
--> 303         self.wv.syn0_all = np.fromstring(file_handle.read(num_vectors * dim * float_size), dtype=dtype)
    304         self.wv.syn0_all = self.wv.syn0_all.reshape((num_vectors, dim))
    305         assert self.wv.syn0_all.shape == (self.bucket + len(self.wv.vocab), self.size), \

OSError: [Errno 22] Invalid argument
```

I use Mac OS Sierra, python 3.6 (in virtaulenv), and gensim version 1.0.1.

Thanks,
Jakub


Discussion:
The corpus seem to be very large to download and hence hard to reproduce. Do you get the same error at a different reasonably sized corpus as well? 
Does a simple `file.read(model_file)' work for you? It might be related to a [bug](http://bugs.python.org/issue24658) with reading large files on osx
@tmylk Indeed, it does produce the same error. Good catch!
Thanks for reporting it. Could you please post a fix or workaround that you find here? Do we need to add `chunksize` to `read`? We need to make it work on OSX.
It may be enough to just use `np.fromfile(file_handle, dtype=dtype)` in place of the wasteful read-it-all-via-a-string `fromstring`/`read`.
Thanks @gojomo for the suggestion. However, in that case I am getting
```
ValueError: cannot reshape array of size 2111621876 into shape (4519370,300)
```
in the next line of `fasttext.py` - `self.wv.syn0_all = self.wv.syn0_all.reshape((num_vectors, dim))`
I see, so the functionally equivalent is ```np.fromfile(file_handle, dtype=dtype, count=num_vectors * dim)```. Let me check whether it works (it takes some time to load the model).
This works, however I noticed a strange thing. Do we set model size correctly? 
The following lines 
```
model = FastText.load_fasttext_format('/Users/smidj/Repositories/snippetbot/data/fasttext/wiki.en/wiki.en')
print(str(model.vector_size))
print(str(model["hello"]))
```
print 100 (should be 300) and then vector of size 300 (correct).
Seems `vector_size` mayalso need to be explicitly set. I would definitely do some sanity checks (or unit tests) bases on tiny example fasttext exports, to be sure there's no mis-sizing/mis-alignment happening (that risks leaving everything free of raised exceptions but totally confused//corrupted about real values).
The problem seems to be at `fasttext.py` in `load_model_params`.

The `dim` argument is assigned to `self.size`, which does not exist yet. I think this line should be as follows 
```
self.vector_size = dim
```

Regarding the suggestion of @gojomo, do we have some reasonable sized fasttext model somewhere or do we need to train it?
I am also thinking that the current functionality does not allow to load just vector file together with the `vector_size`.

We load the `vector_size` in `keyedvectors.py.load_word2vec_format`, but we do not store it anywhere. Maybe we can add also create `self.vector_size` in `KeyedVectors` class and save it for later use in case you just want the model, not the retraining.

What do you think?

Resolving patch:
diff --git a/gensim/models/keyedvectors.py b/gensim/models/keyedvectors.py
--- a/gensim/models/keyedvectors.py
+++ b/gensim/models/keyedvectors.py
@@ -111,6 +111,7 @@ def __init__(self):
         self.syn0norm = None
         self.vocab = {}
         self.index2word = []
+        self.vector_size = None
 
     def save(self, *args, **kwargs):
         # don't bother storing the cached normalized vectors
@@ -194,6 +195,7 @@ def load_word2vec_format(cls, fname, fvocab=None, binary=False, encoding='utf8',
             if limit:
                 vocab_size = min(vocab_size, limit)
             result = cls()
+            result.vector_size = vector_size
             result.syn0 = zeros((vocab_size, vector_size), dtype=datatype)
 
             def add_word(word, weights):
diff --git a/gensim/models/wrappers/fasttext.py b/gensim/models/wrappers/fasttext.py
--- a/gensim/models/wrappers/fasttext.py
+++ b/gensim/models/wrappers/fasttext.py
@@ -259,7 +259,7 @@ def load_binary_data(self, model_binary_file, encoding='utf8'):
     def load_model_params(self, file_handle):
         (dim, ws, epoch, minCount, neg, _, loss, model, bucket, minn, maxn, _, t) = self.struct_unpack(file_handle, '@12i1d')
         # Parameters stored by [Args::save](https://github.com/facebookresearch/fastText/blob/master/src/args.cc)
-        self.size = dim
+        self.vector_size = dim
         self.window = ws
         self.iter = epoch
         self.min_count = minCount
@@ -293,7 +293,7 @@ def load_dict(self, file_handle, encoding='utf8'):
     def load_vectors(self, file_handle):
         num_vectors, dim = self.struct_unpack(file_handle, '@2q')
         # Vectors stored by [Matrix::save](https://github.com/facebookresearch/fastText/blob/master/src/matrix.cc)
-        assert self.size == dim, 'mismatch between model sizes'
+        assert self.vector_size == dim, 'mismatch between model sizes'
         float_size = struct.calcsize('@f')
         if float_size == 4:
             dtype = np.dtype(np.float32)
@@ -301,9 +301,9 @@ def load_vectors(self, file_handle):
             dtype = np.dtype(np.float64)
 
         self.num_original_vectors = num_vectors
-        self.wv.syn0_all = np.fromstring(file_handle.read(num_vectors * dim * float_size), dtype=dtype)
+        self.wv.syn0_all = np.fromfile(file_handle, dtype=dtype, count=num_vectors * dim)
         self.wv.syn0_all = self.wv.syn0_all.reshape((num_vectors, dim))
-        assert self.wv.syn0_all.shape == (self.bucket + len(self.wv.vocab), self.size), \
+        assert self.wv.syn0_all.shape == (self.bucket + len(self.wv.vocab), self.vector_size), \
             'mismatch between weight matrix shape and vocab/model size'
         self.init_ngrams()
 


Test patch:
diff --git a/gensim/test/test_fasttext_wrapper.py b/gensim/test/test_fasttext_wrapper.py
--- a/gensim/test/test_fasttext_wrapper.py
+++ b/gensim/test/test_fasttext_wrapper.py
@@ -8,7 +8,6 @@
 Automated tests for checking transformation algorithms (the models package).
 """
 
-
 import logging
 import unittest
 import os
@@ -19,7 +18,7 @@
 from gensim.models.wrappers import fasttext
 from gensim.models import keyedvectors
 
-module_path = os.path.dirname(__file__) # needed because sample data files are located in the same folder
+module_path = os.path.dirname(__file__)  # needed because sample data files are located in the same folder
 datapath = lambda fname: os.path.join(module_path, 'test_data', fname)
 logger = logging.getLogger(__name__)
 
@@ -28,6 +27,7 @@ def testfile():
     # temporary data will be stored to this file
     return os.path.join(tempfile.gettempdir(), 'gensim_fasttext.tst')
 
+
 class TestFastText(unittest.TestCase):
     @classmethod
     def setUp(self):
@@ -40,8 +40,8 @@ def setUp(self):
 
     def model_sanity(self, model):
         """Even tiny models trained on any corpus should pass these sanity checks"""
-        self.assertEqual(model.wv.syn0.shape, (len(model.wv.vocab), model.size))
-        self.assertEqual(model.wv.syn0_all.shape, (model.num_ngram_vectors, model.size))
+        self.assertEqual(model.wv.syn0.shape, (len(model.wv.vocab), model.vector_size))
+        self.assertEqual(model.wv.syn0_all.shape, (model.num_ngram_vectors, model.vector_size))
 
     def models_equal(self, model1, model2):
         self.assertEqual(len(model1.wv.vocab), len(model2.wv.vocab))
@@ -74,7 +74,7 @@ def testMinCount(self):
             return  # Use self.skipTest once python < 2.7 is no longer supported
         self.assertTrue('forests' not in self.test_model.wv.vocab)
         test_model_min_count_1 = fasttext.FastText.train(
-                self.ft_path, self.corpus_file, output_file=testfile(), size=10, min_count=1)
+            self.ft_path, self.corpus_file, output_file=testfile(), size=10, min_count=1)
         self.assertTrue('forests' in test_model_min_count_1.wv.vocab)
 
     def testModelSize(self):
@@ -83,8 +83,8 @@ def testModelSize(self):
             logger.info("FT_HOME env variable not set, skipping test")
             return  # Use self.skipTest once python < 2.7 is no longer supported
         test_model_size_20 = fasttext.FastText.train(
-                self.ft_path, self.corpus_file, output_file=testfile(), size=20)
-        self.assertEqual(test_model_size_20.size, 20)
+            self.ft_path, self.corpus_file, output_file=testfile(), size=20)
+        self.assertEqual(test_model_size_20.vector_size, 20)
         self.assertEqual(test_model_size_20.wv.syn0.shape[1], 20)
         self.assertEqual(test_model_size_20.wv.syn0_all.shape[1], 20)
 
@@ -118,6 +118,25 @@ def testLoadFastTextFormat(self):
         self.assertEqual(self.test_model.wv.syn0.shape, (vocab_size, model_size))
         self.assertEqual(len(self.test_model.wv.vocab), vocab_size, model_size)
         self.assertEqual(self.test_model.wv.syn0_all.shape, (self.test_model.num_ngram_vectors, model_size))
+        expected_vec = [-0.5714373588562012,
+                        -0.008556111715734005,
+                        0.15747803449630737,
+                        -0.6785456538200378,
+                        -0.25458523631095886,
+                        -0.5807671546936035,
+                        -0.09912964701652527,
+                        1.1446694135665894,
+                        0.23417705297470093,
+                        0.06000664085149765]
+        self.assertTrue(numpy.allclose(self.test_model["hundred"], expected_vec, 0.001))
+        self.assertEquals(self.test_model.min_count, 5)
+        self.assertEquals(self.test_model.window, 5)
+        self.assertEquals(self.test_model.iter, 5)
+        self.assertEquals(self.test_model.negative, 5)
+        self.assertEquals(self.test_model.sample, 0.0001)
+        self.assertEquals(self.test_model.bucket, 1000)
+        self.assertEquals(self.test_model.wv.max_n, 6)
+        self.assertEquals(self.test_model.wv.min_n, 3)
         self.model_sanity(model)
 
     def testLoadModelWithNonAsciiVocab(self):
@@ -145,7 +164,8 @@ def testNSimilarity(self):
         self.assertEqual(self.test_model.n_similarity(['the'], ['and']), self.test_model.n_similarity(['and'], ['the']))
         # Out of vocab check
         self.assertTrue(numpy.allclose(self.test_model.n_similarity(['night', 'nights'], ['nights', 'night']), 1.0))
-        self.assertEqual(self.test_model.n_similarity(['night'], ['nights']), self.test_model.n_similarity(['nights'], ['night']))
+        self.assertEqual(self.test_model.n_similarity(['night'], ['nights']),
+                         self.test_model.n_similarity(['nights'], ['night']))
 
     def testSimilarity(self):
         """Test similarity for in-vocab and out-of-vocab words"""
@@ -169,10 +189,14 @@ def testMostSimilarCosmul(self):
         """Test most_similar_cosmul for in-vocab and out-of-vocab words"""
         # In vocab, sanity check
         self.assertEqual(len(self.test_model.most_similar_cosmul(positive=['the', 'and'], topn=5)), 5)
-        self.assertEqual(self.test_model.most_similar_cosmul('the'), self.test_model.most_similar_cosmul(positive=['the']))
+        self.assertEqual(
+            self.test_model.most_similar_cosmul('the'),
+            self.test_model.most_similar_cosmul(positive=['the']))
         # Out of vocab check
         self.assertEqual(len(self.test_model.most_similar_cosmul(['night', 'nights'], topn=5)), 5)
-        self.assertEqual(self.test_model.most_similar_cosmul('nights'), self.test_model.most_similar_cosmul(positive=['nights']))
+        self.assertEqual(
+            self.test_model.most_similar_cosmul('nights'),
+            self.test_model.most_similar_cosmul(positive=['nights']))
 
     def testLookup(self):
         """Tests word vector lookup for in-vocab and out-of-vocab words"""
@@ -226,6 +250,7 @@ def testHash(self):
         ft_hash = fasttext.FastText.ft_hash('word')
         self.assertEqual(ft_hash, 1788406269)
 
+
 if __name__ == '__main__':
     logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.DEBUG)
     unittest.main()