Repository: piskvorky/gensim

Issue:
Bucket Argument in fasttext not working as expected ?
Hi, For the fasttext native from gensim:

My understanding is that according to the hashing trick, if bucket is < total # of subwords, there will be collisions and some subwords will be mapped to the same integers.  Am I wrong? 
However, it is not what I see on a toy example: 

```python
import gensim
from gensim.models.fasttext import FastText

sent = [['lol', 'dds', 'sdsf'], ['anticonsti']]
model = FastText(min_count = 1, bucket = 20)
model.build_vocab(sentences=sent)
model.train(sentences = sent, epochs = 1, report_delay = 1.0)

model.wv.ngrams
```
#### Expected Results
Dictionary with ngrams and their mappings to integers between 0 and 19 ( buckets = 20)

#### Actual Results
Dictionary with ngrams and their mappings to integers between 0 and 55 ( number of ngrams is 56 here)

#### Versions
>>> import platform; print(platform.platform())
Windows-10-10.0.14393-SP0
>>> import sys; print("Python", sys.version)
Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul  5 2016, 11:41:13) [MSC v.1900 64 bit (AMD64)]
>>> import numpy; print("NumPy", numpy.__version__)
NumPy 1.13.3
>>> import scipy; print("SciPy", scipy.__version__)
SciPy 1.0.0
>>> import gensim; print("gensim", gensim.__version__)
gensim 3.1.0
>>> from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
FAST_VERSION 0
>>> 






Discussion:
Thanks for report @saroufimc1, I went through code with debugger and found that "suspicious resize" of `model.wv.syn0_ngrams` happen in https://github.com/manneshiva/gensim/blob/2560e1d6eaeb1b6e18fb29bd1a6e460a141c5f70/gensim/models/fasttext.py#L332

@manneshiva please look more closely (and fix if this a bug)
@menshikh-iv My code hasn't been merged yet, so I doubt @saroufimc1 used my code (unless he installed Gensim from my branch). The exact same problem appears to be here:
https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/models/fasttext.py#L150
I did not touch/change this part of the code. Will have a look at it soon and give my comments.
Confirmed that this is a bug caused by incorrect setting of `ngram_hash` [here.](https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/models/fasttext.py#L141). Thanks @saroufimc1 for raising this issue.

Resolving patch:
diff --git a/gensim/models/fasttext.py b/gensim/models/fasttext.py
--- a/gensim/models/fasttext.py
+++ b/gensim/models/fasttext.py
@@ -138,7 +138,7 @@ def init_ngrams(self, update=False):
             ngram_indices = []
             new_hash_count = 0
             for i, ngram in enumerate(all_ngrams):
-                ngram_hash = ft_hash(ngram)
+                ngram_hash = ft_hash(ngram) % self.bucket
                 if ngram_hash in self.wv.hash2index:
                     self.wv.ngrams[ngram] = self.wv.hash2index[ngram_hash]
                 else:
@@ -160,7 +160,7 @@ def init_ngrams(self, update=False):
             logger.info("Number of new ngrams is %d", len(new_ngrams))
             new_hash_count = 0
             for i, ngram in enumerate(new_ngrams):
-                ngram_hash = ft_hash(ngram)
+                ngram_hash = ft_hash(ngram) % self.bucket
                 if ngram_hash not in self.wv.hash2index:
                     self.wv.hash2index[ngram_hash] = new_hash_count + self.old_hash2index_len
                     self.wv.ngrams[ngram] = self.wv.hash2index[ngram_hash]


Test patch:
diff --git a/gensim/test/test_fasttext.py b/gensim/test/test_fasttext.py
--- a/gensim/test/test_fasttext.py
+++ b/gensim/test/test_fasttext.py
@@ -477,6 +477,13 @@ def test_persistence_word2vec_format(self):
         self.assertRaises(DeprecationWarning, FT_gensim.load_word2vec_format, tmpf)
         self.assertRaises(NotImplementedError, FastTextKeyedVectors.load_word2vec_format, tmpf)
 
+    def test_bucket_ngrams(self):
+        model = FT_gensim(size=10, min_count=1, bucket=20)
+        model.build_vocab(sentences)
+        self.assertEqual(model.wv.syn0_ngrams.shape, (20, 10))
+        model.build_vocab(new_sentences, update=True)
+        self.assertEqual(model.wv.syn0_ngrams.shape, (20, 10))
+
 
 if __name__ == '__main__':
     logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.DEBUG)