Repository: piskvorky/gensim

Issue:
AssertionError: unexpected number of vectors when loading Korean FB model
Hi,

I downloaded pretrained word vector file (.bin) from facebook 
(https://fasttext.cc/docs/en/crawl-vectors.html)
However, when I tried to use this model it happens to make error.

    from gensim.models import FastText
    fasttext_model = FastText.load_fasttext_format('cc.ko.300.bin', encoding='utf8')

    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 0: invalid continuation byte

But weird thing is that it operates well when I use old version bin file (https://fasttext.cc/docs/en/pretrained-vectors.html)

So I tried to find the solution and found that this problem had happend and solved.

The issue was made through facebook fasttext issue
(https://github.com/facebookresearch/fastText/issues/715)
And they fixed it by
(https://github.com/facebookresearch/fastText/commit/e13484bcb261cda51d33c4940ab5e207aba3ee79)

So, I think gensim load_fasttext_format function had this UnicodeDecodeError because of above problem.

Can you help me to find and solve this problem?

------------------------------------------


I tried changing
word = word_bytes.decode(encoding, errors='replace')
word = word_bytes.decode(encoding, errors='ignore')
in gensim\models\_fasttext_bin.py , line 177
But both made same error
    File "C:\Users\User\PycharmProjects\ksenticnet\venv\lib\site-packages\gensim\models\keyedvectors.py", line 2207, in init_post_load
    assert vectors.shape[0] == vocab_words + self.bucket, 'unexpected number of vectors'
AssertionError: unexpected number of vectors

#### Versions
Windows-10-10.0.17134-SP0
Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit (AMD64)]
NumPy 1.16.2
SciPy 1.2.1
gensim 3.7.1
FAST_VERSION 0




Discussion:
I think this may have been fixed as part of another issue.

Could you please have a look here: https://github.com/RaRe-Technologies/gensim/issues/2378#issuecomment-462808748 and let me know if that fixes your problem?
@mpenkov 
Thanks for your replying.
So I've tried open the unicode branch you said in that comment, 404 Not Found error occurs.
And also, I changed _fasttext_bin.py file as 6dc4aef says.

    for i in range(vocab_size):
        word_bytes = io.BytesIO()
        char_byte = fin.read(1)

        while char_byte != b'\x00':
            word_bytes.write(char_byte)
            char_byte = fin.read(1)
        word_bytes = word_bytes.getvalue()

        word = word_bytes.decode(encoding)
        count, _ = _struct_unpack(fin, '@qb')
        raw_vocab[word] = count

However, the error still occurs.
The reason you're getting a 404 looking for that branch is because that PR got merged.

Please try again with the current `develop` branch and let me know.

(I think the change to _fasttext_bin.py you described is irrelevant to the problem).
@mpenkov 
There is no more unicode decode error, but

```
  Traceback (most recent call last):
  File "C:/Users/사용자/PycharmProjects/Ksenticnet/lab.py", line 16, in <module>
    fasttext_model = FastText.load_fasttext_format('cc.ko.300.bin', encoding='utf8')
  File "C:\Users\사용자\PycharmProjects\Ksenticnet\venv\lib\site-packages\gensim\utils.py", line 1447, in new_func1
    return func(*args, **kwargs)
  File "C:\Users\사용자\PycharmProjects\Ksenticnet\venv\lib\site-packages\gensim\models\fasttext.py", line 965, in load_fasttext_format
    return load_facebook_model(model_file, encoding=encoding)
  File "C:\Users\사용자\PycharmProjects\Ksenticnet\venv\lib\site-packages\gensim\models\fasttext.py", line 1232, in load_facebook_model
    return _load_fasttext_format(path, encoding=encoding, full_model=True)
  File "C:\Users\사용자\PycharmProjects\Ksenticnet\venv\lib\site-packages\gensim\models\fasttext.py", line 1344, in _load_fasttext_format
    model.wv.init_post_load(m.vectors_ngrams)
  File "C:\Users\사용자\PycharmProjects\Ksenticnet\venv\lib\site-packages\gensim\models\keyedvectors.py", line 2235, in init_post_load
    assert vectors.shape[0] == vocab_words + self.bucket, 'unexpected number of vectors'
AssertionError: unexpected number of vectors
```

It happened. 

OK, thank you. I will try reproducing this locally and get back to you with more info.
OK, I've successfully reproduced the error.
There's an inconsistency in the model.

```
(Pdb) p vectors
array([[-2.1880191e-02,  2.4097128e-02,  2.4378835e-01, ...,
        -2.3358483e-02, -4.7362827e-02,  4.2944144e-02],
       [ 1.8272826e-02, -1.2648515e-01, -1.3920291e-02, ...,
         3.3092845e-02, -7.3114678e-02, -4.6189796e-02],
       [-8.3837859e-02, -2.9104140e-02,  5.9615869e-02, ...,
        -7.6464862e-03,  5.7216208e-02,  5.0551396e-02],
       ...,
       [-4.7130170e-03, -1.0642191e-02,  2.4006031e-02, ...,
        -4.8146462e-03,  4.6943068e-03,  9.7508440e-03],
       [ 1.2094555e-02, -3.5290685e-03, -6.1084521e-03, ...,
        -5.9561329e-03,  2.0649242e-04,  8.6368741e-03],
       [-2.1612773e-02, -9.9795591e-04,  6.5802750e-03, ...,
         1.5136318e-02, -3.0362314e-02,  3.0855754e-02]], dtype=float32)
(Pdb) p vectors.shape
(4000000, 300)
(Pdb) p vocab_words
1999989
(Pdb) p self.bucket
2000000
```

The model reports that it has 1999989 words and 200000 buckets, meaning it must have 3999989 vectors (each word and bucket corresponds to a vector).  However, the actual collection of vectors (the matrix) contains 400000 vectors (11 extra vectors).

The reason we're missing 11 vectors is broken unicode. If we enable logging, we see this:

```
ERROR:gensim.models._fasttext_bin:unable to cleanly decode bytes b'\xed\xa1\x9c' to word ''
ERROR:gensim.models._fasttext_bin:unable to cleanly decode bytes b'\xed\xb0\x80' to word ''
ERROR:gensim.models._fasttext_bin:unable to cleanly decode bytes b'\xed\xa7\x80' to word ''
ERROR:gensim.models._fasttext_bin:unable to cleanly decode bytes b'\xed\xb8\xb0' to word ''
ERROR:gensim.models._fasttext_bin:unable to cleanly decode bytes b'\xed\xa0\x9c' to word ''
ERROR:gensim.models._fasttext_bin:unable to cleanly decode bytes b'\xed\xa6\xac' to word ''
ERROR:gensim.models._fasttext_bin:unable to cleanly decode bytes b'\xed\xb3\xb5' to word ''
ERROR:gensim.models._fasttext_bin:unable to cleanly decode bytes b'\xed\xb3\xb4' to word ''
ERROR:gensim.models._fasttext_bin:unable to cleanly decode bytes b'\xed\xa5\xbc' to word ''
ERROR:gensim.models._fasttext_bin:unable to cleanly decode bytes b'\xed\xb3\xa0' to word ''
ERROR:gensim.models._fasttext_bin:unable to cleanly decode bytes b'\xed\xa0\x84' to word ''
ERROR:gensim.models._fasttext_bin:unable to cleanly decode bytes b'\xed\xa0\x95' to word ''
```

There are 12 vocab terms that have broken Unicode. We handle that by ignoring any bad characters. Unfortunately, this sets us up for a collision, because all the above vectors map to the same string (empty string, in this case).

@piskvorky How should we deal with this case? It seems like the model from FB may be broken - perhaps we should reach out to them for more info? The fact that older models work fine (see original post) suggest that something fishy is going on.
Sure, we can report this to FB. But do they actually have a contract in place that models contain valid utf8-encoded Unicode?

IIRC, the original word2vec simply treated strings as binary bytes… (sometimes cutting utf8 multi-byte characters in half when trimming length etc). Inability to decode the strings was treated as "your problem, for us it's just bytes".

Resolving patch:
diff --git a/gensim/models/_fasttext_bin.py b/gensim/models/_fasttext_bin.py
--- a/gensim/models/_fasttext_bin.py
+++ b/gensim/models/_fasttext_bin.py
@@ -29,12 +29,14 @@
 
 """
 
+import codecs
 import collections
 import io
 import logging
 import struct
 
 import numpy as np
+import six
 
 _END_OF_WORD_MARKER = b'\x00'
 
@@ -182,9 +184,9 @@ def _load_vocab(fin, new_format, encoding='utf-8'):
         try:
             word = word_bytes.decode(encoding)
         except UnicodeDecodeError:
-            word = word_bytes.decode(encoding, errors='ignore')
+            word = word_bytes.decode(encoding, errors='backslashreplace')
             logger.error(
-                'failed to decode invalid unicode bytes %r; ignoring invalid characters, using %r',
+                'failed to decode invalid unicode bytes %r; replacing invalid characters, using %r',
                 word_bytes, word
             )
         count, _ = _struct_unpack(fin, '@qb')
@@ -280,3 +282,39 @@ def load(fin, encoding='utf-8', full_model=True):
     model.update(vectors_ngrams=vectors_ngrams, hidden_output=hidden_output)
     model = {k: v for k, v in model.items() if k in _FIELD_NAMES}
     return Model(**model)
+
+
+def _backslashreplace_backport(ex):
+    """Replace byte sequences that failed to decode with character escapes.
+
+    Does the same thing as errors="backslashreplace" from Python 3.  Python 2
+    lacks this functionality out of the box, so we need to backport it.
+
+    Parameters
+    ----------
+    ex: UnicodeDecodeError
+        contains arguments of the string and start/end indexes of the bad portion.
+
+    Returns
+    -------
+    text: unicode
+        The Unicode string corresponding to the decoding of the bad section.
+    end: int
+        The index from which to continue decoding.
+
+    Note
+    ----
+    Works on Py2 only.  Py3 already has backslashreplace built-in.
+
+    """
+    #
+    # Based on:
+    # https://stackoverflow.com/questions/42860186/exact-equivalent-of-b-decodeutf-8-backslashreplace-in-python-2
+    #
+    bstr, start, end = ex.object, ex.start, ex.end
+    text = u''.join('\\x{:02x}'.format(ord(c)) for c in bstr[start:end])
+    return text, end
+
+
+if six.PY2:
+    codecs.register_error('backslashreplace', _backslashreplace_backport)
diff --git a/gensim/models/keyedvectors.py b/gensim/models/keyedvectors.py
--- a/gensim/models/keyedvectors.py
+++ b/gensim/models/keyedvectors.py
@@ -2098,6 +2098,17 @@ def word_vec(self, word, use_norm=False):
             else:
                 ngram_weights = self.vectors_ngrams
             ngram_hashes = ft_ngram_hashes(word, self.min_n, self.max_n, self.bucket, self.compatible_hash)
+            if len(ngram_hashes) == 0:
+                #
+                # If it is impossible to extract _any_ ngrams from the input
+                # word, then the best we can do is return a vector that points
+                # to the origin.  The reference FB implementation does this,
+                # too.
+                #
+                # https://github.com/RaRe-Technologies/gensim/issues/2402
+                #
+                logger.warning('could not extract any ngrams from %r, returning origin vector', word)
+                return word_vec
             for nh in ngram_hashes:
                 word_vec += ngram_weights[nh]
             return word_vec / len(ngram_hashes)


Test patch:
diff --git a/gensim/test/test_fasttext.py b/gensim/test/test_fasttext.py
--- a/gensim/test/test_fasttext.py
+++ b/gensim/test/test_fasttext.py
@@ -1119,6 +1119,13 @@ def test_load_native_vectors(self):
         iv_vector = fbkv['landlady']
         self.assertFalse(np.allclose(oov_vector, iv_vector))
 
+    def test_no_ngrams(self):
+        model = gensim.models.fasttext.load_facebook_model(datapath('crime-and-punishment.bin'))
+
+        v1 = model.wv['']
+        origin = np.zeros(v1.shape, v1.dtype)
+        self.assertTrue(np.allclose(v1, origin))
+
 
 def _train_model_with_pretrained_vectors():
     """Generate toy-model-pretrained.bin for use in test_load_native_pretrained.
@@ -1261,10 +1268,12 @@ def test_bad_unicode(self):
         buf.seek(0)
 
         raw_vocab, vocab_size, nlabels = gensim.models._fasttext_bin._load_vocab(buf, False)
+
         expected = {
-            u'英語版ウィキペディアへの投稿はいつでも': 1,
-            u'административно-территориальн': 2,
+            u'英語版ウィキペディアへの投稿はいつでも\\xe6': 1,
+            u'административно-территориальн\\xd1': 2,
         }
+
         self.assertEqual(expected, dict(raw_vocab))
 
         self.assertEqual(vocab_size, 2)