Repository: piskvorky/gensim

Issue:
FastText segfaults for some ngram ranges
hey,

I feel something is bad with my text data. I used common crawl for text data and even without any preprocesing (to dismiss as error source) the fasttext model stops during training and just exists without any messages. That is only for some ngram ranges like:

```
model=FastText(sentences,min_n=4)
print(model)
```
In this case no modell summary is printed and the model was not built but there is no message at all.

Giving as paramater instead just `min_n=2, max_n=4` the model is built and works fine.

I used my text data and the sentence iterator like here

```
>>> from gensim.utils import tokenize
>>> import smart_open
>>>
>>>
>>> class MyIter(object):
...     def __iter__(self):
...         path = datapath('crime-and-punishment.txt')
...         with smart_open.smart_open(path, 'r', encoding='utf-8') as fin:
...             for line in fin:
...                 yield list(tokenize(line))

```

with above fasttext statement. 


Might this comes from bad encoding like `\xad`? How to find out what is bad in my data?


Discussion:
Thank you for the report. Sounds like a bug as opposed to a data problem.

> In this case no modell summary is printed and the model was not built but there is no message at all.

Not seeing ANY output very odd. Did the Python subprocess segfault? Was there a core dump or non-zero exit code?

We changed the ngram implementation recently. Does the bug persist with Gensim 3.6.0?

Can you truncate the dataset and still reproduce the problem? If yes, please post your truncated data here.

Also, what OS and Python version are you using?
Seems related to [this](https://groups.google.com/forum/#!searchin/gensim/tradphy%7Csort:date/gensim/oclDUS0fKiA/a1-Cpj9WHwAJ) and [this](https://groups.google.com/forum/#!topic/gensim/jNLbT4mVnh4) thread on the mailing list.
Thank you @mpenkov !

>Did the Python subprocess segfault? Was there a core dump or non-zero exit code?

If this would be the case is this shown inside the console? Because there was really no message it just quits.

>We changed the ngram implementation recently. Does the bug persist with Gensim 3.6.0?

That sounds good. I will try this version!

I use windows 10, 4 core 64 bit i7 with python 3.6

I would like not to post my data but I test it on a open data set. Where can I post the data to you personally maybe?
> If this would be the case is this shown inside the console?

I’m not familiar with Windows, so I can’t answer that, sorry.

> Where can I post the data to you personally maybe?

Please try reducing the data to a smaller subset first. It’s highly likely that you can reproduce the bug with a few sentences as opposed to an entire corpus.
@ctrado18 are you the same person as on the mailing list?
Hey guys,

I tested with v3.6. Still same. Also I used the test data https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/test/test_data/alldata-id-10.txt

 I removed my preprocessor to exlude this as a problem. But it is still the same.

So, it is not a problem with my data or iterator and preprocessor. That rises a somehow deeper issue with my hardware? But I uses a very new one.

So, what can I do to find out what is going on here?

Here is my code together with above test data:
```

class MyIter(object):
    def __iter__(self):
        path = datapath('dat.txt')
        with smart_open.smart_open(path, 'r', encoding='utf-8') as fin:
            for line in fin:
                s = sentence_detector.tokenize(line)
                for k in s:
                        if k:
                            yield list(tokenize(k))

model = FastText(sentences=MyIter(),sg=1,hs=1,min_n=4,max_n=6)
print(model)
```
It is just quitting without print anything. For min_n=2 and max_n=4 it works! Strange.

Also using  without sg=hs=1 just:

```
model = FastText(sentences=MyIter()min_n=4,max_n=6)
print(model)
```

it works. So I think it is also about the skipgram modus?

> @ctrado18 are you the same person as on the mailing list?

I am sorry for the inflationary discussion but I felt missunderstood first because problem was clear for me. I hope everything is clear now.

Resolving patch:
diff --git a/gensim/models/word2vec.py b/gensim/models/word2vec.py
--- a/gensim/models/word2vec.py
+++ b/gensim/models/word2vec.py
@@ -1775,33 +1775,7 @@ def create_binary_tree(self, wv):
         Called internally from :meth:`~gensim.models.word2vec.Word2VecVocab.build_vocab`.
 
         """
-        logger.info("constructing a huffman tree from %i words", len(wv.vocab))
-
-        # build the huffman tree
-        heap = list(itervalues(wv.vocab))
-        heapq.heapify(heap)
-        for i in range(len(wv.vocab) - 1):
-            min1, min2 = heapq.heappop(heap), heapq.heappop(heap)
-            heapq.heappush(
-                heap, Vocab(count=min1.count + min2.count, index=i + len(wv.vocab), left=min1, right=min2)
-            )
-
-        # recurse over the tree, assigning a binary code to each vocabulary word
-        if heap:
-            max_depth, stack = 0, [(heap[0], [], [])]
-            while stack:
-                node, codes, points = stack.pop()
-                if node.index < len(wv.vocab):
-                    # leaf node => store its path from the root
-                    node.code, node.point = codes, points
-                    max_depth = max(len(codes), max_depth)
-                else:
-                    # inner node => continue recursion
-                    points = array(list(points) + [node.index - len(wv.vocab)], dtype=uint32)
-                    stack.append((node.left, array(list(codes) + [0], dtype=uint8), points))
-                    stack.append((node.right, array(list(codes) + [1], dtype=uint8), points))
-
-            logger.info("built huffman tree with maximum node depth %i", max_depth)
+        _assign_binary_codes(wv.vocab)
 
     def make_cum_table(self, wv, domain=2**31 - 1):
         """Create a cumulative-distribution table using stored vocabulary word counts for
@@ -1828,6 +1802,65 @@ def make_cum_table(self, wv, domain=2**31 - 1):
             assert self.cum_table[-1] == domain
 
 
+def _build_heap(vocab):
+    heap = list(itervalues(vocab))
+    heapq.heapify(heap)
+    for i in range(len(vocab) - 1):
+        min1, min2 = heapq.heappop(heap), heapq.heappop(heap)
+        heapq.heappush(
+            heap, Vocab(count=min1.count + min2.count, index=i + len(vocab), left=min1, right=min2)
+        )
+    return heap
+
+
+def _assign_binary_codes(vocab):
+    """
+    Appends a binary code to each vocab term.
+
+    Parameters
+    ----------
+    vocab : dict
+        A dictionary of :class:`gensim.models.word2vec.Vocab` objects.
+
+    Notes
+    -----
+    Expects each term to have an .index attribute that contains the order in
+    which the term was added to the vocabulary.  E.g. term.index == 0 means the
+    term was added to the vocab first.
+
+    Sets the .code and .point attributes of each node.
+    Each code is a numpy.array containing 0s and 1s.
+    Each point is an integer.
+
+    """
+    logger.info("constructing a huffman tree from %i words", len(vocab))
+
+    heap = _build_heap(vocab)
+    if not heap:
+        #
+        # TODO: how can we end up with an empty heap?
+        #
+        logger.info("built huffman tree with maximum node depth 0")
+        return
+
+    # recurse over the tree, assigning a binary code to each vocabulary word
+    max_depth = 0
+    stack = [(heap[0], [], [])]
+    while stack:
+        node, codes, points = stack.pop()
+        if node.index < len(vocab):
+            # leaf node => store its path from the root
+            node.code, node.point = codes, points
+            max_depth = max(len(codes), max_depth)
+        else:
+            # inner node => continue recursion
+            points = array(list(points) + [node.index - len(vocab)], dtype=uint32)
+            stack.append((node.left, array(list(codes) + [0], dtype=uint8), points))
+            stack.append((node.right, array(list(codes) + [1], dtype=uint8), points))
+
+    logger.info("built huffman tree with maximum node depth %i", max_depth)
+
+
 class Word2VecTrainables(utils.SaveLoad):
     """Represents the inner shallow neural network used to train :class:`~gensim.models.word2vec.Word2Vec`."""
     def __init__(self, vector_size=100, seed=1, hashfxn=hash):


Test patch: