Repository: piskvorky/gensim

Issue:
Unpredictable ZeroDivisionErrors in direct_confirmation_measure
Hi guys,

first of all a big thank you for your tremendous work.
I am trying to train a LdaMulticore model guided by a coherence measure (UMASS). However I face the following problem:

- A ZeroDivisionError is thrown sporadically (both with the training and test set set, but definitely more frequently with the latter one). This makes it impossible to compare the coherence measure over a range of number of topics.

```
    num_docs = len(corpus)
    split = train_perc * num_docs // 100
    indices = list(range(num_docs))
    random.shuffle(indices)
   
   # Split into two sets and DISCARD empty documents
    train_corpus = [corpus[i] for i in indices[:split] if corpus[i]]
    test_corpus = [corpus[i] for i in indices[split:] if corpus[i]]

    # Number of passes (heuristic) (e.g. min_runs = 10)
    num_passes = max(num_topics, min_runs)

    lda_model = models.LdaMulticore(train_corpus,
                                         id2word=dictionary,
                                         batch=True,
                                         num_topics=num_topics,
                                         alpha="asymmetric",
                                         passes=num_passes)

    if metric == 'coherence':
            left_score = CoherenceModel(model=lda_model,
                                corpus=test_corpus,
                                dictionary=dictionary,
                                coherence="u_mass").get_coherence()



```

Error log:
```
Traceback (most recent call last):
File "DataGenerator.py", line 198, in <module>
main()
File "DataGenerator.py", line 45, in main
lda_model_file=OUTPUT_LDA_MODEL)
File "/cluster/home/lennartv/ma/python/deep/preprocessing/Projector.py", line 217, in create_projection
num_topics, lda_model = optimize_lda(corpus, dictionary)
File "/cluster/home/lennartv/ma/python/deep/preprocessing/Projector.py", line 107, in optimize_lda
coherence="u_mass").get_coherence()
File "/cluster/home/lennartv/.local/lib64/python3.4/site-packages/gensim/models/coherencemodel.py", line 202, in get_coherence
confirmed_measures = measure.conf(segmented_topics, per_topic_postings, num_docs)
File "/cluster/home/lennartv/.local/lib64/python3.4/site-packages/gensim/topic_coherence/direct_confirmation_measure.py", line 40, in log_conditional_probability
m_lc_i = np.log(((len(co_docs) / float(num_docs)) + EPSILON) / (len(w_star_docs) / float(num_docs)))
ZeroDivisionError: float division by zero
```

I tried to assess the problem and I noticed the following things:

- There seems to be a connection to how well the LDA algorithm converged. Varying the number of passes can mitigate the problem.

I don't have enough insight in the algorithm but the bug seems to stem from the fact that `w_star_docs` in the following code block can become an empty set under certain circumstances:

```
    m_lc = []
    for s_i in segmented_topics:
        for w_prime, w_star in s_i:
            w_prime_docs = per_topic_postings[w_prime]
            w_star_docs = per_topic_postings[w_star]
            co_docs = w_prime_docs.intersection(w_star_docs)
            m_lc_i = np.log(((len(co_docs) / float(num_docs)) + EPSILON) / (len(w_star_docs) / float(num_docs)))
            m_lc.append(m_lc_i)

    return m_lc
```

Thanks in advance for your efforts,
  Lennart


Discussion:
Cc @dsquareindia

Resolving patch:
diff --git a/gensim/topic_coherence/direct_confirmation_measure.py b/gensim/topic_coherence/direct_confirmation_measure.py
--- a/gensim/topic_coherence/direct_confirmation_measure.py
+++ b/gensim/topic_coherence/direct_confirmation_measure.py
@@ -37,7 +37,10 @@ def log_conditional_probability(segmented_topics, per_topic_postings, num_docs):
             w_prime_docs = per_topic_postings[w_prime]
             w_star_docs = per_topic_postings[w_star]
             co_docs = w_prime_docs.intersection(w_star_docs)
-            m_lc_i = np.log(((len(co_docs) / float(num_docs)) + EPSILON) / (len(w_star_docs) / float(num_docs)))
+            if  w_star_docs:
+                m_lc_i = np.log(((len(co_docs) / float(num_docs)) + EPSILON) / (len(w_star_docs) / float(num_docs)))
+            else:
+                m_lc_i = 0
             m_lc.append(m_lc_i)
 
     return m_lc


Test patch: