Repository: piskvorky/gensim

Issue:
Incoherent topic word distributions after `malletmodel2ldamodel`
Hi everyone,
first off many thanks for providing such an awesome module! I am using `gensim` to do topic modeling with LDA and encountered the following bug/issue. I have already read about it in the [mailing list](https://groups.google.com/forum/#!topic/gensim/ZesMoKZCf4c), but apparently no issue has been created on Github.

#### Description
After training an LDA model with the gensim mallet wrapper I converted the model to a native gensim LDA model via the `malletmodel2ldamodel` function provided with the wrapper. Before and after the conversion the topic word distributions are quite different. The ldamallet version returns comprehensible topics with sensible weights, whereas the topic word distribution after conversion is nearly uniform, leading to topics without a clear focus. 

I am assuming that the resulting topics are supposed to be at least somewhat similar before and after conversion. Am I doing something wrong? What could be causing this behaviour?

#### Steps/Code/Corpus to Reproduce
```python
import gensim
from sklearn.datasets import fetch_20newsgroups

# select five quite distinct categories from the 20 newsgroups
cat = ['soc.religion.christian', 'comp.graphics', 'rec.motorcycles', 
       'sci.space', 'talk.politics.guns']

# keep and use only the main text
newsgroups_train = fetch_20newsgroups(subset='all', categories=cat,
                                      remove=('headers', 'footers', 'quotes'))

tokenized = [gensim.utils.simple_preprocess(doc) for doc in newsgroups_train.data]
dictionary = gensim.corpora.Dictionary(tokenized)
corpus = [dictionary.doc2bow(text) for text in tokenized]

lda_mallet = gensim.models.wrappers.ldamallet.LdaMallet(
        'c:/mallet/bin/mallet', corpus=corpus, 
        num_topics=5, id2word=dictionary, iterations=1000)

lda_gensim = gensim.models.wrappers.ldamallet.malletmodel2ldamodel(
        lda_mallet, iterations=1000)

for topic in lda_mallet.show_topics(num_topics=5, num_words=10):
    print(topic)
for topic in lda_gensim.show_topics(num_topics=5, num_words=10):
    print(topic)
```
#### Expected Results
These are the results I get from the mallet wrapper using `lda_mallet.show_topics(num_topics=5, num_words=10)`. Those are what one would expect considering the chosen categories from 20newsgroups: 
```python
(0, '0.021*"god" + 0.009*"people" + 0.007*"jesus" + 0.007*"church" + 0.006*"christ" + 0.005*"life" + 0.005*"christian" + 0.005*"bible" + 0.004*"christians" + 0.004*"man"')
(1, '0.014*"don" + 0.011*"ve" + 0.009*"good" + 0.008*"bike" + 0.007*"time" + 0.007*"back" + 0.007*"make" + 0.006*"ll" + 0.006*"problem" + 0.006*"thing"')
(2, '0.017*"space" + 0.006*"nasa" + 0.006*"earth" + 0.005*"system" + 0.005*"launch" + 0.004*"shuttle" + 0.004*"orbit" + 0.003*"years" + 0.003*"mission" + 0.003*"moon"')
(3, '0.012*"people" + 0.011*"gun" + 0.005*"guns" + 0.005*"government" + 0.005*"state" + 0.005*"law" + 0.005*"fire" + 0.005*"control" + 0.004*"don" + 0.004*"fbi"')
(4, '0.013*"image" + 0.009*"graphics" + 0.008*"jpeg" + 0.007*"file" + 0.006*"images" + 0.006*"data" + 0.006*"bit" + 0.006*"software" + 0.006*"ftp" + 0.006*"mail"')
```
#### Actual Results
These are the results I get from the converted native gensim model using `lda_gensim.show_topics(num_topics=5, num_words=10)`. The word probabilities are all very low and not very distinctive, resulting in mostly incoherent topics: 
```python
(0, '0.000*"tribunal" + 0.000*"insruance" + 0.000*"damper" + 0.000*"unfurl" + 0.000*"urinalisys" + 0.000*"saturnation" + 0.000*"stupider" + 0.000*"improved" + 0.000*"waltons" + 0.000*"t_ng"')
(1, '0.000*"ott" + 0.000*"raved" + 0.000*"warped" + 0.000*"onesies" + 0.000*"speculating" + 0.000*"irrigate" + 0.000*"bodies" + 0.000*"inherant" + 0.000*"illustrations" + 0.000*"filler"')
(2, '0.000*"datasets" + 0.000*"addiction" + 0.000*"lr" + 0.000*"overturning" + 0.000*"supertrapp" + 0.000*"collision" + 0.000*"nl__" + 0.000*"someone" + 0.000*"switch" + 0.000*"pirate"')
(3, '0.000*"inbetweens" + 0.000*"hostname" + 0.000*"obsevatory" + 0.000*"dscharge" + 0.000*"ecclesiates" + 0.000*"drills" + 0.000*"ranching" + 0.000*"metz" + 0.000*"omnivorous" + 0.000*"normals"')
(4, '0.000*"uad" + 0.000*"undecidable" + 0.000*"eroded" + 0.000*"summarized" + 0.000*"reposition" + 0.000*"sttod" + 0.000*"sanctas" + 0.000*"broadest" + 0.000*"inception" + 0.000*"turntable"')
```

#### Versions
- Windows-7-6.1.7601-SP1
- Python 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 16:13:55) [MSC v.1900 64 bit (AMD64)]
- NumPy 1.14.2
- SciPy 1.0.1
- gensim 3.4.0
- FAST_VERSION 1
- mallet version 2.0.8


Thanks in advance for any help! Cheers,
Wolfgang
Incoherent topic word distributions after `malletmodel2ldamodel`
Hi everyone,
first off many thanks for providing such an awesome module! I am using `gensim` to do topic modeling with LDA and encountered the following bug/issue. I have already read about it in the [mailing list](https://groups.google.com/forum/#!topic/gensim/ZesMoKZCf4c), but apparently no issue has been created on Github.

#### Description
After training an LDA model with the gensim mallet wrapper I converted the model to a native gensim LDA model via the `malletmodel2ldamodel` function provided with the wrapper. Before and after the conversion the topic word distributions are quite different. The ldamallet version returns comprehensible topics with sensible weights, whereas the topic word distribution after conversion is nearly uniform, leading to topics without a clear focus. 

I am assuming that the resulting topics are supposed to be at least somewhat similar before and after conversion. Am I doing something wrong? What could be causing this behaviour?

#### Steps/Code/Corpus to Reproduce
```python
import gensim
from sklearn.datasets import fetch_20newsgroups

# select five quite distinct categories from the 20 newsgroups
cat = ['soc.religion.christian', 'comp.graphics', 'rec.motorcycles', 
       'sci.space', 'talk.politics.guns']

# keep and use only the main text
newsgroups_train = fetch_20newsgroups(subset='all', categories=cat,
                                      remove=('headers', 'footers', 'quotes'))

tokenized = [gensim.utils.simple_preprocess(doc) for doc in newsgroups_train.data]
dictionary = gensim.corpora.Dictionary(tokenized)
corpus = [dictionary.doc2bow(text) for text in tokenized]

lda_mallet = gensim.models.wrappers.ldamallet.LdaMallet(
        'c:/mallet/bin/mallet', corpus=corpus, 
        num_topics=5, id2word=dictionary, iterations=1000)

lda_gensim = gensim.models.wrappers.ldamallet.malletmodel2ldamodel(
        lda_mallet, iterations=1000)

for topic in lda_mallet.show_topics(num_topics=5, num_words=10):
    print(topic)
for topic in lda_gensim.show_topics(num_topics=5, num_words=10):
    print(topic)
```
#### Expected Results
These are the results I get from the mallet wrapper using `lda_mallet.show_topics(num_topics=5, num_words=10)`. Those are what one would expect considering the chosen categories from 20newsgroups: 
```python
(0, '0.021*"god" + 0.009*"people" + 0.007*"jesus" + 0.007*"church" + 0.006*"christ" + 0.005*"life" + 0.005*"christian" + 0.005*"bible" + 0.004*"christians" + 0.004*"man"')
(1, '0.014*"don" + 0.011*"ve" + 0.009*"good" + 0.008*"bike" + 0.007*"time" + 0.007*"back" + 0.007*"make" + 0.006*"ll" + 0.006*"problem" + 0.006*"thing"')
(2, '0.017*"space" + 0.006*"nasa" + 0.006*"earth" + 0.005*"system" + 0.005*"launch" + 0.004*"shuttle" + 0.004*"orbit" + 0.003*"years" + 0.003*"mission" + 0.003*"moon"')
(3, '0.012*"people" + 0.011*"gun" + 0.005*"guns" + 0.005*"government" + 0.005*"state" + 0.005*"law" + 0.005*"fire" + 0.005*"control" + 0.004*"don" + 0.004*"fbi"')
(4, '0.013*"image" + 0.009*"graphics" + 0.008*"jpeg" + 0.007*"file" + 0.006*"images" + 0.006*"data" + 0.006*"bit" + 0.006*"software" + 0.006*"ftp" + 0.006*"mail"')
```
#### Actual Results
These are the results I get from the converted native gensim model using `lda_gensim.show_topics(num_topics=5, num_words=10)`. The word probabilities are all very low and not very distinctive, resulting in mostly incoherent topics: 
```python
(0, '0.000*"tribunal" + 0.000*"insruance" + 0.000*"damper" + 0.000*"unfurl" + 0.000*"urinalisys" + 0.000*"saturnation" + 0.000*"stupider" + 0.000*"improved" + 0.000*"waltons" + 0.000*"t_ng"')
(1, '0.000*"ott" + 0.000*"raved" + 0.000*"warped" + 0.000*"onesies" + 0.000*"speculating" + 0.000*"irrigate" + 0.000*"bodies" + 0.000*"inherant" + 0.000*"illustrations" + 0.000*"filler"')
(2, '0.000*"datasets" + 0.000*"addiction" + 0.000*"lr" + 0.000*"overturning" + 0.000*"supertrapp" + 0.000*"collision" + 0.000*"nl__" + 0.000*"someone" + 0.000*"switch" + 0.000*"pirate"')
(3, '0.000*"inbetweens" + 0.000*"hostname" + 0.000*"obsevatory" + 0.000*"dscharge" + 0.000*"ecclesiates" + 0.000*"drills" + 0.000*"ranching" + 0.000*"metz" + 0.000*"omnivorous" + 0.000*"normals"')
(4, '0.000*"uad" + 0.000*"undecidable" + 0.000*"eroded" + 0.000*"summarized" + 0.000*"reposition" + 0.000*"sttod" + 0.000*"sanctas" + 0.000*"broadest" + 0.000*"inception" + 0.000*"turntable"')
```

#### Versions
- Windows-7-6.1.7601-SP1
- Python 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 16:13:55) [MSC v.1900 64 bit (AMD64)]
- NumPy 1.14.2
- SciPy 1.0.1
- gensim 3.4.0
- FAST_VERSION 1
- mallet version 2.0.8


Thanks in advance for any help! Cheers,
Wolfgang


Discussion:
I am also having this, or a related, problem, with gensim 3.1. 
I am trying now with gensim 3.5 and I will update if the issue still occurs.
I feel this bug should be fixable. 
I tested with gensim 3.5 and encounter the same problem. This essentially makes `malletmodel2ldamodel` worthless.  
@Wolfi-101 thanks for the report, issue reproduced with `gensim==3.5.0` :+1:
Any news on this? I switched to using mallet for a study I'm doing but would still like to use pyLDAvis for consistency with previous work. I'm stuck with either
AttributeError: 'LdaMallet' object has no attribute 'inference'
or 
gensim.models.wrappers.ldamallet.malletmodel2ldamodel()
returning random terms in the topics.
Using gensim 3.5 mallet 2.0.8

@mikeyearworth this looks unrelated to the current issue, can you provide full code example for reproducing your error please (with all needed data of course)
```python
model = gensim.models.wrappers.LdaMallet('/opt/local/bin/mallet', corpus=mikeycorpus, num_topics=num_topics, id2word=mikeydictionary, workers=3)
data = pyLDAvis.gensim.prepare(model, mikeycorpus, mikeydictionary, mds='pcoa')
pyLDAvis.display(data)

`---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-1-83610b8776d0> in <module>()
    130 
    131 model = gensim.models.wrappers.LdaMallet('/opt/local/bin/mallet', corpus=mikeycorpus, num_topics=num_topics, id2word=mikeydictionary, workers=3)
--> 132 data = pyLDAvis.gensim.prepare(model, mikeycorpus, mikeydictionary, mds='pcoa')
    133 pyLDAvis.display(data)
    134 

/anaconda2/lib/python2.7/site-packages/pyLDAvis/gensim.pyc in prepare(topic_model, corpus, dictionary, doc_topic_dist, **kwargs)
    109     See `pyLDAvis.prepare` for **kwargs.
    110     """
--> 111     opts = fp.merge(_extract_data(topic_model, corpus, dictionary, doc_topic_dist), kwargs)
    112     return vis_prepare(**opts)

/anaconda2/lib/python2.7/site-packages/pyLDAvis/gensim.pyc in _extract_data(topic_model, corpus, dictionary, doc_topic_dists)
     40           gamma = topic_model.inference(corpus)
     41       else:
---> 42           gamma, _ = topic_model.inference(corpus)
     43       doc_topic_dists = gamma / gamma.sum(axis=1)[:, None]
     44 

AttributeError: 'LdaMallet' object has no attribute 'inference'`
```
Whereas
```python
ldamodel = gensim.models.wrappers.ldamallet.malletmodel2ldamodel(model)
data = pyLDAvis.gensim.prepare(ldamodel, mikeycorpus, mikeydictionary, mds='pcoa')
pyLDAvis.display(data)
```

generates 
<img width="1220" alt="screen shot 2018-08-10 at 16 26 08" src="https://user-images.githubusercontent.com/12748225/43966577-2b1c6f6c-9cba-11e8-9f4e-f81c185930ef.png">

etc for all topics, compared to the actual model.


@mikeyearworth as I know, `pyLDAVis` support only `LdaModel` and `LdaMulticore`, not `LdaMallet`.

For viz `LdaMallet` you need to convert it to `LdaModel` using `malletmodel2ldamodel` first (and current thread about this function function, it doesn't works correctly).
Jeri Wieringa wrote a tutorial on using data pyLDAvis with mallett. You load the model directly from the state object and do some transformations. I was able to make this work by adapting her code.

http://jeriwieringa.com/2018/07/17/pyLDAviz-and-Mallet/#comment-4018495276

So here is a work around until malletmodel2ldamodel is fixed.
thanks @groceryheist (and Jeri Wieringa) that works fine. 

Do you know if there is a way force the gensim wrapper for mallet to specify the state filename, or return it? mallet writes a new state file each run to an obscure location /var/folders/1x/93zy0_k93gj_xvrk4v_j96_m0000gp/T/XXXXXX_state.mallet.gz where XXXXXX is random each time.
The prefix parameter in the wrapper does this.
Great! Thanks @groceryheist. 
@menshikh-iv is there any update on this? what is this bug fix priority? Thanks :)
I start to work on this issue.
I am also having this, or a related, problem, with gensim 3.1. 
I am trying now with gensim 3.5 and I will update if the issue still occurs.
I feel this bug should be fixable. 
I tested with gensim 3.5 and encounter the same problem. This essentially makes `malletmodel2ldamodel` worthless.  
@Wolfi-101 thanks for the report, issue reproduced with `gensim==3.5.0` :+1:
Any news on this? I switched to using mallet for a study I'm doing but would still like to use pyLDAvis for consistency with previous work. I'm stuck with either
AttributeError: 'LdaMallet' object has no attribute 'inference'
or 
gensim.models.wrappers.ldamallet.malletmodel2ldamodel()
returning random terms in the topics.
Using gensim 3.5 mallet 2.0.8

@mikeyearworth this looks unrelated to the current issue, can you provide full code example for reproducing your error please (with all needed data of course)
```python
model = gensim.models.wrappers.LdaMallet('/opt/local/bin/mallet', corpus=mikeycorpus, num_topics=num_topics, id2word=mikeydictionary, workers=3)
data = pyLDAvis.gensim.prepare(model, mikeycorpus, mikeydictionary, mds='pcoa')
pyLDAvis.display(data)

`---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-1-83610b8776d0> in <module>()
    130 
    131 model = gensim.models.wrappers.LdaMallet('/opt/local/bin/mallet', corpus=mikeycorpus, num_topics=num_topics, id2word=mikeydictionary, workers=3)
--> 132 data = pyLDAvis.gensim.prepare(model, mikeycorpus, mikeydictionary, mds='pcoa')
    133 pyLDAvis.display(data)
    134 

/anaconda2/lib/python2.7/site-packages/pyLDAvis/gensim.pyc in prepare(topic_model, corpus, dictionary, doc_topic_dist, **kwargs)
    109     See `pyLDAvis.prepare` for **kwargs.
    110     """
--> 111     opts = fp.merge(_extract_data(topic_model, corpus, dictionary, doc_topic_dist), kwargs)
    112     return vis_prepare(**opts)

/anaconda2/lib/python2.7/site-packages/pyLDAvis/gensim.pyc in _extract_data(topic_model, corpus, dictionary, doc_topic_dists)
     40           gamma = topic_model.inference(corpus)
     41       else:
---> 42           gamma, _ = topic_model.inference(corpus)
     43       doc_topic_dists = gamma / gamma.sum(axis=1)[:, None]
     44 

AttributeError: 'LdaMallet' object has no attribute 'inference'`
```
Whereas
```python
ldamodel = gensim.models.wrappers.ldamallet.malletmodel2ldamodel(model)
data = pyLDAvis.gensim.prepare(ldamodel, mikeycorpus, mikeydictionary, mds='pcoa')
pyLDAvis.display(data)
```

generates 
<img width="1220" alt="screen shot 2018-08-10 at 16 26 08" src="https://user-images.githubusercontent.com/12748225/43966577-2b1c6f6c-9cba-11e8-9f4e-f81c185930ef.png">

etc for all topics, compared to the actual model.


@mikeyearworth as I know, `pyLDAVis` support only `LdaModel` and `LdaMulticore`, not `LdaMallet`.

For viz `LdaMallet` you need to convert it to `LdaModel` using `malletmodel2ldamodel` first (and current thread about this function function, it doesn't works correctly).
Jeri Wieringa wrote a tutorial on using data pyLDAvis with mallett. You load the model directly from the state object and do some transformations. I was able to make this work by adapting her code.

http://jeriwieringa.com/2018/07/17/pyLDAviz-and-Mallet/#comment-4018495276

So here is a work around until malletmodel2ldamodel is fixed.
thanks @groceryheist (and Jeri Wieringa) that works fine. 

Do you know if there is a way force the gensim wrapper for mallet to specify the state filename, or return it? mallet writes a new state file each run to an obscure location /var/folders/1x/93zy0_k93gj_xvrk4v_j96_m0000gp/T/XXXXXX_state.mallet.gz where XXXXXX is random each time.
The prefix parameter in the wrapper does this.
Great! Thanks @groceryheist. 
@menshikh-iv is there any update on this? what is this bug fix priority? Thanks :)
I start to work on this issue.

Resolving patch:
diff --git a/gensim/models/wrappers/ldamallet.py b/gensim/models/wrappers/ldamallet.py
--- a/gensim/models/wrappers/ldamallet.py
+++ b/gensim/models/wrappers/ldamallet.py
@@ -588,9 +588,11 @@ def malletmodel2ldamodel(mallet_model, gamma_threshold=0.001, iterations=50):
     """
     model_gensim = LdaModel(
         id2word=mallet_model.id2word, num_topics=mallet_model.num_topics,
-        alpha=mallet_model.alpha, iterations=iterations,
+        alpha=mallet_model.alpha, eta=0,
+        iterations=iterations,
         gamma_threshold=gamma_threshold,
         dtype=numpy.float64  # don't loose precision when converting from MALLET
     )
-    model_gensim.expElogbeta[:] = mallet_model.wordtopics
+    model_gensim.state.sstats[...] = mallet_model.wordtopics
+    model_gensim.sync_state()
     return model_gensim


Test patch:
diff --git a/gensim/test/test_ldamallet_wrapper.py b/gensim/test/test_ldamallet_wrapper.py
--- a/gensim/test/test_ldamallet_wrapper.py
+++ b/gensim/test/test_ldamallet_wrapper.py
@@ -19,9 +19,11 @@
 from gensim.corpora import mmcorpus, Dictionary
 from gensim.models.wrappers import ldamallet
 from gensim import matutils
+from gensim.utils import simple_preprocess
 from gensim.models import ldamodel
 from gensim.test import basetmtests
 from gensim.test.utils import datapath, get_tmpfile, common_texts
+import gensim.downloader as api
 
 dictionary = Dictionary(common_texts)
 corpus = [dictionary.doc2bow(text) for text in common_texts]
@@ -90,6 +92,10 @@ def testMallet2Model(self):
 
         tm1 = ldamallet.LdaMallet(self.mallet_path, corpus=corpus, num_topics=2, id2word=dictionary)
         tm2 = ldamallet.malletmodel2ldamodel(tm1)
+
+        # set num_topics=-1 to exclude random influence
+        self.assertEqual(tm1.show_topics(-1, 10), tm2.show_topics(-1, 10))
+
         for document in corpus:
             element1_1, element1_2 = tm1[document][0]
             element2_1, element2_2 = tm2[document][0]
@@ -101,7 +107,20 @@ def testMallet2Model(self):
             self.assertAlmostEqual(element1_2, element2_2, 1)
             logging.debug('%d %d', element1_1, element2_1)
             logging.debug('%d %d', element1_2, element2_2)
-            logging.debug('%d %d', tm1[document][1], tm2[document][1])
+            logging.debug('%s %s', tm1[document][1], tm2[document][1])
+
+    def testMallet2ModelOn20NewsGroups(self):
+        corpus = [simple_preprocess(doc["data"]) for doc in api.load("20-newsgroups")]
+        dictionary = Dictionary(corpus)
+
+        corpus = [dictionary.doc2bow(text) for text in corpus]
+
+        lda_mallet_model = ldamallet.LdaMallet(
+            self.mallet_path, corpus=corpus,
+            num_topics=20, id2word=dictionary, iterations=500)
+
+        lda_gensim_model = ldamallet.malletmodel2ldamodel(lda_mallet_model, iterations=1000)
+        self.assertEqual(lda_mallet_model.show_topics(20, 50), lda_gensim_model.show_topics(20, 50))
 
     def testPersistence(self):
         if not self.mallet_path: