Repository: piskvorky/gensim

Issue:
LDAmallet shows the same words repeated for all the topics
I run the LDAmallet to generate topics. The results show problem in the implementation.

`# READ DOCUMENTS

```
file_paths = [os.path.join(dir_data, fname) for fname in os.listdir(dir_data) if fname.endswith(".txt") ]
documents = [codecs.open(file_path, 'r', encoding="utf8", errors='ignore').read() for file_path in file_paths ]

# USING NLTK TOKENIZER TO SEPERATE WORDS

tokenizer = RegexpTokenizer(r'\w+')

texts = []
for each in documents:
    tokens = tokenizer.tokenize(each)
    texts.append(tokens)

dictionary = corpora.Dictionary(texts)

corpus = [dictionary.doc2bow(text) for text in texts]

num_topics=numberOfTopics
mallet_path = "C:/mallet-2.0.7/bin/mallet"
ldamodel = gensim.models.wrappers.LdaMallet(mallet_path, corpus, num_topics, dictionary, 4, dir_data, 0 ,1000)

result = (ldamodel.show_topics(num_topics, num_words,formatted=False))
for each in result:
    print each
```

`
However, the output shows the same words repeated for all the topics and a warning for division by 0 when tries to normalize the weights. The output looks like this

`Warning (from warnings module):
  File "C:\Python27\lib\site-packages\gensim\models\wrappers\ldamallet.py", line 239
    topic = topic / topic.sum()  # normalize to probability dist
RuntimeWarning: invalid value encountered in divide
... LDA Model Trained ....
[(nan, u'b1e'), (nan, u'mokak'), (nan, u'viyac'), (nan, u'stipu'), (nan, u'forchange'), (nan, u'dugayiynggu'), (nan, u'hayne'), (nan, u'enrichment'), (nan, u'boseley'), (nan, u'releasesfresh')]
[(nan, u'b1e'), (nan, u'mokak'), (nan, u'viyac'), (nan, u'stipu'), (nan, u'forchange'), (nan, u'dugayiynggu'), (nan, u'hayne'), (nan, u'enrichment'), (nan, u'boseley'), (nan, u'releasesfresh')]
[(nan, u'b1e'), (nan, u'mokak'), (nan, u'viyac'), (nan, u'stipu'), (nan, u'forchange'), (nan, u'dugayiynggu'), (nan, u'hayne'), (nan, u'enrichment'), (nan, u'boseley'), (nan, u'releasesfresh')]
......
`

It is also worth mentioning that the state.mallet.gz output file is almost empty it contains only:

`#doc source pos typeindex type topic
# alpha : 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
# beta : 0.01

`

While the corpus.txt looks good 

`0 0 rest participate apply education attend motivate note application charge interest must local interstate natsiec complete form regional free train half day day day volunteer volunteer volunteer volunteer volunteer volunteer volunteer volunteer look receive
1 0 education education education education train train train train train deputy relate national executive executive number general vocational robinson robinson robinson robinson service employment employment numerou responsible unite research field role role department department department department department include chri chri kingdom prime commissioner commissioner manage centre government government cabinet director director five minister child hold hold hold appoint asqa chief chief chief period officer position senior senior
   .........`

but I am not sure about doctopics.txt output file 

`#doc name topic proportion ...
0   0   5   0.11538461538461539 3   0.11538461538461539 9   0.09615384615384616 8   0.09615384615384616 7   0.09615384615384616 6   0.09615384615384616 4   0.09615384615384616 2   0.09615384615384616 1   0.09615384615384616 0   0.09615384615384616 
`

I thought the document name should be the complete name not ID? Any idea where I went wrong?

Thanks



Discussion:
Which version of gensim are you using?

It is better if you use named parameters in a function call.

The Mallet parameters have changed.

https://github.com/piskvorky/gensim/blob/3d7293f3831250a47cf8e85a7f493a87c7d6ac32/gensim/models/wrappers/ldamallet.py#L54

On Mon, Apr 18, 2016 at 10:25 AM, aaajjj333 notifications@github.com
wrote:

> I run the LDAmallet to generate topics. The results show problem in the
> implementation.
> 
> `# READ DOCUMENTS
> 
> file_paths = [os.path.join(dir_data, fname) for fname in os.listdir(dir_data) if fname.endswith(".txt") ]
> documents = [codecs.open(file_path, 'r', encoding="utf8", errors='ignore').read() for file_path in file_paths ]
> 
> # USING NLTK TOKENIZER TO SEPERATE WORDS
> 
> tokenizer = RegexpTokenizer(r'\w+')
> 
> texts = []
> for each in documents:
>     tokens = tokenizer.tokenize(each)
>     texts.append(tokens)
> 
> dictionary = corpora.Dictionary(texts)
> 
> corpus = [dictionary.doc2bow(text) for text in texts]
> 
> num_topics=numberOfTopics
> mallet_path = "C:/mallet-2.0.7/bin/mallet"
> ldamodel = gensim.models.wrappers.LdaMallet(mallet_path, corpus, num_topics, dictionary, 4, dir_data, 0 ,1000)
> 
> result = (ldamodel.show_topics(num_topics, num_words,formatted=False))
> for each in result:
>     print each
> 
> `
> However, the output shows the same words repeated for all the topics and a
> warning for division by 0 when tries to normalize the weights. The output
> looks like this
> 
> Warning (from warnings module):
> File "C:\Python27\lib\site-packages\gensim\models\wrappers\ldamallet.py",
> line 239
> topic = topic / topic.sum() # normalize to probability dist
> RuntimeWarning: invalid value encountered in divide
> ... LDA Model Trained ....
> [(nan, u'b1e'), (nan, u'mokak'), (nan, u'viyac'), (nan, u'stipu'), (nan,
> u'forchange'), (nan, u'dugayiynggu'), (nan, u'hayne'), (nan,
> u'enrichment'), (nan, u'boseley'), (nan, u'releasesfresh')]
> [(nan, u'b1e'), (nan, u'mokak'), (nan, u'viyac'), (nan, u'stipu'), (nan,
> u'forchange'), (nan, u'dugayiynggu'), (nan, u'hayne'), (nan,
> u'enrichment'), (nan, u'boseley'), (nan, u'releasesfresh')]
> [(nan, u'b1e'), (nan, u'mokak'), (nan, u'viyac'), (nan, u'stipu'), (nan,
> u'forchange'), (nan, u'dugayiynggu'), (nan, u'hayne'), (nan,
> u'enrichment'), (nan, u'boseley'), (nan, u'releasesfresh')]
> ......
> 
> It is also worth mentioning that the state.mallet.gz output file is almost
> empty it contains only:
> 
> #doc source pos typeindex type topic
> #alpha : 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
> #beta : 0.01
> 
> While the corpus.txt looks good
> 
> 0 0 rest participate apply education attend motivate note application
> charge interest must local interstate natsiec complete form regional free
> train half day day day volunteer volunteer volunteer volunteer volunteer
> volunteer volunteer volunteer look receive
> 1 0 education education education education train train train train train
> deputy relate national executive executive number general vocational
> robinson robinson robinson robinson service employment employment numerou
> responsible unite research field role role department department department
> department department include chri chri kingdom prime commissioner
> commissioner manage centre government government cabinet director director
> five minister child hold hold hold appoint asqa chief chief chief period
> officer position senior senior
> .........
> 
> but I am not sure about doctopics.txt output file
> 
> #doc name topic proportion ...
> 0 0 5 0.11538461538461539 3 0.11538461538461539 9 0.09615384615384616 8
> 0.09615384615384616 7 0.09615384615384616 6 0.09615384615384616 4
> 0.09615384615384616 2 0.09615384615384616 1 0.09615384615384616 0
> 0.09615384615384616
> 
> I thought the document name should be the complete name not ID? Any idea
> where I went wrong?
> 
> Thanks
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly or view it on GitHub
> https://github.com/piskvorky/gensim/issues/671

I upgraded gensim, scikit-learn, scipy, and numpy

I changed the LdaMallet call to use named parameters and I still get the same results.

`ldamodel = gensim.models.wrappers.LdaMallet(mallet_path, corpus = mycorpus, num_topics = number_topics, id2word=dictionary, workers = 4, prefix = dir_data, optimize_interval = 0 , iterations= 1000)`

Hi everyone,

I had the same problem while trying to do the Mallet tutorial on the Reuters dataset from [here](http://rare-technologies.com/tutorial-on-mallet-in-python/). Be sure to update the module path of LdaMallet to `model = models.wrappers.LdaMallet` if you want to run this example. It seems to have changed since then. 

In `C:\python34\Lib\site-packages\gensim\models\wrappers\ldamallet.py` editing the function `convert_input`  helped for me. Change
`cmd = self.mallet_path + " import-file --preserve-case --keep-sequence --remove-stopwords --token-regex '\S+' --input %s --output %s"` 
to 
`cmd = self.mallet_path + ' import-file --preserve-case --keep-sequence --remove-stopwords --token-regex "\S+" --input %s --output %s'`. 
This only seems to be a problem on Windows and not on Linux, since single and double quoutes work differently in the respective command lines according to [this answer](http://stackoverflow.com/questions/26605449/infinity-value-error-after-token-regex-command) on SO. The original command causes the file state.mallet.gz to contain only two header lines and the `self.wordtopics` array to be empty. Summing over the empty array in the function `show_topic` yields Zero, resulting in 
`Warning (from warnings module):
  File "C:\Python34\lib\site-packages\gensim\models\wrappers\ldamallet.py", line 220
    topic = topic / topic.sum()  # normalize to probability dist
RuntimeWarning: invalid value encountered in true_divide`.
Hope this helps. Perhaps this is also related to issue #562.

My setup:
- Windows 10 Home
- Python 3.4.3
- Gensim 0.12.4
- Mallet 2.0.8RC3

Oh wow, nice catch @Wolfi-101 ! I didn't know about this [single vs. double quotes issues](http://stackoverflow.com/questions/24173825/what-does-single-quote-do-in-windows-batch-files) in Windows.

@tmylk we should do a review of all process calls in gensim wrappers, to make sure the quotes are correct for Windows too.


Resolving patch:
diff --git a/gensim/models/wrappers/ldamallet.py b/gensim/models/wrappers/ldamallet.py
--- a/gensim/models/wrappers/ldamallet.py
+++ b/gensim/models/wrappers/ldamallet.py
@@ -145,7 +145,7 @@ def convert_input(self, corpus, infer=False, serialize_corpus=True):
                 self.corpus2mallet(corpus, fout)
 
         # convert the text file above into MALLET's internal format
-        cmd = self.mallet_path + " import-file --preserve-case --keep-sequence --remove-stopwords --token-regex '\S+' --input %s --output %s"
+        cmd = self.mallet_path + ' import-file --preserve-case --keep-sequence --remove-stopwords --token-regex "\S+" --input %s --output %s'
         if infer:
             cmd += ' --use-pipe-from ' + self.fcorpusmallet()
             cmd = cmd % (self.fcorpustxt(), self.fcorpusmallet() + '.infer')
@@ -166,7 +166,7 @@ def train(self, corpus):
         logger.info("training MALLET LDA with %s", cmd)
         check_output(cmd, shell=True)
         self.word_topics = self.load_word_topics()
-        # NOTE - we are still keeping the wordtopics variable to not break backward compatibility. 
+        # NOTE - we are still keeping the wordtopics variable to not break backward compatibility.
         # word_topics has replaced wordtopics throughout the code; wordtopics just stores the values of word_topics when train is called.
         self.wordtopics = self.word_topics
 
@@ -260,12 +260,12 @@ def get_version(self, direc_path):
             Check version of mallet via jar file
             """
             archive = zipfile.ZipFile(direc_path, 'r')
-            if u'cc/mallet/regression/' not in archive.namelist():     
+            if u'cc/mallet/regression/' not in archive.namelist():
                 return '2.0.7'
             else:
                 return '2.0.8RC3'
         except Exception:
-            
+
             xml_path = direc_path.split("bin")[0]
             try:
                 doc = et.parse(xml_path + "pom.xml").getroot()
@@ -273,7 +273,7 @@ def get_version(self, direc_path):
                 return doc.find(namespace + 'version').text.split("-")[0]
             except Exception:
                 return "Can't parse pom.xml version file"
-        
+
 
 
     def read_doctopics(self, fname, eps=1e-6, renorm=True):
@@ -304,7 +304,7 @@ def read_doctopics(self, fname, eps=1e-6, renorm=True):
                     if mallet_version == "2.0.7":
                         """
 
-                            1   1   0   1.0780612802674239  30.005575655428533364   2   0.005575655428533364    1   0.005575655428533364    
+                            1   1   0   1.0780612802674239  30.005575655428533364   2   0.005575655428533364    1   0.005575655428533364
                             2   2   0   0.9184413079632608  40.009062076892971008   3   0.009062076892971008    2   0.009062076892971008    1   0.009062076892971008
                             In the above example there is a mix of the above if and elif statement. There are neither `2*num_topics` nor `num_topics` elements.
                             It has 2 formats 40.009062076892971008 and 0   1.0780612802674239 which cannot be handled by above if elif.
@@ -316,14 +316,14 @@ def read_doctopics(self, fname, eps=1e-6, renorm=True):
                         doc = []
                         if len(parts) > 0:
                             while count < len(parts):
-                                """ 
+                                """
                                 if section is to deal with formats of type 2 0.034
                                 so if count reaches index of 2 and since int(2) == float(2) so if block is executed
                                 now  there is one extra element afer 2, so count + 1 access should not give an error
 
                                 else section handles  formats of type 20.034
                                 now count is there on index of 20.034 since float(20.034) != int(20.034) so else block
-                                is executed 
+                                is executed
 
                                 """
                                 if float(parts[count]) == int(parts[count]):


Test patch: