Repository: piskvorky/gensim

Issue:
LDAmallet shows the same words repeated for all the topics
I run the LDAmallet to generate topics. The results show problem in the implementation.

`# READ DOCUMENTS

```
file_paths = [os.path.join(dir_data, fname) for fname in os.listdir(dir_data) if fname.endswith(".txt") ]
documents = [codecs.open(file_path, 'r', encoding="utf8", errors='ignore').read() for file_path in file_paths ]

# USING NLTK TOKENIZER TO SEPERATE WORDS

tokenizer = RegexpTokenizer(r'\w+')

texts = []
for each in documents:
    tokens = tokenizer.tokenize(each)
    texts.append(tokens)

dictionary = corpora.Dictionary(texts)

corpus = [dictionary.doc2bow(text) for text in texts]

num_topics=numberOfTopics
mallet_path = "C:/mallet-2.0.7/bin/mallet"
ldamodel = gensim.models.wrappers.LdaMallet(mallet_path, corpus, num_topics, dictionary, 4, dir_data, 0 ,1000)

result = (ldamodel.show_topics(num_topics, num_words,formatted=False))
for each in result:
    print each
```

`
However, the output shows the same words repeated for all the topics and a warning for division by 0 when tries to normalize the weights. The output looks like this

`Warning (from warnings module):
  File "C:\Python27\lib\site-packages\gensim\models\wrappers\ldamallet.py", line 239
    topic = topic / topic.sum()  # normalize to probability dist
RuntimeWarning: invalid value encountered in divide
... LDA Model Trained ....
[(nan, u'b1e'), (nan, u'mokak'), (nan, u'viyac'), (nan, u'stipu'), (nan, u'forchange'), (nan, u'dugayiynggu'), (nan, u'hayne'), (nan, u'enrichment'), (nan, u'boseley'), (nan, u'releasesfresh')]
[(nan, u'b1e'), (nan, u'mokak'), (nan, u'viyac'), (nan, u'stipu'), (nan, u'forchange'), (nan, u'dugayiynggu'), (nan, u'hayne'), (nan, u'enrichment'), (nan, u'boseley'), (nan, u'releasesfresh')]
[(nan, u'b1e'), (nan, u'mokak'), (nan, u'viyac'), (nan, u'stipu'), (nan, u'forchange'), (nan, u'dugayiynggu'), (nan, u'hayne'), (nan, u'enrichment'), (nan, u'boseley'), (nan, u'releasesfresh')]
......
`

It is also worth mentioning that the state.mallet.gz output file is almost empty it contains only:

`#doc source pos typeindex type topic
# alpha : 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
# beta : 0.01

`

While the corpus.txt looks good 

`0 0 rest participate apply education attend motivate note application charge interest must local interstate natsiec complete form regional free train half day day day volunteer volunteer volunteer volunteer volunteer volunteer volunteer volunteer look receive
1 0 education education education education train train train train train deputy relate national executive executive number general vocational robinson robinson robinson robinson service employment employment numerou responsible unite research field role role department department department department department include chri chri kingdom prime commissioner commissioner manage centre government government cabinet director director five minister child hold hold hold appoint asqa chief chief chief period officer position senior senior
   .........`

but I am not sure about doctopics.txt output file 

`#doc name topic proportion ...
0   0   5   0.11538461538461539 3   0.11538461538461539 9   0.09615384615384616 8   0.09615384615384616 7   0.09615384615384616 6   0.09615384615384616 4   0.09615384615384616 2   0.09615384615384616 1   0.09615384615384616 0   0.09615384615384616 
`

I thought the document name should be the complete name not ID? Any idea where I went wrong?

Thanks



Discussion:
Which version of gensim are you using?

It is better if you use named parameters in a function call.

The Mallet parameters have changed.

https://github.com/piskvorky/gensim/blob/3d7293f3831250a47cf8e85a7f493a87c7d6ac32/gensim/models/wrappers/ldamallet.py#L54

On Mon, Apr 18, 2016 at 10:25 AM, aaajjj333 notifications@github.com
wrote:

> I run the LDAmallet to generate topics. The results show problem in the
> implementation.
> 
> `# READ DOCUMENTS
> 
> file_paths = [os.path.join(dir_data, fname) for fname in os.listdir(dir_data) if fname.endswith(".txt") ]
> documents = [codecs.open(file_path, 'r', encoding="utf8", errors='ignore').read() for file_path in file_paths ]
> 
> # USING NLTK TOKENIZER TO SEPERATE WORDS
> 
> tokenizer = RegexpTokenizer(r'\w+')
> 
> texts = []
> for each in documents:
>     tokens = tokenizer.tokenize(each)
>     texts.append(tokens)
> 
> dictionary = corpora.Dictionary(texts)
> 
> corpus = [dictionary.doc2bow(text) for text in texts]
> 
> num_topics=numberOfTopics
> mallet_path = "C:/mallet-2.0.7/bin/mallet"
> ldamodel = gensim.models.wrappers.LdaMallet(mallet_path, corpus, num_topics, dictionary, 4, dir_data, 0 ,1000)
> 
> result = (ldamodel.show_topics(num_topics, num_words,formatted=False))
> for each in result:
>     print each
> 
> `
> However, the output shows the same words repeated for all the topics and a
> warning for division by 0 when tries to normalize the weights. The output
> looks like this
> 
> Warning (from warnings module):
> File "C:\Python27\lib\site-packages\gensim\models\wrappers\ldamallet.py",
> line 239
> topic = topic / topic.sum() # normalize to probability dist
> RuntimeWarning: invalid value encountered in divide
> ... LDA Model Trained ....
> [(nan, u'b1e'), (nan, u'mokak'), (nan, u'viyac'), (nan, u'stipu'), (nan,
> u'forchange'), (nan, u'dugayiynggu'), (nan, u'hayne'), (nan,
> u'enrichment'), (nan, u'boseley'), (nan, u'releasesfresh')]
> [(nan, u'b1e'), (nan, u'mokak'), (nan, u'viyac'), (nan, u'stipu'), (nan,
> u'forchange'), (nan, u'dugayiynggu'), (nan, u'hayne'), (nan,
> u'enrichment'), (nan, u'boseley'), (nan, u'releasesfresh')]
> [(nan, u'b1e'), (nan, u'mokak'), (nan, u'viyac'), (nan, u'stipu'), (nan,
> u'forchange'), (nan, u'dugayiynggu'), (nan, u'hayne'), (nan,
> u'enrichment'), (nan, u'boseley'), (nan, u'releasesfresh')]
> ......
> 
> It is also worth mentioning that the state.mallet.gz output file is almost
> empty it contains only:
> 
> #doc source pos typeindex type topic
> #alpha : 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
> #beta : 0.01
> 
> While the corpus.txt looks good
> 
> 0 0 rest participate apply education attend motivate note application
> charge interest must local interstate natsiec complete form regional free
> train half day day day volunteer volunteer volunteer volunteer volunteer
> volunteer volunteer volunteer look receive
> 1 0 education education education education train train train train train
> deputy relate national executive executive number general vocational
> robinson robinson robinson robinson service employment employment numerou
> responsible unite research field role role department department department
> department department include chri chri kingdom prime commissioner
> commissioner manage centre government government cabinet director director
> five minister child hold hold hold appoint asqa chief chief chief period
> officer position senior senior
> .........
> 
> but I am not sure about doctopics.txt output file
> 
> #doc name topic proportion ...
> 0 0 5 0.11538461538461539 3 0.11538461538461539 9 0.09615384615384616 8
> 0.09615384615384616 7 0.09615384615384616 6 0.09615384615384616 4
> 0.09615384615384616 2 0.09615384615384616 1 0.09615384615384616 0
> 0.09615384615384616
> 
> I thought the document name should be the complete name not ID? Any idea
> where I went wrong?
> 
> Thanks
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly or view it on GitHub
> https://github.com/piskvorky/gensim/issues/671

I upgraded gensim, scikit-learn, scipy, and numpy

I changed the LdaMallet call to use named parameters and I still get the same results.

`ldamodel = gensim.models.wrappers.LdaMallet(mallet_path, corpus = mycorpus, num_topics = number_topics, id2word=dictionary, workers = 4, prefix = dir_data, optimize_interval = 0 , iterations= 1000)`

Hi everyone,

I had the same problem while trying to do the Mallet tutorial on the Reuters dataset from [here](http://rare-technologies.com/tutorial-on-mallet-in-python/). Be sure to update the module path of LdaMallet to `model = models.wrappers.LdaMallet` if you want to run this example. It seems to have changed since then. 

In `C:\python34\Lib\site-packages\gensim\models\wrappers\ldamallet.py` editing the function `convert_input`  helped for me. Change
`cmd = self.mallet_path + " import-file --preserve-case --keep-sequence --remove-stopwords --token-regex '\S+' --input %s --output %s"` 
to 
`cmd = self.mallet_path + ' import-file --preserve-case --keep-sequence --remove-stopwords --token-regex "\S+" --input %s --output %s'`. 
This only seems to be a problem on Windows and not on Linux, since single and double quoutes work differently in the respective command lines according to [this answer](http://stackoverflow.com/questions/26605449/infinity-value-error-after-token-regex-command) on SO. The original command causes the file state.mallet.gz to contain only two header lines and the `self.wordtopics` array to be empty. Summing over the empty array in the function `show_topic` yields Zero, resulting in 
`Warning (from warnings module):
  File "C:\Python34\lib\site-packages\gensim\models\wrappers\ldamallet.py", line 220
    topic = topic / topic.sum()  # normalize to probability dist
RuntimeWarning: invalid value encountered in true_divide`.
Hope this helps. Perhaps this is also related to issue #562.

My setup:
- Windows 10 Home
- Python 3.4.3
- Gensim 0.12.4
- Mallet 2.0.8RC3

Oh wow, nice catch @Wolfi-101 ! I didn't know about this [single vs. double quotes issues](http://stackoverflow.com/questions/24173825/what-does-single-quote-do-in-windows-batch-files) in Windows.

@tmylk we should do a review of all process calls in gensim wrappers, to make sure the quotes are correct for Windows too.


Resolving patch:
diff --git a/gensim/models/wrappers/ldamallet.py b/gensim/models/wrappers/ldamallet.py
--- a/gensim/models/wrappers/ldamallet.py
+++ b/gensim/models/wrappers/ldamallet.py
@@ -156,9 +156,9 @@ def convert_input(self, corpus, infer=False, serialize_corpus=True):
 
     def train(self, corpus):
         self.convert_input(corpus, infer=False)
-        cmd = self.mallet_path + " train-topics --input %s --num-topics %s  --alpha %s --optimize-interval %s "\
-            "--num-threads %s --output-state %s --output-doc-topics %s --output-topic-keys %s "\
-            "--num-iterations %s --inferencer-filename %s --doc-topics-threshold %s"
+        cmd = self.mallet_path + ' train-topics --input %s --num-topics %s  --alpha %s --optimize-interval %s '\
+            '--num-threads %s --output-state %s --output-doc-topics %s --output-topic-keys %s '\
+            '--num-iterations %s --inferencer-filename %s --doc-topics-threshold %s'
         cmd = cmd % (
             self.fcorpusmallet(), self.num_topics, self.alpha, self.optimize_interval, self.workers,
             self.fstate(), self.fdoctopics(), self.ftopickeys(), self.iterations, self.finferencer(), self.topic_threshold)
@@ -177,7 +177,7 @@ def __getitem__(self, bow, iterations=100):
             bow = [bow]
 
         self.convert_input(bow, infer=True)
-        cmd = self.mallet_path + " infer-topics --input %s --inferencer %s --output-doc-topics %s --num-iterations %s --doc-topics-threshold %s"
+        cmd = self.mallet_path + ' infer-topics --input %s --inferencer %s --output-doc-topics %s --num-iterations %s --doc-topics-threshold %s'
         cmd = cmd % (self.fcorpusmallet() + '.infer', self.finferencer(), self.fdoctopics() + '.infer', iterations, self.topic_threshold)
         logger.info("inferring topics with MALLET LDA '%s'", cmd)
         check_output(args=cmd, shell=True)
diff --git a/gensim/models/wrappers/wordrank.py b/gensim/models/wrappers/wordrank.py
--- a/gensim/models/wrappers/wordrank.py
+++ b/gensim/models/wrappers/wordrank.py
@@ -135,7 +135,7 @@ def train(cls, wr_path, corpus_file, out_path, size=100, window=15, symmetric=1,
         # run wordrank executable with wr_args
         cmd = ['mpirun', '-np', '1', '../wordrank']
         for option, value in wr_args.items():
-            cmd.append("--%s" % option)
+            cmd.append('--%s' % option)
             cmd.append(str(value))
         logger.info("Running wordrank binary '%s'", cmd)
         output = utils.check_output(args=cmd)


Test patch: