Repository: piskvorky/gensim
Pull request: Correctly process empty documents in `AuthorTopicModel`
This is a fix #1589 

initialized empty numpy arrays defualt to `dtype=np.float` making them ineligible for use as index arrays (which must be of `dtype=np.integer` or `dtype=np.bool`)

Problem statement:
Problem using bound function in Author Topic model!!
Hi
I am trying to use author topic model. When I used bound function to evaluate my model I received this error in python:
```
Traceback (most recent call last):
  File "C:\Users\Sara\Desktop\E-COM\Final Project\Working on data\at_clustering.py", line 161, in <module>
    test_corpus = test_corpus, test_d2a = test_doc2author, test_a2d = test_author2doc, limit = 25)
  File "C:\Users\Sara\Desktop\E-COM\Final Project\Working on data\at_clustering.py", line 70, in evaluate_k
    pr = np.exp2(-model.bound(test_corpus, doc2author=test_d2a, author2doc=test_a2d)/number_of_words)
  File "C:\Python27\lib\site-packages\gensim\models\atmodel.py", line 835, in bound
    phinorm = self.compute_phinorm(ids, authors_d, expElogtheta[authors_d, :], expElogbeta[:, ids])
IndexError: arrays used as indices must be of integer (or boolean) type
```
 I think the problem is `` expElogbeta[:, ids] `` because this is not acceptable in python!

Please test bound function and fix source code.
Thanks

Hints from issue discussion:
Hi @sa-matiny,
`expElogbeta[:, ids]` is correct for numpy arrays.
Please add more information
- OS/Python/Gensim/Numpy/Scipy versions
- Code example (for reproducing exception)
Ping @sa-matiny 
Hi,
My system info is windows10/python 2.7.6/numpy 1.13.1/scipy 0.19.1
I just set the eval_every parameter in author topic model True:
```
train_corpus = corpora.MmCorpus("tweet_preprocessing/train_corpus.mm")
train_dictionary = Dictionary.load("tweet_preprocessing/train_dic.dict")
train_tweets = pd.read_json("tweet_preprocessing/train_tweets.txt")
train_doc2author = dict(zip(train_tweets.index, train_tweets.user_ids))
n_topics = 10
model = AuthorTopicModel(corpus=train_corpus, num_topics=n_topics, id2word=train_dictionary,
                                doc2author=d2a, chunksize=2000, eval_every=True, iterations=10 , serialized=True,
                                serialization_path="clustering_temp2/train_corpus.mm")
```
At the time that bound() function is called, I got this error:
```
Traceback (most recent call last):
  File "C:\Users\Sara\Desktop\E-COM\Final Project\Working on data\clustering.py", line 77, in <module>
    n_topics = num_top, d2a = train_doc2author, s_path = "clustering_temp2/train_corpus.mm")
  File "C:\Python27\lib\site-packages\gensim\models\atmodel.py", line 288, in __init__
    self.update(corpus, author2doc, doc2author, chunks_as_numpy=use_numpy)
  File "C:\Python27\lib\site-packages\gensim\models\atmodel.py", line 711, in update
    self.log_perplexity(chunk, chunk_doc_idx, total_docs=lencorpus)
  File "C:\Python27\lib\site-packages\gensim\models\atmodel.py", line 493, in log_perplexity
    perwordbound = self.bound(chunk, chunk_doc_idx, subsample_ratio=subsample_ratio) / (subsample_ratio * corpus_words)
  File "C:\Python27\lib\site-packages\gensim\models\atmodel.py", line 835, in bound
    phinorm = self.compute_phinorm(ids, authors_d, expElogtheta[authors_d, :], expElogbeta[:, ids])
IndexError: arrays used as indices must be of integer (or boolean) type
```
@sa-matiny  OK, let's add more info:
- Load your datasets to any storage and share link with me
- Don't forget to define `d2a` variable (now this is undefined)


This needed for me for reproducing your error.
This is the sample of small data and also code that got error:
https://drive.google.com/file/d/0Bz1OoooH0zo4Y0VqY0pDaFZpRDg/view?usp=sharing
I experienced the same error and found that the errors were cause by empty document (because all the words were filtered out).
Thanks for the information @Ronggui 
After running into this problem as well, i have determined a fix for this. I'm currently reading through the contributions guide to submit a conformant pull request.

This error is being thrown because `numpy.array([])` defaults to `dtype=np.float` instead of `dtype=np.integer`. The contents of the array are used as type hints, but an empty array doesn't provide this context.

Commit message: ignore unused assigned varaible

Patch to review:
diff --git a/gensim/models/atmodel.py b/gensim/models/atmodel.py
index daa7ea4ab2..28d2ddf6c7 100755
--- a/gensim/models/atmodel.py
+++ b/gensim/models/atmodel.py
@@ -460,11 +460,12 @@ def inference(self, chunk, author2doc, doc2author, rhot, collect_sstats=False, c
                 # make sure the term IDs are ints, otherwise np will get upset
                 ids = [int(idx) for idx, _ in doc]
             else:
-                ids = [idx for idx, _ in doc]
-            cts = np.array([cnt for _, cnt in doc])
+                ids = [id for id, _ in doc]
+            ids = np.array(ids, dtype=np.integer)
+            cts = np.array([cnt for _, cnt in doc], dtype=np.integer)
 
             # Get all authors in current document, and convert the author names to integer IDs.
-            authors_d = [self.author2id[a] for a in self.doc2author[doc_no]]
+            authors_d = np.array([self.author2id[a] for a in self.doc2author[doc_no]], dtype=np.integer)
 
             gammad = self.state.gamma[authors_d, :]  # gamma of document d before update.
             tilde_gamma = gammad.copy()  # gamma that will be updated.
@@ -972,9 +973,9 @@ def bound(self, chunk, chunk_doc_idx=None, subsample_ratio=1.0, author2doc=None,
             else:
                 doc_no = d
             # Get all authors in current document, and convert the author names to integer IDs.
-            authors_d = [self.author2id[a] for a in self.doc2author[doc_no]]
-            ids = np.array([id for id, _ in doc])  # Word IDs in doc.
-            cts = np.array([cnt for _, cnt in doc])  # Word counts.
+            authors_d = np.array([self.author2id[a] for a in self.doc2author[doc_no]], dtype=np.integer)
+            ids = np.array([id for id, _ in doc], dtype=np.integer)  # Word IDs in doc.
+            cts = np.array([cnt for _, cnt in doc], dtype=np.integer)  # Word counts.
 
             if d % self.chunksize == 0:
                 logger.debug("bound: at document #%i in chunk", d)
diff --git a/gensim/test/test_atmodel.py b/gensim/test/test_atmodel.py
index 00d4d2aafa..43ff670bd0 100644
--- a/gensim/test/test_atmodel.py
+++ b/gensim/test/test_atmodel.py
@@ -35,7 +35,6 @@
 # increases the bound.
 # Test that models are compatiple across versions, as done in LdaModel.
 
-
 # Assign some authors randomly to the documents above.
 author2doc = {
     'john': [0, 1, 2, 3, 4, 5, 6],
@@ -110,6 +109,19 @@ def testBasic(self):
         jill_topics = matutils.sparse2full(jill_topics, model.num_topics)
         self.assertTrue(all(jill_topics > 0))
 
+    def testEmptyDocument(self):
+        _local_texts = common_texts + [['only_occurs_once_in_corpus_and_alone_in_doc']]
+        _dictionary = Dictionary(_local_texts)
+        _dictionary.filter_extremes(no_below=2)
+        _corpus = [_dictionary.doc2bow(text) for text in _local_texts]
+        _a2d = author2doc.copy()
+        _a2d['joaquin'] = [len(_local_texts) - 1]
+        try:
+            _ = self.class_(_corpus, author2doc=_a2d, id2word=_dictionary, num_topics=2)
+        except IndexError:
+            raise IndexError("error occurs in 1.0.0 release tag")
+        assert(_)
+
     def testAuthor2docMissing(self):
         # Check that the results are the same if author2doc is constructed automatically from doc2author.
         model = self.class_(


Code review comments:
diff_hunk: @@ -110,6 +109,19 @@ def testBasic(self):
         jill_topics = matutils.sparse2full(jill_topics, model.num_topics)
         self.assertTrue(all(jill_topics > 0))
 
+    def testEmptyDocument(self):
+        _local_texts = common_texts + [['only_occurs_once_in_corpus_and_alone_in_doc']]
path: gensim/test/test_atmodel.py
text: @user1:
why vars starts from underscore? please remove underscores from start

diff_hunk: @@ -110,6 +109,19 @@ def testBasic(self):
         jill_topics = matutils.sparse2full(jill_topics, model.num_topics)
         self.assertTrue(all(jill_topics > 0))
 
+    def testEmptyDocument(self):
+        _local_texts = common_texts + [['only_occurs_once_in_corpus_and_alone_in_doc']]
+        _dictionary = Dictionary(_local_texts)
+        _dictionary.filter_extremes(no_below=2)
+        _corpus = [_dictionary.doc2bow(text) for text in _local_texts]
+        _a2d = author2doc.copy()
+        _a2d['joaquin'] = [len(_local_texts) - 1]
+        try:
path: gensim/test/test_atmodel.py
text: @user1:
No need try/except section in test, if test raise unexpected exception - this means that test broken

diff_hunk: @@ -460,11 +460,12 @@ def inference(self, chunk, author2doc, doc2author, rhot, collect_sstats=False, c
                 # make sure the term IDs are ints, otherwise np will get upset
                 ids = [int(idx) for idx, _ in doc]
             else:
-                ids = [idx for idx, _ in doc]
-            cts = np.array([cnt for _, cnt in doc])
+                ids = [id for id, _ in doc]
path: gensim/models/atmodel.py
text: @user1:
Please revert back `idx` (`id` is built-in function name)