Repository: piskvorky/gensim

Issue:
object of type 'zip' has no len()
There seams to be an issue with either `Sparse2Corpus` or `from_corpus`. Here is a simple example:

``` python
import scipy
import gensim

bow = gensim.matutils.Sparse2Corpus(scipy.sparse.rand(10, 100))
dictionary = gensim.corpora.Dictionary.from_corpus(bow)
```

The errror is:

```
Traceback (most recent call last):
  File "experiment.py", line 5, in <module>
    dictionary = gensim.corpora.Dictionary.from_corpus(bow)
  File "/usr/local/lib/python3.4/site-packages/gensim/corpora/dictionary.py", line 356, in from_corpus
    result.num_nnz += len(document)
TypeError: object of type 'zip' has no len()
```

I'm on OS X 10.9.3 using python 3.4.1



Discussion:
Aha. Good catch. This report affects python 3 only, because in python 2, zip produces a list.

I think the safest way is to to return `list(zip(....))` in `Sparse2Corpus.__iter__`, rather than just `zip(...)`.

Can you submit a PR @AndreasMadsen ?

I think it might be better to do a `try: catch:` in the `Dictionary.from_corpus` (and maybe other places in the class) for counting the documents. Probably bad to assume iterables will all have a `len()`. 

There is a loop there that iterates over the corpus there which could also count the length of the document just as well if the `len()` doesn't exist. We do counting for num_docs but not num_nnz: https://github.com/piskvorky/gensim/blob/develop/gensim/corpora/dictionary.py#L355-357

That's also an option, yes @cscorley .

For documents, we can safely assume they fit in memory (so expecting basic structures like `list` is ok). It's just corpora (many documents) where we care about streaming, because they can be large.

@AndreasMadsen is a `Sparse2Corpus.__iter__` PR forthcoming? It's pretty straightforward -- let me know if you can fix it, so we can close this issue.

I'm unsure about what the preferred way of fixing this is. Is it to simply return `list(zip(...))`, since python 2 did that anyway?

Yep :)


Resolving patch:
diff --git a/gensim/matutils.py b/gensim/matutils.py
--- a/gensim/matutils.py
+++ b/gensim/matutils.py
@@ -307,7 +307,7 @@ def __init__(self, sparse, documents_columns=True):
 
     def __iter__(self):
         for indprev, indnow in izip(self.sparse.indptr, self.sparse.indptr[1:]):
-            yield zip(self.sparse.indices[indprev:indnow], self.sparse.data[indprev:indnow])
+            yield list(zip(self.sparse.indices[indprev:indnow], self.sparse.data[indprev:indnow]))
 
     def __len__(self):
         return self.sparse.shape[1]


Test patch:
diff --git a/gensim/test/test_corpora_dictionary.py b/gensim/test/test_corpora_dictionary.py
--- a/gensim/test/test_corpora_dictionary.py
+++ b/gensim/test/test_corpora_dictionary.py
@@ -15,6 +15,8 @@
 import os
 import os.path
 
+import scipy
+import gensim
 from gensim.corpora import Dictionary
 from six import PY3
 from six.moves import zip
@@ -164,6 +166,11 @@ def test_from_corpus(self):
         self.assertEqual(dictionary.num_pos, dictionary_from_corpus.num_pos)
         self.assertEqual(dictionary.num_nnz, dictionary_from_corpus.num_nnz)
 
+        # Ensure Sparse2Corpus is compatiable with from_corpus
+        bow = gensim.matutils.Sparse2Corpus(scipy.sparse.rand(10, 100))
+        dictionary = Dictionary.from_corpus(bow)
+        self.assertEqual(dictionary.num_docs, 100)
+
     def test_dict_interface(self):
         """Test Python 2 dict-like interface in both Python 2 and 3."""
         d = Dictionary(self.texts)