Repository: piskvorky/gensim

Issue:
Error while summarizing text
Hi,

I've received the following error when trying to summarize the body of this news article:

https://www.theguardian.com/media/2016/jun/19/sun-times-brexit-in-out-shake-it-all-about

The error follows:

  File "/home/apps/comment_parser/venv/local/lib/python2.7/site-packages/gensim/summarization/summarizer.py", line 202, in summarize
    most_important_docs = summarize_corpus(corpus, ratio=ratio if word_count is None else 1)
  File "/home/apps/comment_parser/venv/local/lib/python2.7/site-packages/gensim/summarization/summarizer.py", line 161, in summarize_corpus
    pagerank_scores = _pagerank(graph)
  File "/home/apps/comment_parser/venv/local/lib/python2.7/site-packages/gensim/summarization/pagerank_weighted.py", line 24, in pagerank_weighted
    vals, vecs = eigs(pagerank_matrix.T, k=1)  # TODO raise an error if matrix has complex eigenvectors?
  File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/eigen/arpack/arpack.py", line 1271, in eigs
    ncv, v0, maxiter, which, tol)
  File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/eigen/arpack/arpack.py", line 685, in __init__
    raise ValueError("k must be less than ndim(A)-1, k=%d" % k)
ValueError: k must be less than ndim(A)-1, k=1

Regards,



Discussion:
I am also affected by this bug...

@madewild could you post a code snippet to reproduce?

Unfortunately the text fed to the summarizer is confidential, but my guess is that the error was triggered by an unusually high repetition of some sentences... I also notice now that the error raised was not exactly the same:
ValueError: k must be less than **rank**(A)-1, k=1

@tmylk The reason for this failure looks like the number of nodes in the graph that is used to calculate the pagerank of the corpus graph, after removing unreachable nodes the graph is left with only 2 nodes and hence it builds a matrix of shape 2 \* 2 (for which scipy.sparse.linalg.eigs() will fail for k=1). We should probably raise an error if number of nodes (after removing unreachable nodes) goes below 3.

@MridulS could you submit a pr for this?

@tmylk What kind of error should I raise?

The error should say "Please add more sentences to the text. The number of reachable nodes is below 3"

Hi I worked on this issue. I have sent out a pull request for the same. Please review.

Hello, I think that the problem is still open. I replicated this error with the document 1403 from the [Hulth2003 dataset](https://github.com/snkim/AutomaticKeyphraseExtraction)):
`from gensim.summarization import keywords`
`print(keywords('IT: Utilities A look at five utilities to make your PCs more, efficient, effective, and efficacious'))`

Traceback (most recent call last):
  File "\<stdin\>", line 1, in <module>
  File "/gensim/summarization/keywords.py", line 214, in keywords
    pagerank_scores = _pagerank(graph)
  File "/gensim/summarization/pagerank_weighted.py", line 24, in pagerank_weighted
    vals, vecs = eigs(pagerank_matrix.T, k=1)  # TODO raise an error if matrix has complex eigenvectors?
  File "/anaconda3/lib/python3.6/site-packages/scipy/sparse/linalg/eigen/arpack/arpack.py", line 1299, in eigs
    ncv, v0, maxiter, which, tol)
  File "/anaconda3/lib/python3.6/site-packages/scipy/sparse/linalg/eigen/arpack/arpack.py", line 692, in __init__
    raise ValueError("k must be less than ndim(A)-1, k=%d" % k)
ValueError: k must be less than ndim(A)-1, k=1

Looking to the document, make sense say that the possible problems are the terms frequencies! All the terms have frequency equal 1.
Hmm, that's not good, looks like a bug.

Can you suggest a fix @vitordouzi ?
@piskvorky, no, I don't! sorry! Maybe this TODO in the pagerank_weighted.py file can help. 

File "/gensim/summarization/pagerank_weighted.py", line 24, in pagerank_weighted
vals, vecs = eigs(pagerank_matrix.T, k=1) # TODO raise an error if matrix has complex eigenvectors?

What exactly are the complex eigenvectors?
Hello everyone. I started investigating this issue and basically, this is the same one as @MridulS described, but in different function:

> The reason for this failure looks like the number of nodes in the graph that is used to calculate the pagerank of the corpus graph, after removing unreachable nodes the graph is left with only 2 nodes and hence it builds a matrix of shape 2 * 2 (for which scipy.sparse.linalg.eigs() will fail for k=1). We should probably raise an error if number of nodes (after removing unreachable nodes) goes below 3.

On text, given by @vitordouzi we end up with graph:
```
('effect', 'effici'), ('effici', 'effect'), ('effici', 'effici'), ('effect', 'effect')
```
which ends in 2x2 matrix and pagerank fails.

But I'm not sure how to fix this. @vitordouzi, @menshikh-iv any ideas on the desired outcome? An exception this time doesn't feel right. Maybe set some predefined scores instead of running pagerank? Or maybe add special case to pagerank?
Anyway, some notes about `pagerank_weighted`:

https://github.com/RaRe-Technologies/gensim/blob/9481915915bf61aa6e4e719a2f26d509677e6779/gensim/summarization/pagerank_weighted.py#L18-L26

1. Matrix can have complex eigenvalues and eigenvector, but we a finding the largest eigenvalue, which is proven to be real. (because `pagerank_matrix` is positive and [Perron–Frobenius theorem](https://en.wikipedia.org/wiki/Perron%E2%80%93Frobenius_theorem))
2. Either assert about less than 3 vertices should be added, or, maybe, special handling of this case.
3. It took me awhile to figure out why `eigs()` is used instead of `eig()` on a dense matrix. It was discussed in #441, #438. I'd like to add a comment about this.

About (1), (2) @xelez - need to handle special case, the comment from (3) should be useful too.
 

Resolving patch:
diff --git a/gensim/summarization/pagerank_weighted.py b/gensim/summarization/pagerank_weighted.py
--- a/gensim/summarization/pagerank_weighted.py
+++ b/gensim/summarization/pagerank_weighted.py
@@ -2,7 +2,9 @@
 # -*- coding: utf-8 -*-
 #
 # Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html
+import numpy
 from numpy import empty as empty_matrix
+from scipy.linalg import eig
 from scipy.sparse import csr_matrix
 from scipy.sparse.linalg import eigs
 from six.moves import xrange
@@ -21,9 +23,10 @@ def pagerank_weighted(graph, damping=0.85):
 
     pagerank_matrix = damping * adjacency_matrix.todense() + (1 - damping) * probability_matrix
 
-    vals, vecs = eigs(pagerank_matrix.T, k=1)  # TODO raise an error if matrix has complex eigenvectors?
+    vec = principal_eigenvector(pagerank_matrix.T)
 
-    return process_results(graph, vecs.real)
+    # Because pagerank_matrix is positive, vec is always real (i.e. not complex)
+    return process_results(graph, vec.real)
 
 
 def build_adjacency_matrix(graph):
@@ -56,9 +59,23 @@ def build_probability_matrix(graph):
     return matrix
 
 
-def process_results(graph, vecs):
+def principal_eigenvector(a):
+    # Note that we prefer to use `eigs` even for dense matrix
+    # because we need only one eigenvector. See #441, #438 for discussion.
+
+    # But it doesn't work for dim A < 3, so we just handle this special case
+    if len(a) < 3:
+        vals, vecs = eig(a)
+        ind = numpy.abs(vals).argmax()
+        return vecs[:, ind]
+    else:
+        vals, vecs = eigs(a, k=1)
+        return vecs[:, 0]
+
+
+def process_results(graph, vec):
     scores = {}
     for i, node in enumerate(graph.nodes()):
-        scores[node] = abs(vecs[i, :])
+        scores[node] = abs(vec[i])
 
     return scores


Test patch:
diff --git a/gensim/test/test_keywords.py b/gensim/test/test_keywords.py
--- a/gensim/test/test_keywords.py
+++ b/gensim/test/test_keywords.py
@@ -89,6 +89,12 @@ def test_keywords_ratio(self):
 
         self.assertAlmostEqual(float(len(selected_docs_21)) / len(selected_docs_12), float(21) / 12, places=1)
 
+    def test_text_keywords_with_small_graph(self):
+        # regression test, we get graph 2x2 on this text
+        text = 'IT: Utilities A look at five utilities to make your PCs more, efficient, effective, and efficacious'
+        kwds = keywords(text, words=1, split=True)
+        self.assertTrue(len(kwds))
+
 
 if __name__ == '__main__':
     logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.DEBUG)