Repository: piskvorky/gensim

Issue:
keywords.py gives `IndexError: list index out of range` when `words` parameter is provided.
Really confused why I'm getting this error. Perhaps I'm making a silly mistake I'm not familiar with gensim and nlp in general.
Im running on Windows 10 Home 64-bit, conda version : 4.7.11, conda-build version : 2.18.8, python version : 3.7.3.final.0
My code is attempting to get keywords per sentence in a loop. To simplify matters I've isolated the following code that causes this, trying to get keywords from gensim's `keywords.py`.

```python
s = "Don’t dive right into solving without a plan (and somehow hope you can muddle your way through)."
keywords(s, words=4, scores=False, split=True, lemmatize=True)

File "C:\Users\username\Anaconda3\envs\gensim\lib\site-packages\gensim\summarization\keywords.py", line 521, in keywords
    extracted_lemmas = _extract_tokens(graph.nodes(), pagerank_scores, ratio, words)
  File "C:\Users\username\Anaconda3\envs\gensim\lib\site-packages\gensim\summarization\keywords.py", line 304, in _extract_tokens
    return [(scores[lemmas[i]], lemmas[i],) for i in range(int(length))]
  File "C:\Users\username\Anaconda3\envs\gensim\lib\site-packages\gensim\summarization\keywords.py", line 304, in <listcomp>
    return [(scores[lemmas[i]], lemmas[i],) for i in range(int(length))]
IndexError: list index out of range
```

I've tried setting `scores=True`, `lemmatize=False`, and `split=False` but the same error persists. I've also tried removing the parenthesis and removing the apostrophe, the error persisted. What did work is removing the `words` parameter altogether, but still it shouldn't create an error if it's provided. Thanks for the help in advance!


Discussion:
This could well be a bug.

Are you able to step through the problem with a debugger?
Yes going through with the debugger I can see why the following lines (in `keywords._extract_tokens()` ) are giving an error. `words==4` while `len(lemmas)==2` and so there will be in index out of range error. 
```python
length = len(lemmas) * ratio if words is None else words
return [(scores[lemmas[i]], lemmas[i],) for i in range(int(length))]
```
I'm unsure what would be the best way to handle this, perhaps just cap the maximum value of the final range if `words > len(lemmas)`
@arielFrischer Can you go through the source and determine what are the invariants are of these variables? `words`, `lemmas`, `lengths` etc? What are they for? Why do you need `words`?

Unfortunately none of are familiar with this code. 
I looked at the code; IMO there is an implicit (unchecked) invariant that `words <= len(lemmas)`.

If `words` is provided and greater than `len(lemmas)`, such as in your example where `lemmas=[u'dive', u'right']`, the code will fail.

@mpenkov can we somehow track the original contributor and check with them what this is about?

Because I'm not sure if a simple `words = min(int(words), len(lemmas))` is enough, or if it's a deeper problem.
Hi, can I have a go at this?
Sure. I think the main part of this ticket will be investigating what the original contributor was trying to achieve, as @piskvorky pointed out.

Resolving patch:
diff --git a/gensim/summarization/keywords.py b/gensim/summarization/keywords.py
--- a/gensim/summarization/keywords.py
+++ b/gensim/summarization/keywords.py
@@ -302,7 +302,7 @@ def _extract_tokens(lemmas, scores, ratio, words):
 
     """
     lemmas.sort(key=lambda s: scores[s], reverse=True)
-    length = len(lemmas) * ratio if words is None else words
+    length = len(lemmas) * ratio if words is None else min(words, len(lemmas))
     return [(scores[lemmas[i]], lemmas[i],) for i in range(int(length))]
 
 


Test patch:
diff --git a/gensim/test/test_keywords.py b/gensim/test/test_keywords.py
--- a/gensim/test/test_keywords.py
+++ b/gensim/test/test_keywords.py
@@ -101,6 +101,12 @@ def test_text_keywords_without_graph_edges(self):
         kwds = keywords(text, deacc=False, scores=True)
         self.assertFalse(len(kwds))
 
+    def test_keywords_with_words_greater_than_lemmas(self):
+        # words parameter is greater than number of words in text variable
+        text = 'Test string small length'
+        kwds = keywords(text, words=5, split=True)
+        self.assertIsNotNone(kwds)
+
 
 if __name__ == '__main__':
     logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.DEBUG)