Repository: piskvorky/gensim
Pull request: Fix out of range issue in gensim.summarization.keywords
Fix #2598 

The issue arises when the parameter `words`, which is defined as _The maximum amount of words to return_, is greater than the number of `lemmas` or keywords found. This caused an index out of range error from line 306 because it wasn't handling the case where `words > len(lemmas)`.

**Before the fix:** 
`length = len(lemmas) * ratio if words is None else words`
`return [(scores[lemmas[i]], lemmas[i],) for i in range(int(length))]`
As one can see, if `words > len(lemmas)`, the return statement will throw an index out of range error because the for loop will continue past the length of the lemmas array.

**Example of reproducing issue:**
`lemmas = ["test", "issue", "reproduction"]`
`words = 5`

Before Fix:
Index out of range in line 306 because length = 5 and len(lemmas) = 3.

After Fix:
No issue because length = len(lemmas) = 3.

Problem statement:
keywords.py gives `IndexError: list index out of range` when `words` parameter is provided.
Really confused why I'm getting this error. Perhaps I'm making a silly mistake I'm not familiar with gensim and nlp in general.
Im running on Windows 10 Home 64-bit, conda version : 4.7.11, conda-build version : 2.18.8, python version : 3.7.3.final.0
My code is attempting to get keywords per sentence in a loop. To simplify matters I've isolated the following code that causes this, trying to get keywords from gensim's `keywords.py`.

```python
s = "Don’t dive right into solving without a plan (and somehow hope you can muddle your way through)."
keywords(s, words=4, scores=False, split=True, lemmatize=True)

File "C:\Users\username\Anaconda3\envs\gensim\lib\site-packages\gensim\summarization\keywords.py", line 521, in keywords
    extracted_lemmas = _extract_tokens(graph.nodes(), pagerank_scores, ratio, words)
  File "C:\Users\username\Anaconda3\envs\gensim\lib\site-packages\gensim\summarization\keywords.py", line 304, in _extract_tokens
    return [(scores[lemmas[i]], lemmas[i],) for i in range(int(length))]
  File "C:\Users\username\Anaconda3\envs\gensim\lib\site-packages\gensim\summarization\keywords.py", line 304, in <listcomp>
    return [(scores[lemmas[i]], lemmas[i],) for i in range(int(length))]
IndexError: list index out of range
```

I've tried setting `scores=True`, `lemmatize=False`, and `split=False` but the same error persists. I've also tried removing the parenthesis and removing the apostrophe, the error persisted. What did work is removing the `words` parameter altogether, but still it shouldn't create an error if it's provided. Thanks for the help in advance!

Hints from issue discussion:
This could well be a bug.

Are you able to step through the problem with a debugger?
Yes going through with the debugger I can see why the following lines (in `keywords._extract_tokens()` ) are giving an error. `words==4` while `len(lemmas)==2` and so there will be in index out of range error. 
```python
length = len(lemmas) * ratio if words is None else words
return [(scores[lemmas[i]], lemmas[i],) for i in range(int(length))]
```
I'm unsure what would be the best way to handle this, perhaps just cap the maximum value of the final range if `words > len(lemmas)`
@arielFrischer Can you go through the source and determine what are the invariants are of these variables? `words`, `lemmas`, `lengths` etc? What are they for? Why do you need `words`?

Unfortunately none of are familiar with this code. 
I looked at the code; IMO there is an implicit (unchecked) invariant that `words <= len(lemmas)`.

If `words` is provided and greater than `len(lemmas)`, such as in your example where `lemmas=[u'dive', u'right']`, the code will fail.

@mpenkov can we somehow track the original contributor and check with them what this is about?

Because I'm not sure if a simple `words = min(int(words), len(lemmas))` is enough, or if it's a deeper problem.
Hi, can I have a go at this?
Sure. I think the main part of this ticket will be investigating what the original contributor was trying to achieve, as @piskvorky pointed out.

Commit message: Fixed out of range error in keywords.py

Patch to review:
diff --git a/gensim/summarization/keywords.py b/gensim/summarization/keywords.py
index db7c8a0dc7..584b755b65 100644
--- a/gensim/summarization/keywords.py
+++ b/gensim/summarization/keywords.py
@@ -302,7 +302,7 @@ def _extract_tokens(lemmas, scores, ratio, words):
 
     """
     lemmas.sort(key=lambda s: scores[s], reverse=True)
-    length = len(lemmas) * ratio if words is None else words
+    length = len(lemmas) * ratio if words is None else words if words <= len(lemmas) else len(lemmas)
     return [(scores[lemmas[i]], lemmas[i],) for i in range(int(length))]
 
 


Code review comments:
diff_hunk: @@ -302,7 +302,7 @@ def _extract_tokens(lemmas, scores, ratio, words):
 
     """
     lemmas.sort(key=lambda s: scores[s], reverse=True)
-    length = len(lemmas) * ratio if words is None else words
+    length = len(lemmas) * ratio if words is None else words if words <= len(lemmas) else len(lemmas)
path: gensim/summarization/keywords.py
text: @user1:
```suggestion
    length = len(lemmas) * ratio if words is None else min(words, len(lemmas))
```

More readable.

@author:
Applied change in latest commit