Repository: piskvorky/gensim

Issue:
KeyError in LsiModel constructor
#### Problem description

Hi, I'm trying to use an `LsiModel` to reduce a corpus.  When constructing the `LsiModel`, I sometimes see `KeyError`s raised by logging code (specifically, [here](https://github.com/RaRe-Technologies/gensim/blob/f8110c036f30e030e4c27101398e076cd1254819/gensim/models/lsimodel.py#L673) via [here](https://github.com/RaRe-Technologies/gensim/blob/f8110c036f30e030e4c27101398e076cd1254819/gensim/models/lsimodel.py#L522)).  If I manually remove that logging, I can generate the model without errors.

I found [a similar issue](https://github.com/RaRe-Technologies/gensim/issues/1995), where @menshikh-iv recommended "removing gaps" from my dictionary.  I definitely could do that, but it would require an extra pass over my input, which can be pretty large.  Instead, it would be trivial to do a more safe lookup into `id2word` than using `[val]`.  For example, this workaround fixes my problem:

```diff
diff --git i/gensim/models/lsimodel.py w/gensim/models/lsimodel.py
index 97cc921f..4b703dcf 100644
--- i/gensim/models/lsimodel.py
+++ w/gensim/models/lsimodel.py
@@ -670,7 +670,7 @@ class LsiModel(interfaces.TransformationABC, basemodel.BaseTopicModel):
         c = np.asarray(self.projection.u.T[topicno, :]).flatten()
         norm = np.sqrt(np.sum(np.dot(c, c)))
         most = matutils.argsort(np.abs(c), topn, reverse=True)
-        return [(self.id2word[val], 1.0 * c[val] / norm) for val in most]
+        return [(self.id2word.get(val, "<None>"), 1.0 * c[val] / norm) for val in most]
 
     def show_topics(self, num_topics=-1, num_words=10, log=False, formatted=True):
         """Get the most significant topics.
```

#### Steps/code/corpus to reproduce

```python
>>> from gensim.models import LsiModel
>>>
>>> tfidf = [[(0, 0.0), (1, 1.0), (2, 2.0)]]
>>> id2word = {0: "a", 1: "b", 2: "c", 100: "d"}  # note that there is a gap here!
>>> LsiModel(tfidf, id2word=id2word)
Traceback (most recent call last):
  File "gensim_test.py", line 5, in <module>
    LsiModel(tfidf, id2word=id2word)
  File "/path/to/gensim/models/lsimodel.py", line 445, in __init__
    self.add_documents(corpus)
  File "/path/to/gensim/models/lsimodel.py", line 518, in add_documents
    self.print_topics(5)
  File "/path/to/gensim/models/basemodel.py", line 36, in print_topics
    return self.show_topics(num_topics=num_topics, num_words=num_words, log=True)
  File "/path/to/gensim/models/lsimodel.py", line 699, in show_topics
    topic = self.print_topic(i, topn=num_words)
  File "/path/to/gensim/models/basemodel.py", line 18, in print_topic
    return ' + '.join('%.3f*"%s"' % (v, k) for k, v in self.show_topic(topicno, topn))
  File "/path/to/gensim/models/lsimodel.py", line 669, in show_topic
    return [(self.id2word[val], 1.0 * c[val] / norm) for val in most]
  File "/path/to/gensim/models/lsimodel.py", line 669, in <listcomp>
    return [(self.id2word[val], 1.0 * c[val] / norm) for val in most]
KeyError: 69
```

#### Versions

Please provide the output of:

```python
>>> import platform; print(platform.platform())
>>> import sys; print("Python", sys.version)
>>> import struct; print("Bits", 8 * struct.calcsize("P"))
>>> import numpy; print("NumPy", numpy.__version__)
>>> import scipy; print("SciPy", scipy.__version__)
>>> import gensim; print("gensim", gensim.__version__)
>>> from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
Linux-5.4.0-64-generic-x86_64-with-debian-buster-sid
Python 3.7.10 (default, Feb 26 2021, 18:47:35) 
[GCC 7.3.0]
Bits 64
NumPy 1.19.1
SciPy 1.5.3
gensim 3.8.3
FAST_VERSION 0
```



Discussion:
As far as I can see, this can only happen with super small dictionaries: `len(id2word) < topn`.

The default `topn=10`, so does your dictionary really have less than 10 terms in it? Is that expected?
> As far as I can see, this can only happen with super small dictionaries: `len(id2word) < topn`.
> 
> The default `topn=10`, so does your dictionary really have less than 10 terms in it? Is that expected?

I believe this can happen even for larger corpora if you see lots of repeats at the beginning.
For example, I get the same error for a corpus like this:
```python
from gensim.models import LsiModel

tfidf = [[(0, 1.0)] for i in range(10_000)]
tfidf.append([(100, 1.0)])

id2word = {0: "a", 100: "b"}
LsiModel(tfidf, id2word=id2word)
```
That dictionary is still smaller than 10, `len(id2word) == 2`.

But I think you may be right, this can happen whenever the trained model hasn't seen at least 10 terms when `show_topics()` is called. For example, when the first 20,000 documents (default `chunksize=20000`) contained less than 10 unique terms.

Still a pretty weird setup, but it can happen. We should detect when that happens, and output only the top `min(topn, actual_active_terms)` words, instead of `topn` words.

Am I reading that correctly?

> That dictionary is still smaller than 10, `len(id2word) == 2`.

In this contrived example, yep.

> But I think you may be right, this can happen whenever the trained model hasn't seen at least 10 terms when `show_topics()` is called. For example, when the first 20,000 documents (default `chunksize=20000`) contained less than 10 unique terms.
> 
> Still a pretty weird setup, but it can happen. We should detect when that happens, and output only the top `min(topn, actual_active_terms)` words, instead of `topn` words.
> 
> Am I reading that correctly?

Yeah, I think we're on the same page.  I was looking for something like `actual_active_terms`, but I didn't see it being tracked anywhere.  I think that would solve the issue, yes.
Can you open a PR that replaces that line with `return [(self.id2word[val], 1.0 * c[val] / norm) for val in most if val in self.id2word]`?

And verify that it fixes your original problem. I'm still not clear about where such issue could realistically appear – less than 10 unique words in 20k documents is definitely suspicious. Are you sure your document pipeline is OK?
> Can you open a PR that replaces that line with `return [(self.id2word[val], 1.0 * c[val] / norm) for val in most if val in self.id2word]`?

I've verified the fix I posted above locally.  I can open a PR in a moment ..

> And verify that it fixes your original problem. I'm still not clear about where such issue could realistically appear – less than 10 unique words in 20k documents is definitely suspicious.

Here's another example that repro's the problem with > 10 terms:
```python
from gensim.models import LsiModel

# 20k docs with only nine features
tfidf = [[(i, 1.0) for i in range(9)] for _ in range(20_000)]
id2word = {i: "x" for i in range(9)}

# Discontinuous column index
tfidf.append([(100, 1.0)])
id2word[100] = "y"

LsiModel(tfidf, id2word=id2word)
```

> Are you sure your document pipeline is OK?

Haha good question.  This is definitely not a common case (< 1 in 10k).  I think we could see the issue if users uploaded lots of empty documents or small, duplicate documents, though.

Resolving patch:
diff --git a/gensim/models/lsimodel.py b/gensim/models/lsimodel.py
--- a/gensim/models/lsimodel.py
+++ b/gensim/models/lsimodel.py
@@ -670,7 +670,9 @@ def show_topic(self, topicno, topn=10):
         c = np.asarray(self.projection.u.T[topicno, :]).flatten()
         norm = np.sqrt(np.sum(np.dot(c, c)))
         most = matutils.argsort(np.abs(c), topn, reverse=True)
-        return [(self.id2word[val], 1.0 * c[val] / norm) for val in most]
+
+        # Output only (word, score) pairs for `val`s that are within `self.id2word`.  See #3090 for details.
+        return [(self.id2word[val], 1.0 * c[val] / norm) for val in most if val in self.id2word]
 
     def show_topics(self, num_topics=-1, num_words=10, log=False, formatted=True):
         """Get the most significant topics.
diff --git a/gensim/utils.py b/gensim/utils.py
--- a/gensim/utils.py
+++ b/gensim/utils.py
@@ -835,6 +835,9 @@ def __getitem__(self, val):
             return str(val)
         raise ValueError("internal id out of bounds (%s, expected <0..%s))" % (val, self.num_terms))
 
+    def __contains__(self, val):
+        return 0 <= val < self.num_terms
+
     def iteritems(self):
         """Iterate over all keys and values.
 


Test patch: