Repository: piskvorky/gensim

Issue:
Update: assert self.projection.u is not None, "decomposition not initialized yet"
So I am also receiving the error below (one that was previously reported, but closed for lack of information  #2278 ) - but the good news is I have a working minimal example that replicates it. It seems to be more an error handling issue than anything, as it requires you to set the input to dictionary to the empty list (which is invalid syntax).

Error:
assert self.projection.u is not None, "decomposition not initialized yet"

To replicate:
```
from gensim import corpora, models, similarities

texts = []
dictionary = corpora.Dictionary(texts)
corpus = []
lsi = models.LsiModel(corpus, id2word=dictionary, num_topics=2)
index = similarities.MatrixSimilarity(lsi[corpus])
```


Discussion:
Hi Mark, the error message seems correct – you provided no training data, so the LSI model is uninitialized.

What exactly is the issue? What would you expect instead?
Hi piskvorky - I suppose "No training data provided - LSI model could not be initialized." would be useful, but that's just an idea. The message "assert self.projection.u is not None" is not very descriptive, and raising an AssertionError because of bad inputs is quite odd. I think also the addition of the word "yet" is quite misleading, as it implies the model is in the process of initialising and it has somehow been requested during this process.

I also think it's odd that the error is only raised on the final line, rather than in the construction of the LSI.
I see. Let's go point by point:

> I suppose "No training data provided - LSI model could not be initialized." would be useful, but that's just an idea. The message "assert self.projection.u is not None" is not very descriptive, and

The message now is `decomposition not initialized yet`. I agree something like `No training data provided - LSI model not initialized yet` is slightly better.

> raising an AssertionError because of bad inputs is quite odd.

That is true. This would be better as `ValueError`. Or `RuntimeError`? Instead of `AssertionError`.

> I think also the addition of the word "yet" is quite misleading, as it implies the model is in the process of initialising and it has somehow been requested during this process.

The model is indeed in the process of initializing – you can call `train` as many times as you like, the model supports incremental online learning. So the word "yet" fits well.

> I also think it's odd that the error is only raised on the final line, rather than in the construction of the LSI.

Not sure what you mean. Whether training data has been provided or not can only be known when you use the model. Otherwise the data could be coming next, why raise an error?
Yeah I agree with most of this - I'll do the same and go point by point:

> The message now is `decomposition not initialized yet`. I agree something like `No training data provided - LSI model not initialized yet` is slightly better

Yeah this message looks good to me!

> That is true. This would be better as ValueError. Or RuntimeError? Instead of AssertionError.

Just had a look at the python docs and this seems to meet the definition of ValueError well - it's a List so the correct type, but the list has an inappropriate value as it's of length 0: https://docs.python.org/3/library/exceptions.html#ValueError

> The model is indeed in the process of initializing – you can call train as many times as you like, the model supports incremental online learning. So the word "yet" fits well.

Ah fair enough - yeah that makes sense then, I agree

> Not sure what you mean. Whether training data has been provided or not can only be known when you use the model. Otherwise the data could be coming next, why raise an error?

So I didn't realise you could add data in segments in this way, but I see where you're coming from. Does it ever make sense to add to the model no data though? Because suppose: 

1. I add empty data to model
2. I add a list of data to the model
3. I inspect the model

Surely step 1 is always redundant, even if followed by step 2? I suppose I can see the point that it's defining the object like defining a list to be later appended to.
> Does it ever make sense to add to the model no data though?

Special cases are bad. Empty list is also a list, and LSI model has no problem "updating" from it (no-op), so why single it out? 

Making a special case out of empty list feels conceptually wrong. And also possibly undesirable for existing users who update the model automatically e.g. every day, regardless of whether the increment happens to be empty this day or not. Suddenly throwing exceptions there would break backward compatibility for no good reason, I'm -1 on that. The error belongs where somebody tries to use an uninitialized model, because we truly don't know what to do then => exception.

Although logging a warning may be OK – `LsiModel.train() called but no documents provided, is this intended?` or similar.

@mark-todd will you be able to submit a PR clarifying the messaging around LsiModel, for this edge case? Thanks.
> Although logging a warning may be OK – LsiModel.train() called but no documents provided, is this intended? or similar.

Yes I agree a warning would be much more appropriate here

> @mark-todd will you be able to submit a PR clarifying the messaging around LsiModel, for this edge case? Thanks.

I'm not that familiar with Gensim, but in terms of correcting the error message happy to have a go when I get a chance. Unsure about where to place the warning though; I can't find the train method in the source - could you point me to the method you mean?

OK, great! The method is actually called `add_documents`, not `train`, sorry.

Resolving patch:
diff --git a/gensim/models/lsimodel.py b/gensim/models/lsimodel.py
--- a/gensim/models/lsimodel.py
+++ b/gensim/models/lsimodel.py
@@ -70,6 +70,7 @@
 
 from gensim import interfaces, matutils, utils
 from gensim.models import basemodel
+from gensim.utils import is_empty
 
 logger = logging.getLogger(__name__)
 
@@ -489,7 +490,8 @@ def add_documents(self, corpus, chunksize=None, decay=None):
             chunksize = self.chunksize
         if decay is None:
             decay = self.decay
-
+        if is_empty(corpus):
+            logger.warning('LsiModel.add_documents() called but no documents provided, is this intended?')
         if not scipy.sparse.issparse(corpus):
             if not self.onepass:
                 # we are allowed multiple passes over the input => use a faster, randomized two-pass algo
@@ -590,7 +592,8 @@ def __getitem__(self, bow, scaled=False, chunksize=512):
             Latent representation of corpus in BoW format if `bow` is corpus.
 
         """
-        assert self.projection.u is not None, "decomposition not initialized yet"
+        if self.projection.u is None:
+            raise ValueError('No training data provided - LSI model not initialized yet')
 
         # if the input vector is in fact a corpus, return a transformed corpus as a result
         is_corpus, bow = utils.is_corpus(bow)
diff --git a/gensim/utils.py b/gensim/utils.py
--- a/gensim/utils.py
+++ b/gensim/utils.py
@@ -30,6 +30,7 @@
 from copy import deepcopy
 from datetime import datetime
 import platform
+import types
 
 import numpy as np
 import scipy.sparse
@@ -2084,3 +2085,19 @@ def effective_n_jobs(n_jobs):
     elif n_jobs < 0:
         n_jobs = max(multiprocessing.cpu_count() + 1 + n_jobs, 1)
     return n_jobs
+
+
+def is_empty(corpus):
+    """Is the corpus (an iterable or a scipy.sparse array) empty?"""
+    if scipy.sparse.issparse(corpus):
+        return corpus.shape[1] == 0  # by convention, scipy.sparse documents are columns
+    if isinstance(corpus, types.GeneratorType):
+        return False  # don't try to guess emptiness of generators, may lose elements irretrievably
+    try:
+        # list, numpy array etc
+        first_doc = next(iter(corpus))  # noqa: F841 (ignore unused variable)
+        return False  # first document exists => not empty
+    except StopIteration:
+        return True
+    except Exception:
+        return False


Test patch: