piskvorky/gensim — Issue #74: Add more data sanity checks
Author: piskvorky
Created: 2012-01-29 13:24:55+00:00
State: open

There's been a steady stream of reports that LSI/LDA misbehave, produce degenerate models, crash Python etc.

Typically this is a user data problem (bad input data, feature id mismatch, ...), but since gensim targets the wide general public, this is gensim's "fault" anyway.

Create utility functions that perform basic sanity checks on user's input data:

1. check that the all feature ids in a corpus are compatible with the user-provided dictionary (should avoid issues like http://projects.scipy.org/scipy/ticket/1582 )
2. check that the data range is valid -- look for NaNs, Infs, explicit zeros => these are all illegal in gensim input.
3. check that the data is not degenerate => all vectors identical/empty/?