added test directory to MANIFEST.in, so it gets distributed with source.tgz
added hierarchical logging to all modules
Merge branch 'issue17' into develop
fixed Dieter's name in changelog
removed trailing whitespace; see github wiki

https://github.com/piskvorky/gensim/wiki
cleanup of BleiCorpus code
added __version__ attribute
set all logger levels to NOTSET

There was a request for the logging level to be configurable from a single point, preferably with a single command.
See http://groups.google.com/group/gensim/browse_thread/thread/ff363fb5f07b6d01# for a discussion of the solution (NOTSET = the default logger level).
Fix a bug in the log_entropy_model 

The previous implementation had the mistake that it divided by the context diversity of a term instead of the total number of documents. It happened because I read it from a paper where the notation was misleading. Of course it could not be like this because then all terms with a context diversity of 1 would lead to a division by zero. 

For reference of the normalization see: 
Pincombe, B. (2004). Comparison of human and LSA judgements of pairwise document similarities for a news corpus. dspace.dsto.defence.gov.au.

also the tests are changed to fit the new implementation
Merge remote branch 'dedan/log_entropy_fix' into develop
fixed bz2/gzip input for MmCorpus

* was broken by adding IndexedCorpus
optimized memory in lda
recompute id2word after updating Dictionary with new docs

`id2word` has "lazy" semantics (computed only when asked to), it acts like a cache.
Previously the cache never got updated; now it's updated whenever the Dictionary changes and `id2word` is requested again.
Automated test to reproduce the results of Lee et al. (2005)

Lee et al. (2005) compares different models for semantic
similarity and verifies the results with similarity judgements from humans.
The main result is that semantic similarites modelled by LSA have a correlation
of 0.6 with human similarity judgements.
As a validation of the gensim implementation we reproduced the results
of  Lee et al. (2005) in this test.

Many thanks to Michael D. Lee (michael.lee@adelaide.edu.au) who provideded us
with his corpus and similarity data.

If you need to reference this dataset, please cite:

Lee, M., Pincombe, B., & Welsh, M. (2005).
An empirical evaluation of models of text document similarity.
Proceedings of the 27th Annual Conference of the Cognitive Science Society
removed (undocumented) dependencies on nose

* switched test_lee.py from nose test to unittest (consistent with the rest of gensim)
* added a matutils.triu_indices fnc for numpy < 1.4
* removed citation section from top-level README
improved document chunking code in utils
minor logging fixes
fixed threaded chunking over an empty corpus
make the top level directory of the repo a python module

In the top level of the gensim repo you have to
put a a __init__.py containing the following line:
__path__ = './src/gensim'

this points to the actual module. For me it is very useful because I want to
record the commit hashes of the modules I am using for an experiment.

basically it is the answer to the question:
for our experiments we use a framework which records the version of all modules and if they are under version control also the hash of the commit. Usually I have gensim in my home directory and link it to the site-packages. But I link /Users/me/gensim/src/gensim  and the .git directory is in /Users/me/gensim so this framework does not see that the module is under version control. Is there a way to link to /Users/me/gensim and somehow tell python that there is an module in ./src/gensim ?
allows to evaluate lsi models with lower dimensionality than originally trained

It can be useful to train a model with e.g. 100 topics and then check how good the results would have been with only 10 topics. This can be done now by simply setting the model.numTopics variable to a lower level than before.