piskvorky/gensim — Issue #56: Dictionary.save_as_text/load_from_text is dangerous
Author: heuer
Created: 2011-10-11 10:55:08+00:00
State: open

I thought that Dictionary.save_as_text and load_from_text is equivalent to Dictionary.save/load, but it isn't. The text format does not keep the "num_docs" and after loading a Dict from a text, several methods do not work anymore::

>>> dct = Dictionary()
>>> _ = dct.doc2bow(['Hi', 'there'], allow_update=True)
>>> _ = dct.doc2bow(['Hi', 'all'], allow_update=True)
>>> _ = dct.doc2bow(['Hi', 'there', 'world'], allow_update=True)
>>> print dct
Dictionary(4 unique tokens)
>>> print dct.num_docs
3
>>> dct.save('./test.dct')
>>> d2 = Dictionary.load('./test.dct')
>>> print d2.num_docs
3
>>> # Filter extremes
>>> d2.filter_extremes(no_below=1)
>>> print d2
Dictionary(2 unique tokens)
>>> print d2.token2id
{'world': 0, 'all': 1}

Everything works as expected. Now the text version (assuming the dct from above)::

>>> dct.save_as_text('./test.txt')
>>> d2 = Dictionary.load_from_text('./test.txt')
>>> print d2.num_docs
0
>>> print d2 # Everything is fine here, expecting 4 tokens
Dictionary(4 unique tokens)
>>> # Filter extremes
>>> d2.filter_extremes(no_below=1)
>>> print d2 # Whoops, all tokens are gone, expected 2, see above
Dictionary(0 unique tokens)
>>> print d2.token2id
{}

This behaviour is not documented anywhere and I'd think that save_as_text and load_from_text should lead to a fully functional Dictionary object