piskvorky/gensim — pull request #196 by piskvorky (2014-04-19 22:32:03+00:00)
Title: Py3k

All tests pass, but this is still work-in-progress, because all tutorials/docs need to be updated too (NOTE: look into ipy notebooks, #180 ).

Changes aim to be backward compatible, but the new official gensim API will recommend **using unicode wherever strings are expected** (now: asks for utf8). Starting with Python 3.3, there's no longer a huge memory overhead with unicode strings (Python 2 takes a hit though).

Highlights:
* all I/O is binary, converted to unicode explicitly inside gensim (=not using the new `encoding` params to `open`&co). Uses `utils.smart_open()` consistently.
* all internally stored tokens/keys are expected to be unicode now. Find bytestring/ambiguity => file a bug.
* need more backward compat tests =make sure utf8 inputs continue work, too. Don't want the same migration drama py3 itself is having.
* I think this warrants a full version bump => `0.10.0`

Ping @larsmans @simonzack @moustaki can you please run `setup.py test` unit tests on your machines? I'll ping Steph for a Windows run too.