piskvorky/gensim — pull request #149 by darenr (2013-12-20 18:55:24+00:00)
Title: eat newline when present in binary dfformat

fixes issue with binary models. Some models have a newline after the word vectors, some do not.

From the hexdump analysis on an older trained file (from a few weeks ago)

1b bd 0a 73 65 76 65 6e 20 81 29 06 be 69 52 60 |...seven .)..iR`

notice that before the new word there is indeed a newline (bold), yet in the freebase model

00001f70 00 00 cc 3c 2f 65 6e 2f 62 61 72 61 63 6b 5f 6f |...