piskvorky/gensim — pull request #169 by josephcc (2014-02-17 20:23:44+00:00)
Title: Create a dictionary that maps document id to some metadata of the original corpus

This commit parses the Wikipedia Page id and title as metadata, and creates a Python dictionary that maps from gensim doc id to to the metadata. It also changes the serialization so that the dictionary is stored in a pickle file. Other corpus class can also be modified to support adding a metadata dictionary, but with the default argument the changes shouldn't effect corpus classes other than WikiCorpus. It also shouldn't effect the behavior of the WikiCorpus class if no additional parameters were specified.