Repository: piskvorky/gensim

Issue:
wiki examples: sample code to get from tfidf doc to wikipedia title/uri and vice versa
I very much like the LSI and LDA [wiki examples](https://github.com/RaRe-Technologies/gensim/blob/develop/docs/src/wiki.rst), but one aspect that i think is missing is: how to get from tf-idf doc vectors (or later LSI / LDA vecs) back to Wikipedia URIs (or titles if easier) and vice versa?

Am i missing something obvious, or do i have to run another pass over the wiki dump, as the titles aren't saved anywhere?

I'll happily make a PR to extend the examples with this...


Discussion:
@joernhees can you please explain your issue clearly. 
i think it's already about as clear as i can...

- how to get from Wikipedia URI to vector?
- how to get from vector to Wikipedia URI?

after following the tutorial, i wasn't able to easily find a link between "topic #0" and its wikipedia article
@joernhees As I remember, Wikipedia doesn't store URI in dumb (but you can re-store it using "Title").

About your questions: you should pass your page bag-of-words to any model (TfIdf, Lsi, Lda, etc), for this you should use `__getitem__` method, for example `model[my_bow]`

From vector, you can't retrieve original article (but you can store mapping `article <-> vector`)


@joernhees the tutorial doesn't construct any url/id<->vector mapping by default.

IIRC there was some discussion around adding this, since it's such a useful and commonly requested feature. In fact, it may have been added in one of the recent notebooks -- @menshikh-iv ?
Found it: https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/corpora/wikicorpus.py#L278

@menshikh-iv @joernhees can you open an issue that will set `metadata=True` in the `make_wiki` script? (and test that it works)
The vectors without titles are kinda useless, so `True` is a better default.

At the same time, fix the language in the docs, because it reads poorly and is not very informative. Related to #1161.
Hello!

What is the status of this issue? If it is not then I'd like to take it up. The task is to create a mapping from the doc with the Tf-Idf values to the actual corpus article?

Thanks.
Hello @AlokDebnath, feel free to contribute! The task is as https://github.com/RaRe-Technologies/gensim/issues/1322#issuecomment-322459901 suggested + some kind of mapping with vectors (as optional thing)
can I take up this issue ?
@Naba7 of course, feel free to contribute!

Resolving patch:
diff --git a/gensim/scripts/make_wikicorpus.py b/gensim/scripts/make_wikicorpus.py
--- a/gensim/scripts/make_wikicorpus.py
+++ b/gensim/scripts/make_wikicorpus.py
@@ -12,13 +12,15 @@
 Convert articles from a Wikipedia dump to (sparse) vectors. The input is a
 bz2-compressed dump of Wikipedia articles, in XML format.
 
-This actually creates three files:
+This actually creates several files:
 
-* `OUTPUT_PREFIX_wordids.txt`: mapping between words and their integer ids
-* `OUTPUT_PREFIX_bow.mm`: bag-of-words (word counts) representation, in
-  Matrix Matrix format
-* `OUTPUT_PREFIX_tfidf.mm`: TF-IDF representation
-* `OUTPUT_PREFIX.tfidf_model`: TF-IDF model dump
+* `OUTPUT_PREFIX_wordids.txt.bz2`: mapping between words and their integer ids
+* `OUTPUT_PREFIX_bow.mm`: bag-of-words (word counts) representation in Matrix Market format
+* `OUTPUT_PREFIX_bow.mm.index`: index for `OUTPUT_PREFIX_bow.mm`
+* `OUTPUT_PREFIX_bow.mm.metadata.cpickle`: titles of documents
+* `OUTPUT_PREFIX_tfidf.mm`: TF-IDF representation in Matix Market format
+* `OUTPUT_PREFIX_tfidf.mm.index`: index for `OUTPUT_PREFIX_tfidf.mm`
+* `OUTPUT_PREFIX.tfidf_model`: TF-IDF model
 
 The output Matrix Market files can then be compressed (e.g., by bzip2) to save
 disk space; gensim's corpus iterators can work with compressed input, too.
@@ -80,7 +82,7 @@
         dictionary.allow_update = True  # start collecting document frequencies
         wiki = WikiCorpus(inp, lemmatize=lemmatize, dictionary=dictionary)
         # ~4h on my macbook pro without lemmatization, 3.1m articles (august 2012)
-        MmCorpus.serialize(outp + '_bow.mm', wiki, progress_cnt=10000)
+        MmCorpus.serialize(outp + '_bow.mm', wiki, progress_cnt=10000, metadata=True)
         # with HashDictionary, the token->id mapping is only fully instantiated now, after `serialize`
         dictionary.filter_extremes(no_below=20, no_above=0.1, keep_n=DEFAULT_DICT_SIZE)
         dictionary.save_as_text(outp + '_wordids.txt.bz2')
@@ -91,7 +93,7 @@
         # only keep the most frequent words (out of total ~8.2m unique tokens)
         wiki.dictionary.filter_extremes(no_below=20, no_above=0.1, keep_n=DEFAULT_DICT_SIZE)
         # save dictionary and bag-of-words (term-document frequency matrix)
-        MmCorpus.serialize(outp + '_bow.mm', wiki, progress_cnt=10000)  # another ~9h
+        MmCorpus.serialize(outp + '_bow.mm', wiki, progress_cnt=10000, metadata=True)  # another ~9h
         wiki.dictionary.save_as_text(outp + '_wordids.txt.bz2')
         # load back the id->word mapping directly from file
         # this seems to save more memory, compared to keeping the wiki.dictionary object from above


Test patch: