Package: wscraper
Wikipedia ScraperParsing, tokenizing text using a Wikipedia dump XML file.AuthorName: T.FurukawaEmail:tfurukawa.mail@gmail.comInstallationpipinstallwscraperSupportlanguagejapaneseJapanese WikipediaenglishEnglish WikipediaHow to Work (Command)Check Console CommandsPlease run this command.wscraper--helpExecutable commands to be listed.InitializeFor start, you have to execute this command.It creates necessary directory and files.wscraperinitializewscraper root directory is created at$HOME/.wscraper.If you change this path, please set environmentWSCRAPER_ROOT.Set Global Parameterswscraperrootset--languagejapanese--page_chunk1000languageDefault language. If you do not set the parameterlanguagefor each corpus, this default language is used.page_chunkA Wikipedia dump XML file has large text data as many pages. For analysis, it is separated to several small files because of memory efficiency.Seewscraper root set -hImport a Wikipedia XML FileA file wikipedia.xml assumes like(lang)wiki-(date)-pages-articles-multistream.xmlwscraperimport/path/to/sample.xml
wscraperimport/path/to/wikipedia.xml--namemy_wpSeewscraper import -h.Check Wikipedia ResourcesIt can check Wikipedia corpus resources.wscraperlistoutputAvailable wikipedia:
  - sample
  - my_wpSwitch Current Corpuswscraperswitchmy_wpCheck the Status of Current Corpuswscraperstatusoutputcurrent: my_wp

language [default]: japaneseSet Parameters for Current CorpusRequired parameters should be set for current corpus.wscraperset--languageenglishparameters:languageUnset ParametersYou can delete parameters by running following command.wscraperunset--languageRename a Corpus NameYou can rename a corpus name from$sourceto$target.wscraperrename$source$targetDelete a CorpusWhen a corpus (example:$target) is unnecessary, it can be removed.wscraperdelete$targetHow to Work (Python)Importing iterator classes.fromwscraper.analysisimport*You can iterate pages of a corpus by writing this.# entryentry_iterator=EntryIterator()# You can specify corpus name and language.# If parameter is not given, current Wikipedia corpus is used.# >>> EntryIterator(name = "sample", language = "japanese")both_iterator=BothIterator()redirection_iterator=RedirectionIterator()fori,binenumerate(both_iterator):print(f"both:{i}:{type(b)}")fori,einenumerate(entry_iterator):print(f"entry{i}:{e.title}{len(e.mediawiki)}")fori,rinenumerate(redirection_iterator):print(f"redirection:{i}:{r.source}->{r.target}")For example, you can give iterator a ML model.defto_words(x):returnx.split()# return word list for each iterationiterator=ArticleIterator(tagger=to_words)# If you set `type = dict`, you can get records as dictionary# ex: { "title", "ABC...", "article": "eval(to_words(article))" }# Iterators:#   ArticleIterator#     - 1 page / record#     - dict keys: ["title", "article"]#   ParagraphIterator:#     - N page / record#     - Description like "== A ==" is delimiter of paragraphs.#     - dict keys: ["page_title", "paragraph_title", "paragraph"]# For example, gensim word2vec can interpret this iterator# from gensim.models.word2vec import Word2VecWord2Vec(iterator)# You can concatenate iterators to train ML model using CombinedIterator.Word2Vec(CombinedIterator(iterator,another_iterator))LicenseThe source code is licensed MIT.Please check the file LICENSE.