Package: kwnlp-dump-downloader
Kensho Wikimedia for Natural Language Processing - Dump Downloaderkwnlp_dump_downloader is a Python package to help you download rawWikimedia dumps.Quick Install (Requires Python >= 3.6)pipinstallkwnlp-dump-downloaderPython PackageThis python package provides two main pieces of functionality.  The first allows you to check the status of a Wikipedia dump and the second allows you to download specific parts of a Wikipedia dump (called jobs).Checking Statusfromkwnlp_dump_downloaderimportget_dump_statuswp_yyyymmdd="20200920"wds=get_dump_status(wp_yyyymmdd)print(wds.report())abstractsdump:done✅
abstractsdumprecombine:done✅
allpagetitlesdump:done✅
articlesdump:done✅
articlesdumprecombine:done✅
articlesmultistreamdump:done✅
articlesmultistreamdumprecombine:done✅
babeltable:done✅
categorylinkstable:done✅
categorytable:done✅
changetagdeftable:done✅
changetagstable:done✅
externallinkstable:done✅
flaggedimagestable:done✅
flaggedpageconfigtable:done✅
flaggedpagependingtable:done✅
flaggedpagestable:done✅
flaggedrevspromotetable:done✅
flaggedrevsstatisticstable:done✅
flaggedrevstable:done✅
flaggedrevstrackingtable:done✅
flaggedtemplatestable:done✅
geotagstable:done✅
imagelinkstable:done✅
imagetable:done✅
iwlinkstable:done✅
langlinkstable:done✅
metacurrentdump:done✅
metacurrentdumprecombine:done✅
metahistory7zdump:done✅
metahistorybz2dump:done✅
namespaces:done✅
pagelinkstable:done✅
pagepropstable:done✅
pagerestrictionstable:done✅
pagetable:done✅
pagetitlesdump:done✅
protectedtitlestable:done✅
redirecttable:done✅
sitestable:done✅
sitestatstable:done✅
templatelinkstable:done✅
userformergroupstable:done✅
usergroupstable:done✅
wbcentityusagetable:done✅
xmlpagelogsdump:done✅
xmlpagelogsdumprecombine:done✅
xmlstubsdump:done✅
xmlstubsdumprecombine:done✅Downloading Jobsfromkwnlp_dump_downloaderimportdownload_jobswp_yyyymmdd="20200920"wd_yyyymmdd="20200921"data_path="/path/where/data/should/live"jobs_to_download=["pagetable","articlesdump"]download_jobs(wp_yyyymmdd,wd_yyyymmdd,data_path=data_path,jobs_to_download=jobs_to_download)Any of the jobs listed in the status report above can be specified in thejobs_to_downloadkwarg. In addition, there are two special job strings,pageviewcomplete: used to download monthly pageviews (e.g.pageviews-20200901-user.bz2)wikidata: used to download Wikidata json dumps (e.g.wikidata-20200921-all.json.bz2)Command Line InterfaceIf you prefer to use the command line to check status and download dumps, you can do that too. After pip installing this package, you should have two new commands available,Checking Statususage:kwnlp-get-dump-status[-h][--mirror_urlMIRROR_URL][--wikiWIKI][--loglevelLOGLEVEL]wp_yyyymmdd

getWikipediadumpstatus

positionalarguments:wp_yyyymmdddatestringforWikipediadump(e.g.20200920)optionalarguments:-h,--helpshowthishelpmessageandexit--mirror_urlMIRROR_URLbaseURLforWikimediadumps(e.g.https://dumps.wikimedia.org)--wikiWIKIselectswhichlanguagewikipediatouse(e.g.enwiki)--loglevelLOGLEVELpythonlogginglevelinteger(e.g.20)Downloading Jobsusage:kwnlp-download-jobs[-h][--data_pathDATA_PATH][--mirror_urlMIRROR_URL][--wikiWIKI][--jobsJOBS][--loglevelLOGLEVEL]wp_yyyymmddwd_yyyymmdd

downloadWikimediadata

positionalarguments:wp_yyyymmdddatestringforWikipediadump(e.g.20200920)wd_yyyymmdddatestringforWikidatadump(e.g.20200921)optionalarguments:-h,--helpshowthishelpmessageandexit--data_pathDATA_PATHpathtotopleveldatadirectory(e.g./data/wikimedia-ingestion)--mirror_urlMIRROR_URLbaseURLforWikimediadumps(e.g.https://dumps.wikimedia.org)--wikiWIKIselectswhichlanguagewikipediatouse(e.g.enwiki)--jobsJOBScommaseparatedlistofjobnamestodownload(e.g.pagecounts,pagetable)--loglevelLOGLEVELpythonlogginglevelinteger(e.g.20)LicenseLicensed under the Apache 2.0 License. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.Copyright 2020-present Kensho Technologies, LLC. The present date is determined by the timestamp of the most recent commit in the repository.