Package: wikipedia_tools
Welcome to the Wikipedia Periodic Revisions(wikipedia_tools)wikipedia_toolsis a Python Package to download wikipedia revisions for pages belonging to certaincategories, based on a period of time. This package also provides overview stats for the downloaded data.Dependencies and CreditsWikipedia APIThis package is built on top of theWikipedia API. This code was forked under thebasesubpackage.ajoer/WikiRevParserAlso we forked the code fromajoer/WikiRevParserand we modified it to supportfromandtodatetime to fetch revisions between certain periods; the modified code iswikipedia_toools.scraper.wikirevparser_with_time.py.Note: No need to download these two projects, they are already integrated as part of this project.InstallationVia PIPpip install wikipedia_toolsOr install manually by cloning and then runningpip install -e wikipedia_toolswikipedia_tools packageThis packages is responsible for:fetching the wikipages revisions based on a period of timeload them into parquet, andprovide basic analysisIt contains three main subpackages and theutilspackage which contains few helpers functions:Downlaod Wiki Article Revisions [wikipedia_tools.scraper]This subpackage is responsible for downloading the wikipedia revisions from the web.The code below shows how to download all the revisions of pages:belonging to theClimate_changecategory.revisions between start of 8 months ago (1.1.2022) and now (29.9.2022). Theget_x_months_ago_datefunction returns the datetime of the beginning of 8 months ago.fromwikipedia_tools.utilsimportutilsutils.get_x_months_ago_date(8)if  save_each_page= True: each page is fetched and downloaded on the spot under the folderdata/periodic_wiki_batches/{categories_names}/from{month-year}_to{month-year}. Otherwise, all the page revisions are fetched first and then saved into one jsonl file.fromwikipedia_tools.scraperimportdownloaderfromdatetimeimportdatetimewikirevs=downloader.WikiPagesRevision(categories=["Climate_change"],revisions_from=utils.get_x_months_ago_date(8),revisions_to=datetime.now(),save_each_page=True)count,destination_folder=wikirevs.download()For german wiki revisions, you can set thelangattribute tode- For example, you can download the German Wikipedia page revisions for the Climate_change category, as follows:fromwikipedia_tools.scraperimportdownloaderfromdatetimeimportdatetimewikirevs=downloader.WikiPagesRevision(categories=["Klimaveränderung"],revisions_from=utils.get_x_months_ago_date(1),# beginning of last month, you can use instead datetime.now() + dateutil.relativedelta.relativedelta() to customize past datetime relativelyrevisions_to=datetime.now(),save_each_page=True,lang="de")count,destination_folder=wikirevs.download()You can then process each file by, for example, reading the parquet file using pandas:importpandasaspdfromglobimportglobfiles=f"{destination_folder}/*.parquet"# Loop over all wiki page revisions with this period and read each wiki page revs as a pandas dataframeforpage_pathinglob(files):page_revs_df=pd.read_parquet(page_name)# dataframe with columns ['page', 'lang', 'timestamp', 'categories', 'content', 'images', 'links', 'sections', 'urls', 'user']# process/use file ....Overview Stats## Initialize the analyzer objectfromwikipedia_tools.analyzer.revisionsimportWikipediaRevisionAnalyzeranalyzer=WikipediaRevisionAnalyzer(category=category,period=properties.PERIODS._YEARLY_,corpus=CORPUS,root=ROOT_PATH)# Get the yearly number of articles that were created/edit at least onceunique_created_updated_articles=analyzer.get_edited_page_count(plot=True,save=True)# Returned the number of created articles over timeunique_created_articles=analyzer.get_created_page_count(plot=True,save=True)# Returns the number of revisions over timerev_overtime_df=analyzer.get_revisions_over_time(save=True)# Returns the number of words over timewords_overtime_df=analyzer.get_words_over_time(save=True)# Returns the number of users over time, grouped by user typeusers_overtime_df=analyzer.get_users_over_time(save=True)# return the top n wikipedia articles over timetop_edited=analyzer.get_most_edited_articles(top=4)# return the articles sorted from most to least edited over timemost_to_least_revised=analyzer.get_periodic_most_to_least_revised(save=True)You can find the full example under the examples folder.