Package: wikipedia-histories
Wikipedia HistoriesA tool to pull the complete revision history of a Wikipedia page.InstallationTo install Wikipedia Histories, simply run:$pipinstallwikipedia-historiesWikipedia Histories is compatible with Python 3.6+.UsageThe module has basic functionality which allows it to be used to collect the revision history and metadata from a Wikipedia page in a convenient list of objects, which can be converted into a DataFrame. This also includes the article quality from every revision.>>>importwikipedia_histories# Generate a list of revisions for a specified page>>>golden_swallow=wikipedia_histories.get_history('Golden swallow')# Show the revision IDs for every edit>>>golden_swallow# [130805848, 162259515, 167233740, 195388442, ...# Show the user who made a specific edit>>>golden_swallow[16].user# u'Snowmanradio'# Show the text of at the time of a specific edit>>>golden_swallow[16].content# u'The Golden Swallow (Tachycineta euchrysea) is a swallow.  The Golden Swallow formerly'...>>>golden_swallow[200].content# u'The golden swallow (Tachycineta euchrysea) is a passerine in the swallow family'...# Get the article rating at the time of the edit>>>ratings=[revision.ratingforrevisioningolden_swallow]>>>ratings# ['NA', 'NA', 'NA', 'NA', 'stub', 'stub', ...# Get the time of each edit as a datetime object>>>times=[revision.timeforrevisioningolden_swallow]>>>times# [datetime.datetime(2007, 5, 14, 16, 15, 31), datetime.datetime(2007, 10, 4, 15, 36, 29), ...# Generate a dataframe with text and metadata from a the list of revisions>>>df=wikipedia_histories.to_df(golden_swallow)Additional metadata for the article, including
An example of this workflow is available intests/demo.py.Domain level analysisThis module also contains functionality for advanced analysis of large sets of Wikipedia articles by generation social networks based on the editors who edited an article. This functionality can be utilized by installing:pip install wikipedia_histories[networks]The toolkit is available atwikipedia_histories.networks.analyze_networksandwikipedia_histories.networks.network_builder.First, a domain is defined as adictionaryorjson, where keys are domain names and values are lists of categories which represent that domain. For example, a set of domains representing "culture" and "politics":{"culture":["Category:Television_in_the_United_States","Category:American_films","Category:American_novels"],"politics":["Category:Conservatism","Category:Liberalism"]}An example of this format is available inexamples/domains.json.The articles represented by those domains, up to a certain depth of nested categories, can be collected and saved as acsv, with the category and domain attributes attached usingwikipedia_histories.networks.get_category_articles.find_articles(). Once this set of articles is collected, the articles themselves can be downloaded usingwikipedia_histories.get_history()either with revision text or without. This set of articles can be used for analysis on Wikipedia revision behavior across categories or domains.Once a set of articles is downloaded using this methodology, it's possible to collect aggregate metadata for those articles, including the number of unique editors, average added words per edit and average deleted words per edit, the article age, and the total number of edits, and save that information into a DataFrame usingwikipedia_histories.get_metadata().An example of this workflow is available inexamples/collect_articles.py.Social network analysisIt is also possible to build and analyze the networks of users who edited those articles, and study how domains relate to one another. For this analysis, first a set of articles representing categorical domains must be downloaded using and saved to folders representing domains and the metadata sheet must be saved.Once this is set up, a set of networks representing connections within a domain or between domains can be generated. Adomainis passed as input to signify which domain should be used to build the networks, if nodomainis passed as input the networks generated will represent connections between categories from different domains.In each network created, nodes represent articles and weighted edges represent the number of common editors between two articles. The functionwikipedia_histories.networks.network_builder.generate_networks()allows generation of a certain number of networks with a specific number of nodes and a specific count--because they are generated by sampling from the downloaded articles, generating many networks represents bootstrapping of the dataset.The function call:networks=wikipedia_histories.networks.network_builder.generate_networks(count=1000,size=300,domain=domain,metadata_path=metadata_path,articles_path=articles_path,)would generate 1000 networks, each with 300 nodes, or 150 nodes from each selected category. Because the category input isNone, the two selected categories would be from different domains. Themetadata_pathparameter is a path to the metadata sheet generated by thefind_articles()function and thearticles_pathparameter is a path to the articles downloaded based on thefind_articles()metadata.The function returns a list ofNetworkXobjects. Networks can be written to the disk as.graphmlfiles as part of the function by toggling thewriteparameter toTrueand passing anoutput_folder(note that this aspect is necessary for analysis).Once generated, the networks can be analyzed using theget_network_metadata()function, which returns a DataFrame containing purity scores based on Louvain communities detected and assortativity scores for each network based on the categories represented by the networks.An example of this workflow is available inexamples/collect_networks.py.Wikipedia Histories is compatible with Python 3.6+.NotesThis package was used for a paper published by the McGill .txtlab:https://txtlab.org/2020/09/do-wikipedia-editors-specialize/.