Package: wikicat
wikicatA Python toolkit for managing and navigating graphs of Wikipedia categories 🔖SimplePython APIfor exploring graph offlineUseful CLI forprocessingandlaunching appInteractive visualization of categoriesUI to display information and filter nodesNoteIf you need help at any time, you can head over tothe official documentations.Main APINoteThe reference can be found on thedoc pageor indocs/wikicat.mdThe mainwikicatAPI allows you work with category graphs generated from a certain dump by Wikipedia. Once the dump is processed viawikicat.processing, you can easily navigate the graph using simple and clear Python code, all offline (i.e., you do not need to make web requests to Wikipedia, and you can choose dump going back to any date you prefer). The API is designed to be as simple as possible, and is intended to be used by researchers and developers who want to work with the Wikipedia category graph.To install the API, run:pip3 install wikicatwikicatcontains two classes to work with the Wikipedia category graph:CategoryGraphandPage. TheCategoryGraphclass is used to load the graph from a file, and to navigate the graph. ThePageclass is used to represent a Wikipedia page, and to retrieve information about the page from Wikipedia. They are meant to be used together, as shown in the following example:importwikicataswc# Load the graphcg=wc.CategoryGraph.read_json('~/.wikicat_data/enwiki_<yyyy>_<mm>_<dd>/category_graph.json')# Get the page for "Montreal"page=cg.get_page_from_title('Montreal','article')# Get the categories for "Montreal"cats=cg.get_parents(page=page)print(f"Category tags of{page.title}:{cats}")# Get URL of "Montreal"print("URL:",page.get_url())By default, the path will be~/.wikicat_data/, but the JSON can be stored anywhere you want (seewikicat.processingbelow for more information).wikicat.processingwikicat.processingis a command line interface (CLI) for downloading and processing the dataNoteThe reference can be found on thedoc pageor indocs/wikicat/processing.mdTo install the processing tools, run:pip3 install wikicat[processing]Now, following those instructions to download and process the data:# 1. Download DB dump of Wikipedia categories (extension .sql.gz)python3-mwikicat.processing.download_dump\--year<yyyy>\--month<mm>\--day<dd>\--base_dir~/.wikicat_data/# optional, default is ~/.wikicat_data/# 2. Process individual dumps (.sql.gz) into csv files (to be merged later)python3-mwikicat.processing.process_dump\-y<yyyy>-m<mm>-d<dd># 3. Merge the individual table csvs into a single category graph csvpython3-mwikicat.processing.merge_tables\-y<yyyy>-m<mm>-d<dd># 4. Convert CSV category graph into JSON category graph (the final output)python3-mwikicat.processing.generate_graph\-y<yyyy>-m<mm>-d<dd>Notes (by step):If you do not specify--base_dir, it will automatically be saved to~/.wikicat_data/enwiki_<yyyy>_<mm>_<dd>.This may take a while depending on your hardware, and will need plenty of RAM. It will generate two CSV files with the relevant tables.This should take under 30 mins depending on your hardware, and will generate a single CSV file with the relevant category graph links.The results will be saved in~/.wikicat_data/enwiki_<yyyy>_<mm>_<dd>/category_graph.json.wikicat.viewerwikicat.vieweris an application that lets you visually explore a category graphNoteThe reference can be found on thedoc pageor indocs/wikicat/viewer.mdTo install the viewer, run:pip3installwikicat[viewer]To run the viewer, run:python3-mwikicat.viewer-y<yyyy>-m<mm>-d<dd>--port8050Then, open your browser tohttp://0.0.0.0:8050.UsageThe viewer let you interact with the nodes. You can zoom in and out, move and click the nodes in the graph.When you click on a node, you will see various information (including a list of children articles) appear on the middle panel.On the right panel, you will see various checklists of children and parents of the selected node. When you click on "Update", the checked parents and children will be added to the graph.There's a dropdown and a validated input. For the input, a green check will appear if a valid article title is input, otherwise it remains red. The dropdown lets you choose one of28 top-level categories. The input let you type the name of an article (not category). When the title is valid, you can click on the "Compute Path" button, which willtryto find a valid path between the top-level category and the article you chose.Click on the "Reset" button to go back to the original view.Accessing componentswikicat.viewerwas built usingDash, a Python framework for building web applications. The application is composed of several components, which can be accessed insidewikicat.viewer.components. For example, to access theNetworkcomponent, you can run:importwikicat.viewer.componentsascomp# Build the networkcytoscape_graph=comp.build_cytoscape_graph(...)# Build the right panelpanel=comp.build_panel(...)Those can be reused in your custom Dash application. You can also create your own component and add it to the viewer. For example:importwikicat.vieweraswcv# ...# Define appapp=dash.Dash(__name__,external_stylesheets=[style],title=title,**kwargs)# Define your custom componentsdefbuild_btn(...):# ...# Build regular componentscyto_graph=wcv.components.build_cytoscape_graph(root)# ...cards=wcv.components.build_cards(cl=cl,sw=sw)cards_column=wcv.components.build_card_column(cards)# Build layoutapp.layout=wcv.components.build_layout(...)# Assign callbacks to make app interactivewcv.components.assign_callbacks(app=app,...)# Run apprun(app=app,...)See thewikicat.viewer.build_app()function for more details.WarningBecause of the size of the graph, some parts of the API (such as the viewer and the processing CLI) require a lot of memory. We recommend using a machine with at least 32 GB of RAM. We are working on a more memory-efficient version of the API.Alternativeswikicatwas designed for offline and high-throughput workflows, with support for different versions of Wikipedia (as categories change over time). As a result, there's a high overhead (long build time and high RAM usage). If this is not what you are looking for, you can check out alternatives to this library:MediaWiki: This is Wikipedia's web API, and contains documentations for accessing categories (seeAPI:CategoriesandAPI:Categorymembers)Wikipedia Histories: This library contains adomain-level analysismodule that allows you to query articles associated with a certain category. Since it utilizes the Wikipedia Web API, it does not have the same overhead.