Package: mcwiki
mcwikiA scraping library for theMinecraft Wiki.importmcwikipage=mcwiki.load("Data Pack")print(page["pack.mcmeta"].extract(mcwiki.TREE))[TAG_Compound]
The root object.
└─ pack
   [TAG_Compound]
   Holds the data pack information.
   ├─ description
   │  [TAG_String, TAG_List, TAG_Compound]
   │  A JSON text that appears when hovering over the data pack's name in
   │  the list given by the /datapack list command, or when viewing the pack
   │  in the Create World screen.
   └─ pack_format
      [TAG_Int]
      Pack version: If this number does not match the current required
      number, the data pack displays a warning and requires additional
      confirmation to load the pack. Requires 4 for 1.13–1.14.4. Requires 5
      for 1.15–1.16.1. Requires 6 for 1.16.2–1.16.5. Requires 7 for 1.17.IntroductionThe Minecraft Wiki is a well-maintained source of information but is a bit too organic to be used as anything more than a reference. This project tries its best to make it possible to locate and extract the information you're interested in and use it as a programmatic source of truth for developing Minecraft-related tooling.FeaturesEasily navigate through page sectionsExtract paragraphs, code blocks and recursive tree-like hierarchiesCreate custom extractors or extend the provided onesInstallationThe package can be installed withpip.$pipinstallmcwikiGetting StartedTheloadfunction allows you to load a page from the Minecraft Wiki. The page can be specified by providing a URL or simply the title of the page.mcwiki.load("https://minecraft.fandom.com/wiki/Data_Pack")mcwiki.load("Data Pack")You can use theload_filefunction to read from a page downloaded locally or thefrom_markupfunction if you already have the html loaded in a string.mcwiki.load_file("Data_Pack.html")mcwiki.from_markup("<!DOCTYPE html>\n<html ...")Page sections can then be manipulated like dictionaries. Keys are case-insensitive and are associated to subsections.page=mcwiki.load("https://minecraft.fandom.com/wiki/Advancement/JSON_format")print(page["List of triggers"])<PageSection ['minecraft:bee_nest_destroyed', 'minecraft:bred_animals', ...]>Extracting DataThere are 4 built-in extractors. Extractors are instantiated with a CSS selector and define aprocessmethod that produces an item for each element returned by the selector.ExtractorTypeExtracted ItemPARAGRAPHTextExtractor("p")String containing the text content of a paragraphCODETextExtractor("code")String containing the text content of a code spanCODE_BLOCKTextExtractor("pre")String containing the text content of a code blockTREETreeExtractor()An instance ofmcwiki.Treecontaining the treeview dataPage sections can invoke extractors by using theextractandextract_allmethods. Theextractmethod will return the first item in the page section orNoneif the extractor couldn't extract anything.print(page.extract(mcwiki.PARAGRAPH))Custom advancements in data packs of a Minecraft world store the advancement data for that world as separate JSON files.You can use theindexargument to specify which paragraph to extract.print(page.extract(mcwiki.PARAGRAPH,index=1))All advancement JSON files are structured according to the following format:Theextract_allmethod will return a lazy sequence-like container of all the items the extractor could extract from the page section.forparagraphinpage.extract_all(mcwiki.PARAGRAPH):print(paragraph)You can use thelimitargument or slice the returned sequence to limit the number of extracted items.# Both yield exactly the same listparagraphs=page.extract_all(mcwiki.PARAGRAPH)[:10]paragraphs=list(page.extract_all(mcwiki.PARAGRAPH,limit=10))Tree StructuresTheTREEextractor returns recursive tree-like hierarchies. You can use thechildrenproperty to iterate through the direct children of a tree.defprint_nodes(tree:mcwiki.Tree):forkey,nodeintree.children:print(key,node.text,node.icons)print_nodes(node.content)print_nodes(section.extract(mcwiki.TREE))Folded entries are automatically fetched, inlined, and cached. This means that iterating over thechildrenproperty can yield a node that's already been visited so make sure to handle infinite recursions where appropriate.Tree nodes have 3 attributes that can all be empty:Thetextattribute holds the text content of the nodeTheiconsattribute is a tuple that stores the names of the icons associated to the nodeThecontentattribute is a tree containing the children of the nodeYou can transform the tree into a shallow dictionary with theas_dictmethod.# Both yield exactly the same dictionarynodes=tree.as_dict()nodes=dict(tree.children)ContributingContributions are welcome. Make sure to first open an issue discussing the problem or the new feature before creating a pull request. The project usespoetry.$poetryinstallYou can run the tests withpoetry run pytest.$poetryrunpytestThe project must type-check withpyright. If you're using VSCode thepylanceextension should report diagnostics automatically. You can also install the type-checker locally withnpm installand run it from the command-line.$npmrunwatch
$npmruncheckThe code follows theblackcode style. Import statements are sorted withisort.$poetryrunisortmcwikitests
$poetryrunblackmcwikitests
$poetryrunblack--checkmcwikitestsLicense -MIT