Package: hatch-openzim
hatch-openzimThis provides aHatch(ling) plugin for common openZIM operations:automatically populate common project metadatasinstall static files (e.g. external JS dependencies) at build timeThis plugin intentionally has few dependencies, using the Python standard library whenever possible and hence limiting footprint to a minimum.hatch-openzim adheres to openZIM'sContribution Guidelines.hatch-openzim has implemented openZIM'sPython bootstrap, conventions and policiesv1.0.0.Quick startAssuming you have an openZIM project, you could use such a configuration in yourpyproject.toml# Use the hatchling build backend, with the hatch-openzim plugin.[build-system]requires=["hatchling","hatch-openzim"]build-backend="hatchling.build"[project]name="MyAwesomeScraper"requires-python=">=3.11,<3.12"description="Awesome scraper"readme="README.md"# These project metadatas are dynamic because they will be generated from hatch-openzim# and version plugins.dynamic=["authors","classifiers","keywords","license","version","urls"]# Enable the hatch-openzim metadata hook to generate default openZIM metadata.[tool.hatch.metadata.hooks.openzim-metadata]additional-keywords=["awesome"]# some additional keywordskind="scraper"# indicate this is a scraper, so that additional keywords are added# Additional author #1[[tool.hatch.metadata.hooks.openzim-metadata.additional-authors]]name="Bob"email="bob@acme.com"# Additional author #2[[tool.hatch.metadata.hooks.openzim-metadata.additional-authors]]name="Alice"email="alice@acme.com"# Enable the hatch-openzim build hook to install files (e.g. JS libs) at build time.[tool.hatch.build.hooks.openzim-build]toml-config="openzim.toml"# optional location of the configuration filedependencies=["zimscraperlib==3.1.0"]# optional dependencies needed for file installationsNOTA: thedependenciesattribute is not specific to our hook(s), it is a generichatch(ling) feature.Metadata hook usageConfiguration (inpyproject.toml)VariableRequiredDescriptionadditional-authorsNList of authors that will be appended to the automatic oneadditional-classifiersNList of classifiers that will be appended to the automatic onesadditional-keywordsNList of keywords that will be appended to the automatic oneskindNIf set toscraper, scrapers keywords will be automatically added as wellorganizationNOverride organization (otherwise detected from Github repository to set author and keyword appropriately). Case-insentive. Supported values areopenzim,kiwixandoffspotpreserve-authorsNBoolean indicating that we do not want to setauthorsmetadata but use the ones ofpyproject.tomlpreserve-classifiersNBoolean indicating that we do not want to setclassifiersmetadata but use the ones ofpyproject.tomlpreserve-keywordsNBoolean indicating that we do not want to setkeywordsmetadata but use the ones ofpyproject.tomlpreserve-licenseNBoolean indicating that we do not want to setlicensemetadata but use the one ofpyproject.tomlpreserve-urlsNBoolean indicating that we do not want to seturlsmetadata but use the ones ofpyproject.tomlBehaviorThe metadata hook will set:authorsto[{"email": "dev@kiwix.org", "name": "Kiwix"}]classifierswill contain:License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)allProgramming Language :: Python :: xandProgramming Language :: Python :: x.ymatching therequired-versionskeywordswill contain:at leastkiwixifkindisscraper, it will addzimandofflineandadditional-keywordspassed in the configurationlicenseto{"text": "GPL-3.0-or-later"}urlstoDonate:https://www.kiwix.org/en/support-us/Homepage: Github repository URL (e.g.https://github.com/openzim/hatch-openzim) if code is a git clone, otherwisehttps://www.kiwix.orgBuild hook usageHigh-level configuration (inpyproject.toml)VariableRequiredDescriptiontoml-configNLocation of the configuration, default toopenzim.tomlDetails configuration (inopenzim.toml)The build hook detailed configuration is done in a TOML file namedopenzim.toml(if not customized
viatoml-config, see above). This file must be placed your project root folder, next to yourpyproject.toml.The build hook supports to download web resources at various location at build time.To configure, this you first have to create afilessection in theopenzim.tomlconfiguration
and declare itsconfigconfiguration. Name of the section (assetsin example below) is
free (do not forgot to escape it if you want to use special chars like.in the name).[files.assets.config]target_dir="src/hatch_openzim/templates/assets"execute_after=["touch somewhere/something.txt"]VariableRequiredDescriptiontarget_dirYBase directory where all downloaded content will be placedexecute_afterNList of shell commands to execute once all actions (see below) have been executed; actions are executed withtarget_diras current working directoryImportant:Theexecute_aftercommands arealwaysexecuted, no matter how many action are
present or how many actions have been ignored (see below for details about why an action might be ignored).Nota: The exampleexecute_aftercommand (touch) is not representative of what you would usually do ^^Once this section configuration is done, you will then declare multiple actions. All
actions in a given section share the same base configuration declared above.Three kinds of actions are supported:get_file: downloads a file to a locationextract_all: extracts all content of a zip file to a locationextract_items: extracts some items of a zip file to some locationsEach action is declared in its own TOML table. Action names are free.[files.assets.actions.some_name]action=...get_fileaction configuration (inopenzim.toml)This action downloads a file to a location.Important:Iftarget_fileis already present, the action is not executed, it is simply ignored.VariableRequiredDescriptionactionYMust be "get_file"sourceYURL of the online resource to downloadtarget_fileYRelative path to the file target location, relative to the sectiontarget_direxecute_afterNList of shell commands to execute once file installation is completed; actions are executed with the sectiontarget_diras current working directoryYou will find a sample below.[files.assets.actions."jquery.min.js"]action="get_file"source="https://code.jquery.com/jquery-3.5.1.min.js"target_file="jquery.min.js"extract_allaction configuration (inopenzim.toml)This action downloads a ZIP and extracts it to a location. Some items in the Zip content
can be removed afterwards.Important:Iftarget_diris already present, the action is not executed, it is simply ignored.VariableRequiredDescriptionactionYMust be "extract_all"sourceYURL of the online ZIP to downloadtarget_dirYRelative path of the directory where ZIP content will be extracted, relative to the sectiontarget_dirremoveNList of glob patterns of ZIP content to remove after extraction (relative to actiontarget_dir)execute_afterNList of shell commands to execute once files extraction is completed; actions are executed with the sectiontarget_diras current working directoryYou will find a sample below.Nota:the ZIP is first saved to a temporary location before extraction, consuming some disk space[files.assets.actions.chosen]action="extract_all"source="https://github.com/harvesthq/chosen/releases/download/v1.8.7/chosen_v1.8.7.zip"target_dir="chosen"remove=["docsupport","chosen.proto.*","*.html","*.md"]extract_itemsaction configuration (inopenzim.toml)This action extracts a ZIP to a temporary directory, and move selected items to some locations.
Some sub-items in the Zip content can be removed afterwards.Important:If anytarget_pathsis already present, the action is not executed, it is simply ignored.VariableRequiredDescriptionactionYMust be "extract_all"sourceYURL of the online ZIP to downloadzip_pathsYList of relative path in ZIP to selecttarget_pathsYRelative path of the target directory where selected items will be moved (relative to ZIP home folder)removeNList of glob patterns of ZIP content to remove after extraction (must include the necessarytarget_paths, they are relative to the sectiontarget_dir)execute_afterNList of shell commands to execute once ZIP extraction is completed; actions are executed with the sectiontarget_diras current working directoryNota:thezip_pathsandtarget_pathsare matched one-by-one, and must hence have the same length.the ZIP is first saved to a temporary location before extraction, consuming some disk spaceall content is extracted before selected items are moved, and the rest is deletedYou will find a sample below.[files.assets.actions.ogvjs]action="extract_items"source="https://github.com/brion/ogv.js/releases/download/1.8.9/ogvjs-1.8.9.zip"zip_paths=["ogvjs-1.8.9"]target_paths=["ogvjs"]remove=["ogvjs/COPYING","ogvjs/*.txt","ogvjs/README.md"]Full sampleA full example with two distinct sections and three actions in total is below.Nota: Thetouchcommand inexecute_afteris not representative of what you would usually do ^^[files.assets.config]target_dir="src/hatch_openzim/templates/assets"execute_after=["fix_ogvjs_dist .",][files.assets.actions."jquery.min.js"]action="get_file"source="https://code.jquery.com/jquery-3.5.1.min.js"target_file="jquery.min.js"execute_after=["touch done.txt",][files.assets.actions.chosen]action="extract_all"source="https://github.com/harvesthq/chosen/releases/download/v1.8.7/chosen_v1.8.7.zip"target_dir="chosen"remove=["docsupport","chosen.proto.*","*.html","*.md"][files.videos.config]target_dir="src/hatch_openzim/templates/videos"[files.videos.actions.ogvjs]action="extract_items"source="https://github.com/brion/ogv.js/releases/download/1.8.9/ogvjs-1.8.9.zip"zip_paths=["ogvjs-1.8.9"]target_paths=["ogvjs"]remove=["ogvjs/COPYING","ogvjs/*.txt","ogvjs/README.md"]