Package: wiki-as-base
wiki_as_base-py[MVP] Use MediaWiki Wiki page content as read-only database. Python library implementation. Seehttps://github.com/fititnt/openstreetmap-serverless-functions/tree/main/function/wiki-as-baseInstallingEnvironment variablesCommand line UsageQuickstartUse of permanent IDs for pages, the WikiMedia pageidsRequest multiple pages at once, either by pageid or titlesAdvanced filter with jqSave JSON-LD extracted as filesLibrary usageBasic useCache remote requests locallySafe inferred data as individual filesThe JSON-LD SpecificationDisclaimer / TriviaLicenseInstallingpipinstallwiki_as_base--upgrade## Alternative:# pip install wiki_as_base==0.5.10Environment variablesCustomize for your needs. They're shared between command line and the library.exportWIKI_API='https://wiki.openstreetmap.org/w/api.php'exportWIKI_NS='osmwiki'exportCACHE_TTL='82800'# 82800 seconds = 23 hoursSuggested. Customize user agent.Follows the logic of MediaWiki user agent. WithoutWIKI_AS_BASE_BOT_CONTACTcustomization, for recursive and pagination requests already not cached locally will far slower, with a delay 10 seconds. This default may increase in future releases. Does not affect direct requests (likely ones with less than 50 pages).exportWIKI_AS_BASE_BOT_CONTACT='https://github.com/fititnt/wiki_as_base-py; generic@example.org'Command line UsageQuickstartThese examples will request two wikies, OpenStreetMap (default)live pageand Wikidatalive page.wiki_as_base--help## Use remote storage (defined on WIKI_API)wiki_as_base--titles'User:EmericusPetro/sandbox/Wiki-as-base'# The output is JSON-LD. Feel free to further filter the datawiki_as_base--titles'User:EmericusPetro/sandbox/Wiki-as-base'|jq.data[1]## Example of, instead of use WIKI_API, parse Wiki markup directly. Output JSON- LDcattests/data/multiple.wiki.txt|wiki_as_base--input-stdin## Output zip file instead of JSON-LD. --verbose also adds wikiasbase.jsonld to filecattests/data/chatbot-por.wiki.txt|wiki_as_base--input-stdin--verbose--output-zip-filetests/temp/chatbot-por.zip## Use different Wiki with ad-hoc change of the env WIKI_API and WIKI_NSWIKI_NS=wikidatawiki\WIKI_API=https://www.wikidata.org/w/api.php\wiki_as_base--titles'User:EmericusPetro/sandbox/Wiki-as-base'Click to see more examples for other wikies# For suggestion of RDF namespaces, see https://dumps.wikimedia.org/backup-index.htmlWIKI_NS=specieswiki\WIKI_API=https://species.wikimedia.org/w/api.php\wiki_as_base--titles'Paubrasilia_echinata'# @TODO implement support for MediaWiki version used by wikies like this oneWIKI_NS=smwwiki\WIKI_API=https://www.semantic-mediawiki.org/w/api.php\wiki_as_base--titles'Help:Using_SPARQL_and_RDF_stores'Use of permanent IDs for pages, the WikiMedia pageidsIn case the pages are already know upfront (such as automation) then the use of numeric pageid is a better choice.# "--pageids '295916'" is equivalent to "--titles 'User:EmericusPetro/sandbox/Wiki-as-base'"wiki_as_base--pageids'295916'However, if for some reason (such as strictly enforce not just an exact page,
but exact version of one or more pages) and getting the latest version is not fully essential, then you can userevids,# "--revids '2460131'" is an older version of --pageids '295916' and# "--titles 'User:EmericusPetro/sandbox/Wiki-as-base'"wiki_as_base--revids'2460131'Request multiple pages at once, either by pageid or titlesEach MediaWiki API may have different limits for batch requests,
however even unauthenticated users often have decent limits (e.g. 50 pages).Some Wikies may allow very high limits for authenticated accounts (500 pages),
however the current version does not implement authenticated requests.## All the following commands are equivalent for the default WIKI_APIwiki_as_base--input-autodetect'295916|296167'wiki_as_base--input-autodetect'User:EmericusPetro/sandbox/Wiki-as-base|User:EmericusPetro/sandbox/Wiki-as-base/data-validation'wiki_as_base--pageids'295916|296167'wiki_as_base--titles'User:EmericusPetro/sandbox/Wiki-as-base|User:EmericusPetro/sandbox/Wiki-as-base/data-validation'Trivia:since this library and CLI fetch directly from WikiMedia API,
and parse Wikitext (not raw HTML),
it causes much less server load to request several pages this way than big ones with higher number of template calls 😉.Advanced filter with jqWhen working with the JSON-LD output, you can use jq ("jq is a lightweight and flexible command-line JSON processor."), see more onhttps://stedolan.github.io/jq/, to filter the data## Filter tableswiki_as_base--titles'User:EmericusPetro/sandbox/Wiki-as-base'|jq'.data[] | select(.["@type"] == "wtxt:Table")'## Filter Templateswiki_as_base--titles'User:EmericusPetro/sandbox/Wiki-as-base'|jq'.data[] | select(.["@type"] == "wtxt:Template")'Save JSON-LD extracted as filesUse--output-zip-fileparameter. One example:wiki_as_base--input-autodetect'Category:References'--output-zip-file~/Downloads/Category:References.zipLibrary usageNOTE: for production usage (if you can't review releases or are not locked into Docker images)
consider enforce a very specific releaseProduction usage# requirements.txt
wiki_as_base==0.5.10Other cases (or use in your local machine)# Run this via cli for force redownload lastest. Do not use --pre (pre-releases)pipinstallwiki_as_base--upgrade# requirements.txt
wiki_as_baseBasic useimportjsonfromwiki_as_baseimportWikitextAsDatawtxt=WikitextAsData().set_pages_autodetect("295916|296167")wtxt_jsonld=wtxt.output_jsonld()print(f'Total:{len(wtxt_jsonld["data"])}')forresourceinwtxt_jsonld["data"]:ifresource["@type"]=="wtxt:Table":print("table found!")print(resource["wtxt:tableData"])print("Pretty print full JSON output")print(json.dumps(wtxt.output_jsonld(),ensure_ascii=False,indent=2))Cache remote requests locallyTODO: port therequests-cacheapproach (local SQLite cache database) used onhttps://github.com/fititnt/openstreetmap-serverless-functions/blob/main/function/wiki-as-base/handler.py.Safe inferred data as individual filesimportsysimportzipfilefromwiki_as_baseimportWikitextAsDatawtxt=WikitextAsData().set_pages_autodetect("295916|296167")# Both output_jsonld() and output_zip() call prepare() (which actually# make the remote request) plus is_success() on demand.# However the pythonic way woud be try/exceptifnotwtxt.prepare().is_success():print("error")print(wtxt.errors)sys.exit(1)wtxt.output_zip("/tmp/wikitext.zip")# Using Python zipfile.ZipFile, you can process the file with pythonzip=zipfile.ZipFile("/tmp/wikitext.zip")print("Files inside the zip:")print(zip.namelist())# @TODO improve this example on future releasesThe JSON-LD SpecificationNOTE: work in progress.https://wtxt.etica.ai/Disclaimer / TriviaThe wiki_as_base allowsno-as-completedata extraction from MediaWiki markup text directly by its API or direct input,
without need to install server extensions.Check also thewikimedia/Wikibase, a full server version (which inspired the name).LicensePublic domain