Package: info.gianlucacosta.wikiprism
WikiPrismParse wiki pages and create dictionaries, fast, with PythonIntroductionWikiPrismis a Python library designed to:Parse awiki XML filein order to extract itspages- contained within <page> tagsParse each page so as to extracttermsfrom it - that will be added to a customDictionaryinstanceThe above tasks can be easily combined in a sophisticated but user-friendly automated process calledextraction pipeline- configurable via a dedicated descriptor object.WikiPrism focuses onspeed- a goal achieved by introducing careful design choices, including thetransparentuse ofparallelismvia bothmultithreadingandmultiprocessing, while orchestrating several elements provided by the Eos library and without losing architecturalsimplicity.In this guide, we are going to see an overview of the library.InstallationTo installWikiPrism, just run:pip install info.gianlucacosta.wikiprismor, if you are using Poetry:poetry add info.gianlucacosta.wikiprismThen, you'll be able to access theinfo.gianlucacosta.wikiprismpackage and its subpackages.Extracting pages from a wiki sourceTo performblazing-fastparsing, WikiPrism relies on customizedSAX; more precisely, it expects a wiki file having whatever structure one prefers, as long as its pages comply with the following schema:<page>...<title>Thepagetitle</title>...<text>Thepagecontent</text>...</page>Autrement dit, as long as the XML file contains - anywhere - one or more<page>tags having the described structure, WikiPrism will be able to detect them.In practiceParsing can be performed via Python's standard functions within thexml.saxnamespace - especially:parse()- to parse an XML fileparseString()- to parse an XML stringBoth functions require aContentHandlerand an optionalErrorHandler- and that's precisely why WikiPrism provides two dedicated classes:WikiContentHandlerWikiErrorHandlerIn particular,WikiContentHandler's constructor expects:acallback, that receives aPageobject - containing just thetitleandtextattributes - as soon as a valid page is found by the SAX parseraContinuationProvider, that is a () -> bool function periodically called: should it return False, the parsing would end by raising aWikiSaxCanceledExceptionFor more details, please consult the docstrings.Creating dictionariesPython has dictionaries - intended as hash maps - but WikiPrism introduces the abstract class namedDictionary[TTerm]as a generic container ofterms, which can be language elements like nouns, verbs, conjunctions, or even anything else, according to one's linguistic model... the exact purpose of each dictionary is up to the developer.Dictionary[TTerm]is a sort of generic repository forterms, via its 2 main abstract methods:add_term(): adds a term to the dictionary - actually writing to the storage technology and maybe preventing duplicationexecute_command(): runs a command string on the (arbitrary) underlying storage and returns aResult[DictionaryView]- that is, à la Rust, either aDictionaryViewobject (a table-like DTO with both headers and data rows) or anExceptionDictionary also has other abstract methods to implement, but it never makes assumptions about its internal data storage; consequently, for convenience, WikiPrism also provides concrete subclasses:InMemoryDictionary[TTerm]- adding terms to a Python set, but unable to perform commandsSqliteDictionary[TTerm]- dictionary backed by a SQLite db and passing commands to the related SQL interpreterFor further details, please consult the docstrings.The extraction pipelineTo combine wiki page extraction and dictionary creation into a performant, automated and easy-to-use process, WikiPrism definesextraction pipelines.Running an extraction pipeline basically resolves to:Creating a custom subclass ofPipelineStrategy[TTerm]- or its SQLite-oriented subclass,SqlitePipelineStrategy[TTerm]Invoking therun_extraction_pipeline()function, which only expects an instance of the strategyrun_extraction_pipeline()executes the pipelinein a separate thread(plus its own subthreads and a process pool), therefore it returns aPipelineHandle- an object with the following methods:join()- to wait for its completion - and supporting the same parameters as Thread'sjoin()methodrequest_cancel()- to stop the pipeline in a clean way, as soon as possibleFor more details, please consult the docstrings, the tests, and possibly the whole open sourceCervantesproject.Related projectsCervantes- WikiPrism applied to Wikcionario in order to exploreSpanishmorphology with PythonEos-core- type-checked and dependency-free modern utility library for Python