Package: pawpaw
PawpawPawpaw is a high performance parsing & text segmentation framework that allows you to quickly and easily build complex, pipelined parsers.  Segments are automatically organized into tree graphs that can be serialized, traversed, and searched using a powerful structured query language calledplumule.Indexed string and substring representationEfficient memory utilizationFast processingPythonic relative indexing and slicingRuntime & polymorphic value extractionTree graphs for all indexed textSearch and QuerySearch trees using plumule: a powerful structured query language similar to XPATHCombined multiple axes, filters, and subqueries sequentially and recursively to any depthOptionally pre-compile queries for increased performanceRules Pipelining EngineDevelop complex lexical parsers with just a few lines of codeQuickly and easily convert unstructured text into structured, indexed, & searchable tree graphsPre-process text for downstream NLP/AI/ML consumersXML ProcessingFeatures a drop-in replacement forElementTree.XmlParserFull text indices for all elements, attributes, tags, text, etc.Search the resulting XML using either XPATH and/or plumuleExtractbothElementTreeand Pawpaw datastructures in one go; with cross-linked nodes between treesNLP Support:Pawpaw is ideal for both a)preprocessingunstructured text for downstream NLP consumption and b)storing and searchingNLP generated contentWorks with other libraries, such asNLTKEfficient pickling and JSON persistenceSecurity option enables persistence of index-only data, with reference strings re-injected during de-serializationStable & Defect FreeOver 5,000 unit tests and counting!Pure Python, with only one external dependency:regexExplore the docs•Request a feature or report a bug•Explore the codeExampleWith Pawpaw, you can start with flattened text like this:ARTICLE I
Section 1: Congress
All legislative Powers herein granted shall be vested in a Congress of the United States,
which shall consist of a Senate and House of Representatives.

Section 2: The House of Representatives
The House of Representatives shall be composed of Members chosen every second Year by the
People of the several States, and the Electors in each State shall have the Qualifications
requisite for Electors of the most numerous Branch of the State Legislature.

No Person shall be a Representative who shall not have attained to the Age of twenty five
Years, and been seven Years a Citizen of the United States, and who shall not, when elected,
be an Inhabitant of that State in which he shall be chosen.and quickly and easily produce a tree that look like this:graph TD;
  A1["[article]<br/>#quot;ARTICLE I…#quot;"]:::dark_brown --> A1_k["[key]<br/>#quot;I#quot;"]:::dark_brown;
  A1--->Sc1["[section]<br/>#quot;Section 1…#quot;"]:::light_brown;
  Sc1-->Sc1_k["[key]<br/>#quot;1#quot;"]:::light_brown
  Sc1--->Sc1_p1["[paragraph]<br/>#quot;All legislative Powers…#quot;"]:::peach
  Sc1_p1-->Sc1_p1_s1["[sentence]<br/>#quot;All legislative Powers…#quot;"]:::dark_green
  Sc1_p1_s1-->Sc1_p1_s1_w1["[word]<br/>#quot;All#quot;"]:::light_green
  Sc1_p1_s1-->Sc1_p1_s1_w2["[word]<br/>#quot;legislative#quot;"]:::light_green
  Sc1_p1_s1-->Sc1_p1_s1_w3["[word]<br/>#quot;Powers#quot;"]:::light_green
  Sc1_p1_s1-->Sc1_p1_s1_w4["..."]:::ellipsis

  A1--->Sc2["[section]<br/>#quot;Section 2#quot;"]:::light_brown;
  Sc2-->Sc2_k["[key]<br/>#quot;2#quot;"]:::light_brown
  Sc2--->Sc2_p1["[paragraph]<br/>#quot;The House of…#quot;"]:::peach
  Sc2_p1---->Sc2_p1_s1["[sentence]<br/>#quot;The House of…#quot;"]:::dark_green
  Sc2_p1_s1-->Sc2_p1_s1_w1["[word]<br/>#quot;The#quot;"]:::light_green
  Sc2_p1_s1-->Sc2_p1_s1_w2["[word]<br/>#quot;House#quot;"]:::light_green
  Sc2_p1_s1-->Sc2_p1_s1_w3["[word]<br/>#quot;of#quot;"]:::light_green
  Sc2_p1_s1-->Sc2_p1_s1_w4["..."]:::ellipsis
  Sc2--->Sc2_p2["[paragraph]<br/>#quot;No Person shall…#quot;"]:::peach
  Sc2_p2---->Sc2_p2_s1["[sentence]<br/>#quot;No Person shall…#quot;"]:::dark_green
  Sc2_p2_s1-->Sc2_p2_s1_w1["[word]<br/>#quot;No#quot;"]:::light_green
  Sc2_p2_s1-->Sc2_p2_s1_w2["[word]<br/>#quot;Person#quot;"]:::light_green
  Sc2_p2_s1-->Sc2_p2_s1_w3["[word]<br/>#quot;shall#quot;"]:::light_green
  Sc2_p2_s1-->Sc2_p2_s1_w4["..."]:::ellipsis

  classDef dark_brown fill:#533E30,stroke:#000000,color:#FFFFFF;
  classDef light_brown fill:#D2AC70,stroke:#000000,color:#000000;
  classDef peach fill:#E4D1AE,stroke:#000000,color:#000000;
  classDef dark_green fill:#517D3D,stroke:#000000,color:#FFFFFF;
  classDef light_green fill:#90C246,stroke:#000000,color:#FFFFFF;

  classDef ellipsis fill:#FFFFFF,stroke:#FFFFFF,color:#000000;You can then search your tree using plumule: a powerful structured query language:'**[d:section]{**[d:word] & [lcs:power,right]}'# Plumule query to find sections that containing words 'power' or 'right'Try outthis demoyourself, which shows how easy it is to parse, visualize, and query the US Constitution using Pawpaw.UsagePawpaw has extensive features and capabilities you can read about in theDocs.  As a quick example, say you have some text that would like to perform nlp-like segmentation on.>>>s='nine 9 ten 10 eleven 11 TWELVE 12 thirteen 13'You can use a regular expression for segmentation as follows:>>>importregex>>>re=regex.compile(r'(?:(?P<phrase>(?P<word>(?P<char>\w)+) (?P<number>(?P<digit>\d)+))\s*)+')You can then use this regex to feedPawpaw:>>>importpawpaw>>>doc=pawpaw.Ito.from_match(re.fullmatch(s))[0]With this single line of code, Pawpaw generates a fully hierarchical, tree of phrases, words, chars, numbers, and digits.  You can visualize the tree:>>>tree_vis=pawpaw.visualization.pepo.Tree()>>>print(tree_vis.dumps(doc))(0,45)'0':'nine 9 ten 10 eleven…ELVE 12 thirteen 13'├──(0,6)'phrase':'nine 9'│├──(0,4)'word':'nine'││├──(0,1)'char':'n'││├──(1,2)'char':'i'││├──(2,3)'char':'n'││└──(3,4)'char':'e'│└──(5,6)'number':'9'│└──(5,6)'digit':'9'├──(7,13)'phrase':'ten 10'│├──(7,10)'word':'ten'││├──(7,8)'char':'t'││├──(8,9)'char':'e'││└──(9,10)'char':'n'│└──(11,13)'number':'10'│├──(11,12)'digit':'1'│└──(12,13)'digit':'0'├──(14,23)'phrase':'eleven 11'│├──(14,20)'word':'eleven'││├──(14,15)'char':'e'││├──(15,16)'char':'l'││├──(16,17)'char':'e'││├──(17,18)'char':'v'││├──(18,19)'char':'e'││└──(19,20)'char':'n'│└──(21,23)'number':'11'│├──(21,22)'digit':'1'│└──(22,23)'digit':'1'├──(24,33)'phrase':'TWELVE 12'│├──(24,30)'word':'TWELVE'││├──(24,25)'char':'T'││├──(25,26)'char':'W'││├──(26,27)'char':'E'││├──(27,28)'char':'L'││├──(28,29)'char':'V'││└──(29,30)'char':'E'│└──(31,33)'number':'12'│├──(31,32)'digit':'1'│└──(32,33)'digit':'2'└──(34,45)'phrase':'thirteen 13'├──(34,42)'word':'thirteen'│├──(34,35)'char':'t'│├──(35,36)'char':'h'│├──(36,37)'char':'i'│├──(37,38)'char':'r'│├──(38,39)'char':'t'│├──(39,40)'char':'e'│├──(40,41)'char':'e'│└──(41,42)'char':'n'└──(43,45)'number':'13'├──(43,44)'digit':'1'└──(44,45)'digit':'3'And you can search the tree using Pawpaw'splumule, a powerful XPATH-like structured query language:>>>print(*doc.find_all('**[d:digit]'),sep=', ')# all digits9,1,0,1,1,1,2,1,3>>>print(*doc.find_all('**[d:number]{</*[s:i]}'),sep=', ')# all numbers with 'i' in their name9,13This example uses regular expressions as a source, however, Pawpaw is able to work with many other input types.  For example, you can use libraries such asNLTKto grow Pawpaw trees, or, you can use Pawpaw's included parser framework to build your own sophisticated parsers quickly and easily.(back to top)Getting StartedPrerequisitesPawpaw has been written and tested using Python 3.10.  The only dependency isregex, which will be fetched and installed automatically if you install Pawpaw
with pip or conda.Installation OptionsThere are lots of ways to install Pawpaw.  Versioned instances that have passed all automated tests are available fromPyPI:Install with pip from PyPI:pip install pawpawInstall with conda from PyPI:conda activate myenv
conda install git pip
pip install pawpawAlternatively, you can pull from the main branch at GitHub.  This will ensure that you have the latest code, however, the main branch can potentially have internal inconsistencies and/or failed tests:Install with pip from GitHub:pip install git+https://github.com/rlayers/pawpaw.gitInstall with conda from GitHub:conda activate myenv
conda install git pip
pip install git+https://github.com/rlayers/pawpaw.gitClone the repo with git from GitHub:git clone https://github.com/rlayers/pawpawVerify InstallationWhichever way you fetch Pawpaw, you can easily verify that it is installed correctly.  Just open Python prompt and type:>>>frompawpawimportIto>>>Ito('Hello, World!')Ito(span=(0,13),desc='',substr='Hello, World!')If your last line looks like this, you are up and running with Pawpaw!(back to top)History & RoadmapPawpaw is a rewrite ofdesponia, a now-deprecated Python 2.x segmentation framework that was itself based on a prior framework calledIto.  Currently in release-candidate status, many components and features are production ready.  However, documentation is still being written and some newer features are still undergoing work.  A rough outline of which components are finalized is as follows:arborformitoratorDescExtractReflectSplitValueFuncpostoratorStackedReduceWindowedJoincoreErrorsInfixItoItoChildrennucoSpanTypesdocumentation & examplesqueryradicle query engineplumulenlpvisualizationasciboxhighlighterpeposgrxmlXmlHelperXmlParser(back to top)ContributingContributions to Pawpaw aregreatly appreciated- please refer to thecontributing guildelinesfor details.(back to top)LicenseDistributed under the MIT License. SeeLICENSEfor more information.(back to top)ContactsRobert L. Ayers:a.nov.guy@gmail.com(back to top)ReferencesMatthew Barnett's regexNLTK(back to top)