Package: post-processor-pi
post-processor-piis a Python module that exploits the output of Document AI (tool available
on GCP) to create a JSON file that stores the text contained in a document in an organized way, so
that it reflects its original structure. To work with different types of
documents having different structures, post-processor-pi has to be configured using the GUI
that I developed.InstallationThe easiest way to install post-processor-pi is using pip:pip install post-processor-piDocumentationAs I mentioned above in order to implement the post-processor we need to configure it first and this
can be done using the Configuration GUI that I developed.Configuration GUIIn order to start the Configuration GUI we need an example corresponding to the output of Document
AI for one document. Below you can find the code example that enables you to start the Configuration
GUI.fromdai_post_processorimportpost_processoraspp# Open one example from Document AI outputpath="your/path/to/DocumentAI/example"doc_ai=pp.open_doc_ai(path)# Start Configuration GUImywin=pp.configGUI(doc_ai)mywin.start()After running the above instructions the GUI should open.Configuration GUI TuningThe Configuration GUI consists of two windows:Main: here you can find the essential instructions that are needed to
configure post-processor-pi.Select text of interest: in this section you can select the portion
of the pages in the documents you're interested in. In thePage numberfield you can provide the page number of the Document example you want to open.
I you press theDraw Boxbutton a new window will open, and here you can
select the portion of the page drawing a rectangle. All the text outside this
rectangle will be ignored for all pages and documents. Once you performed the
selection you can close the interactive window. Below you have a demo.Structuring Filter: in this section you can specify the first and second
levelpointsthat give structure to the document. In theMain Structurefield you need to
specify the regex rule that lets you find all the first levelpoints. If
you press theAdd Pointbutton an additional field, where you can specify
the regex rule for the second levelpoints, will appear.Output: in this section you can specify what to include in the JSON output
of the Post Processor. You can choose to include the list of all lines and
paragraphs in the document and the structured text. You can also choose to
ignore those lines that are recognised asHeaderswith the Filter Headers
checkbox.Advanced: here you can perform some more advanced configurations:Line threshold: helpful for reconstructing lines in the document when
Document AI fails to do so. If the difference between the y top coordinates
of a line and the previous one is less thanLine Thresholdin absolute
value, these two lines are merged together to form a newreconstructed line.Line repeat: specify how many lines from the Document AI output are
needed to build a singlereconstructed lineParagraph Multiplier: helpful for reconstructing paragraphs in the
document when  Document AI fails to do so.
Every line in the document that has a vertical
gap with respect to the previous line lower than the median vertical gap of
all lines in the document multiplied by theParagraph Multiplier, will be
appended to the previous line to form a paragraph.Header Multiplier: has a similar role of the Paragraph Multiplier, but
it's useful for finding the centered headers in the document.Once you're happy with your changes you can press theSavebutton and a
configuration file will be created and saved as data/config.jsonExampleIn this example I'm going to show how to implement post-processor-pi on the
output of Document AI. The example document is a World Bank Loan Agreement and has
this kind of structure.The first level points are the dotted numbers such as2.07and the second
level points are the letters between round brackets such as(a). Below you
can find the code implementation that lets create the structured JSON using the
config.json file that we created using the GUI.fromdai_post_processorimportpost_processorasppimportjson# Open one example from Document AI outputpath="your/path/to/DocumentAI/example"doc_ai=pp.open_doc_ai(path)# Open config.json filewithopen("data/config.json")asjson_file:config_dict=json.load(json_file)# Apply the Post Processor to the Document AI outputdoc_class=pp.DocumentClass(doc_ai,config_dict)structured_doc=doc_class.create_json(file_name="example")After running the above code the structured JSON will be saved as
data/pp_output/example.json. Below you can find a snippet of example.json, in
particular thestructured_outputsection. As you can see the JSON file reflects
the original structure of the document.