Package: amazon-textract-response-parser
Textract Response ParserYou can use Textract response parser library to easily parser JSON returned by Amazon Textract. Library parses JSON and provides programming language specific constructs to work with different parts of the document.textractoris an example of PoC batch processing tool that takes advantage of Textract response parser library and generate output in multiple formats.Installationpython -m pip install amazon-textract-response-parserPipeline and Serializer/DeserializerSerializer/DeserializerBased on themarshmallowframework, the serializer/deserializer allows for creating an object represenation of the Textract JSON response.Deserialize Textract JSON# j holds the Textract JSON dictfromtrp.trp2importTDocument,TDocumentSchemat_doc=TDocumentSchema().load(j)Serialize Textractfromtrp.trp2importTDocument,TDocumentSchemat_doc=TDocumentSchema().dump(t_doc)Deserialize Textract AnalyzeId JSON# j holds the Textract JSONfromtrp.trp2_analyzeidimportTAnalyzeIdDocument,TAnalyzeIdDocumentSchemat_doc=TAnalyzeIdDocumentSchema().load(json.loads(j))Serialize Textract AnalyzeId object to JSONfromtrp.trp2_analyzeidimportTAnalyzeIdDocument,TAnalyzeIdDocumentSchemat_doc=TAnalyzeIdDocumentSchema().dump(t_doc)PipelineWe added some commonly requested features as easily consumable components that modify the Textract JSON Schema and ideally don't require big changes to any  existing workflow.Order blocks (WORDS, LINES, TABLE, KEY_VALUE_SET) by geometry y-axisBy default Textract does not put the elements identified in an order in the JSON response.The sample implementationorder_blocks_by_geoof a function using the Serializer/Deserializer shows how to change the structure and order the elements while maintaining the schema. This way no change is necessary to integrate with existing processing.# the sample code below makes use of the amazon-textract-callerpython-mpipinstallamazon-textract-callerfromtextractcaller.t_callimportcall_textract,Textract_Featuresfromtrp.trp2importTDocument,TDocumentSchemafromtrp.t_pipelineimportorder_blocks_by_geoimporttrpimportjsonj=call_textract(input_document="path_to_some_document (PDF, JPEG, PNG)",features=[Textract_Features.FORMS,Textract_Features.TABLES])# the t_doc will be not orderedt_doc=TDocumentSchema().load(j)# the ordered_doc has elements ordered by y-coordinate (top to bottom of page)ordered_doc=order_blocks_by_geo(t_doc)# send to trp for further processing logictrp_doc=trp.Document(TDocumentSchema().dump(ordered_doc))Page orientation in degreesAmazon Textract supports all in-plane document rotations. However the response does not include a single number for the degree, but instead each word and line does have polygon points which can be used to calculate the degree of rotation. The following code adds this information as a custom field to Amazon Textract JSON response.fromtrp.t_pipelineimportadd_page_orientationimporttrp.trp2ast2importtrpast1# assign the Textract JSON dict to jj=<call_textract(input_document="path_to_some_document (PDF, JPEG, PNG)")oryourJSONdict>t_document:t2.TDocument=t2.TDocumentSchema().load(j)t_document=add_page_orientation(t_document)doc=t1.Document(t2.TDocumentSchema().dump(t_document))# page orientation can be read now for each pageforpageindoc.pages:print(page.custom['PageOrientationBasedOnWords'])Using the pipeline on command lineThe amazon-textract-response-parser package also includes a command line tool to test pipeline components like the add_page_orientation or the order_blocks_by_geo.Here is one example of the usage (in combination with theamazon-textractcommand from amazon-textract-helper and thejqtool (https://stedolan.github.io/jq/))>amazon-textract--input-document"s3://somebucket/some-multi-page-pdf.pdf"|amazon-textract-pipeline--componentsadd_page_orientation|jq'.Blocks[] | select(.BlockType=="PAGE") | .Custom'm{"Orientation":7}{"Orientation":11}...{"Orientation":-7}{"Orientation":0}Merge or link tables across pagesSometimes tables start on one page and continue across the next page or pages. This component identifies if that is the case based on the number of columns and if a header is present on the subsequent table and can modify the output Textract JSON schema for down-stream processing. Other custom-logic is possible to develop for specific use cases.The MergeOptions.MERGE combines the tables and makes them appear as one for post processing, with the drawback that the geometry information is not accuracy any longer. So overlaying with bounding boxes will not be accuracy.The MergeOptions.LINK maintains the geometric structure and enriches the table information with links between the table elements. There is a custom['previus_table'] and custom['next_table'] attribute added to the TABLE blocks in the Textract JSON schema.Usage is simplefromtrp.t_pipelineimportpipeline_merge_tablesimporttrp.trp2ast2j=<call_textract(input_document="path_to_some_document (PDF, JPEG, PNG)")oryourJSONdict>t_document:t2.TDocument=t2.TDocumentSchema().load(j)t_document=pipeline_merge_tables(t_document,MergeOptions.MERGE,None,HeaderFooterType.NONE)Using from command line example# from the root of the repositorycatsrc-python/tests/data/gib_multi_page_table_merge.json|amazon-textract-pipeline--componentsmerge_tables|amazon-textract--stdin--pretty-printTABLES# compare to cat src-python/tests/data/gib_multi_page_table_merge.json | amazon-textract --stdin --pretty-print TABLESAdd OCR confidence score to KEY and VALUEIt can be useful for some use cases to validate the confidence score for a given KEY or the VALUE from an Analyze action with FORMS feature result.The Confidence property of a BlockType 'KEY_VALUE_SET' expresses the confidence in this particular prediction being a KEY or a VALUE, but not the confidence of the underlying text value.Simplified example:{"Confidence":95.5,"Geometry":{<...>},"Id":"v1","Relationships":[{"Type":"CHILD","Ids":["c1"]}],"EntityTypes":["VALUE"],"BlockType":"KEY_VALUE_SET"},{"Confidence":99.2610092163086,"TextType":"PRINTED","Geometry":{<...>},"Id":"c1","Text":"2021-Apr-08","BlockType":"WORD"},In this example the confidence in the prediction of the VALUE to be an actual value in a key/value relationship is 95.5.The confidence in the actual text representation is 99.2610092163086.
For simplicity in this example the value consists of just one word, but is not limited to that and could contain multiple words.The KV_OCR_Confidence pipeline component adds confidence scores for the underlying OCR to the JSON. After executing the example JSON will look like this:{"Confidence":95.5,"Geometry":{<...>},"Id":"v1","Relationships":[{"Type":"CHILD","Ids":["c1"]}],"EntityTypes":["VALUE"],"BlockType":"KEY_VALUE_SET","Custom":{"OCRConfidence":{"mean":99.2610092163086,"min":99.2610092163086}}},{"Confidence":99.2610092163086,"TextType":"PRINTED","Geometry":{<...>},"Id":"c1","Text":"2021-Apr-08","BlockType":"WORD"},Usage is simplefromtrp.t_pipelineimportadd_kv_ocr_confidenceimporttrp.trp2ast2j=<call_textract(input_document="path_to_some_document (PDF, JPEG, PNG)")oryourJSONdict>t_document:t2.TDocument=t2.TDocumentSchema().load(j)t_document=add_kv_ocr_confidence(t_document)# further processingUsing from command line example and validating the output:# from the root of the repositorycat"src-python/tests/data/employment-application.json"|amazon-textract-pipeline--componentskv_ocr_confidence|jq'.Blocks[] | select(.BlockType=="KEY_VALUE_SET") 'Parse JSON response from TextractfromtrpimportDocumentdoc=Document(response)# Iterate over elements in the documentforpageindoc.pages:# Print lines and wordsforlineinpage.lines:print("Line:{}--{}".format(line.text,line.confidence))forwordinline.words:print("Word:{}--{}".format(word.text,word.confidence))# Print tablesfortableinpage.tables:forr,rowinenumerate(table.rows):forc,cellinenumerate(row.cells):print("Table[{}][{}] ={}-{}".format(r,c,cell.text,cell.confidence))# Print fieldsforfieldinpage.form.fields:print("Field: Key:{}, Value:{}".format(field.key.text,field.value.text))# Get field by keykey="Phone Number:"field=page.form.getFieldByKey(key)if(field):print("Field: Key:{}, Value:{}".format(field.key,field.value))# Search fields by keykey="address"fields=page.form.searchFieldsByKey(key)forfieldinfields:print("Field: Key:{}, Value:{}".format(field.key,field.value))TestClone the repo and run pytestgitclonehttps://github.com/aws-samples/amazon-textract-response-parser.gitcdamazon-textract-response-parser
python-mvenvvirtualenv
virtualenv/bin/activate

python-mpipinstallpip--upgrade
python-mpipinstallpytest
python-mpipinstallsetuptools
python-mpipinstalltabulate
pythonsrc-python\setup.pyinstall
pytestOther ResourcesLarge scale document processing with Amazon Textract - Reference ArchitectureBatch processing toolCode samplesLicense SummaryThis sample code is made available under the Apache License Version 2.0. See the LICENSE file.