Package: nlm-ingestor
AboutThis repo provides the service code forllmsherpaAPI to connect.
This repo contains custom RAG (retrieval augmented generation) friendly parsers for the following file formats:PDFThe PDF parser is a rule based parser which uses text co-ordinates (boundary box), graphics and font data from nlmatics modified version of tika found herehttps://github.com/nlmatics/nlm-tika.
The PDF parser works off text layer and also offers a OCR option (apply_ocr) to automatically use OCR if there are scanned pages in your PDFs. The OCR feature is based off a nlmatics modified version of tika which uses tesseract underneath.
Check out the notebookpdf_visual_ingestor_step_by_stepto experiment directly with the PDF parser.The PDF Parser offers the following features:Sections and subsections along with their levels.Paragraphs - combines lines.Links between sections and paragraphs.Tables along with the section the tables are found in.Lists and nested lists.Join content spread across pages.Removal of repeating headers and footers.Watermark removal.OCR with boundary boxesHTMLA special HTML parser that creates layout aware blocks to make RAG performance better with higher quality chunks.TextA special text parser which tries to figure out lists, tables, headers etc. purely by looking at the text and no visual, font or bbox information.DOCX, PPTX and any other format supported by Apache TikaThere are two ways to process these types of documentshtml output from tika for these file types is used and parsed by the html parserInstallation steps:Run each step directlyInstall latest version of java fromhttps://www.oracle.com/java/technologies/downloads/Run the tika server:java -jar <path_to_nlm_ingestor>/jars/tika-server-standard-nlm-modified-2.4.1_v6.jarInstall the ingestor!pip install nlm-ingestorRun the ingestorpython -m nlm_ingestor.ingestion_daemonRun the docker fileA docker image is available via public github container registry.Pull the docker imagedocker pull ghcr.io/nlmatics/nlm-ingestor:latestRun the docker image mapping the port 5001 to port of your choice.docker run -p 5010:5001 ghcr.io/nlmatics/nlm-ingestor:latest-<version>Once you have the server running, you can use thellmsherpaAPI library to get chunks and use them for your LLM projects. Your llmsherpa_url will be:
"http://localhost:5010/api/parseDocument?renderFormat=all"to apply OCR add &applyOcr=yesto use the new indent parser which uses a different alogrithm to assign header levels, add &useNewIndentParser=yesthis server is good for your development - in production it is recommended to run this behind a secure gateway using nginx or cloud gatewaysTest the ingestor serverSample test code to test the server with llmsherpa parser is in thisnotebook.Rule based parser vs model based parserOver the course of 4 years, nlmatics team evaluated a variety of options including a yolo based vision parser developed by Tom Liu and Yi Zhang. Ultimately, we settled with the rule based parser due to the following reasons.It is substantially (100x) faster compared to any vision parser as bare miniumum you have to create images out of all pages of a PDF (even for the ones with text layer) to use a vision parser. It is our opinion that vision parser is a better option for OCRd PDF without a text layer, or for small PDF files consisting form like data, but for larger text layer PDFs, spanning hundreds of pages, a rule based parser like ours is more practical.No special hardware is needed to run this parser if you are not using the PDF OCR feature. You can run this with hardware from early 2000s!We found vision parser (or any parser for that matter including this) to be error prone and the solution to fix errors in a model were not pretty:Add more examples to your training set which may make the accuracy of the model from previous learning degrade and result in uncertainty in previously working codeUsing rule based ideas to fix model based parser issue gets us back to writing a lot of rules again.CreditsThe PDFparservisual_ingestorandnew_indent_parserwas written by Ambika Sukla with additional contributions from Reshav Abraham who wrote the initial code to modify tika, Tom Liu who wrote the original Indent Parser and Kiran Panicker who made several improvements to the parsing speed, table parsing accuracy, indent parsing accuracy and reordering accuracy.TheHTML Ingestorwas written by Tom Liu.TheMarkdown Parserwas written by Yi Zhang.TheText Ingestorwas written by Reshav Abraham.TheXML Ingestorwas written by Ambika Sukla primarily to process PubMed XMLs.Theline_parserwhich serves as a core sentence processing utility for all the other parsers was written by Ambika Sukla.Also we are thankful to the Apache PDFBox and Tika developer community for their years of work in providing the base for the PDF Parser.Nlm Modified TikaNlm modified version of Tika can be found in the 2.4.1-nlm branch herehttps://github.com/nlmatics/nlm-tika/tree/2.4.1-nlmFor convenience, a compiled jar file of the code is included in this repo in jars/ folder.
In some cases, your PDFs may result in errors in the Java server and you will need to modify the code there to resolve the issue and recompile the jar file.The following files are changed:https://github.com/nlmatics/nlm-tika/blob/2.4.1-nlm/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/PDF2XHTML.javahttps://github.com/nlmatics/nlm-tika/blob/2.4.1-nlm/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/AbstractPDF2XHTML.javaThe above is to add font and co-ordinates to every text element. It also removes watermarks.https://github.com/nlmatics/nlm-tika/blob/2.4.1-nlm/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/GraphicsStreamProcessor.javaThe above is to add lines and rectangles that can potentially help with table detection.To see the impact of these changes, see the first part of the notebook here:https://github.com/nlmatics/nlm-ingestor/blob/main/notebooks/pdf_visual_ingestor_step_by_step.ipynbSome ideas for future work:Make the changes independent of tika by writing own wrapper over pdfboxUpgrade to latest version of tikaCleanup the format of returned html to make it more css friendly