Package: mmda
MMDA - multimodal document analysisThis is work in progress...Setupcondacreate-nmmdapython=3.8
pipinstall-e'.[dev,<extras_require section from setup.py>]'For most users, we recommend using recipes:pipinstall-e'.[dev,recipes]'Unit testingNote that pytest is running coverage, which checks the unit test coverage of the code.
The percent coverage can be found in setup.cfg file.pytestfor latest failed testpytest--lf--no-cov-n0for specific test name of class namepytest-k'TestFigureCaptionPredictor'--no-cov-n0Quick start1. Create a Document for the first time from a PDFIn this example, we use theCoreRecipeto convert a PDF into a bunch of text and images.frommmda.typesimportDocumentfrommmda.recipesimportCoreReciperecipe=CoreRecipe()doc:Document=recipe.from_path(pdfpath='...pdf')2. Understanding the output: theDocumentclassWhat is aDocument? At minimum, it is some text, saved under the.symbolsfield, which is just a<str>.  For example:doc.symbols>"Language Models as Knowledge Bases?\nFabio Petroni1 Tim Rockt..."But the usefulness of this library really is when you have multiple different ways of segmenting.symbols. For example, segmenting the paper into Pages, and then each page into Rows:forpageindoc.pages:print(f'\n=== PAGE:{page.id}===\n\n')forrowinpage.rows:print(row.symbols)>...>===PAGE:5===>['tence x, s′ will be linked to s and o′ to o. In']>['practice, this means RE can return the correct so-']>['lution o if any relation instance of the right type']>['was extracted from x, regardless of whether it has']>...shows two nice aspects of this library:Documentprovides iterables for different segmentations ofsymbols.  Options include things likepages, tokens, rows, sents, paragraphs, sections, ....  Not every Parser will provide every segmentation, though.  For example,SymbolScraperParseronly providespages, tokens, rows.  More on how to obtain other segmentations later.Each one of these segments (in our library, we call themSpanGroupobjects) is aware of (and can access) other segment types. For example, you can callpage.rowsto get all Rows that intersect a particular Page.  Or you can callsent.tokensto get all Tokens that intersect a particular Sentence.  Or you can callsent.rowsto get the Row(s) that intersect a particular Sentence.  These indexes are builtdynamicallywhen theDocumentis created and each time a newSpanGrouptype is loaded.  In the extreme, one can do:forpageindoc.pages:forparagraphinpage.paragraphs:forsentinparagraph.sents:forrowinsent.rows:...as long as those fields are available in the Document. You can check which fields are available in a Document via:doc.fields>['pages','tokens','rows']3. Understanding intersection of SpanGroupsNote thatSpanGroupdon't necessarily perfectly nest each other. For example, what happens if:forsentindoc.sents:forrowinsent.rows:print([token.symbolsfortokeninrow.tokens])Tokens that areoutsideeach sentence can still be printed. This is because when we jump from a sentence to its rows, we are looking forallrows that haveanyoverlap with the sentence. Rows can extend beyond sentence boundaries, and as such, can contain tokens outside that sentence.Here's another example:forpageindoc.pages:print([sent.symbolsforsentinpage.sents])Sentences can cross page boundaries. As such, adjacent pages may end up printing the same sentence.Butforpageindoc.pages:print([row.symbolsforrowinpage.rows])print([token.symbolsfortokeninpage.tokens])rows and tokens adhere strictly to page boundaries, and thus will not repeat when printed across pages.A key aspect of using this library is understanding how these different fields are defined & anticipating how they might interact with each other. We try to make decisions that are intuitive, but we do ask users to experiment with fields to build up familiarity.4. What's in aSpanGroup?EachSpanGroupobject stores information about its contents and position:.spans: List[Span], ASpanis a pointer intoDocument.symbols(that is,Span(start=0, end=5)corresponds tosymbols[0:5]) and a singleBoxrepresenting its position & rectangular region on the page..box_group: BoxGroup, ABoxGroupobject stores.boxes: List[Box]..metadata: Metadata, A free form dictionary-like object to store extra metadata about thatSpanGroup. These are usually empty.5. How can I manually create my ownDocument?If you look at what is happening inCoreRecipe, it's basically stitching together 3 types of tools:Parsers,RasterizersandPredictors.Parserstake a PDF as input and return aDocumentcompared of.symbolsand other fields. The example one we use is a wrapper aroundPDFPlumber- MIT License utility.Rasterizerstake a PDF as input and return anImageper page that is added toDocument.images. The example one we use isPDF2Image- MIT License.Predictorstake aDocumentand apply some operation to compute a new set ofSpanGroupobjects that we can insert into ourDocument. These are all built in-house and can be either simple heuristics or full machine-learning models.If we look at howCoreRecipeis implemented, what's happening in.from_path()is:def from_path(self, pdfpath: str) -> Document:
        logger.info("Parsing document...")
        doc = self.parser.parse(input_pdf_path=pdfpath)

        logger.info("Rasterizing document...")
        images = self.rasterizer.rasterize(input_pdf_path=pdfpath, dpi=72)
        doc.annotate_images(images=images)

        logger.info("Predicting words...")
        words = self.word_predictor.predict(document=doc)
        doc.annotate(words=words)

        logger.info("Predicting blocks...")
        blocks = self.effdet_publaynet_predictor.predict(document=doc)
        equations = self.effdet_mfd_predictor.predict(document=doc)
        doc.annotate(blocks=blocks + equations)

        logger.info("Predicting vila...")
        vila_span_groups = self.vila_predictor.predict(document=doc)
        doc.annotate(vila_span_groups=vila_span_groups)

        return docYou can see how theDocumentis first created using theParser, thenImagesare added to theDocumentby using theRasterizerand.annotate_images()method. Then we layer on multiplePredicorsworth of predictions, each added to theDocumentusing.annotate().6. How can I save myDocument?importjsonwithopen('filename.json','w')asf_out:json.dump(doc.to_json(with_images=True),f_out,indent=4)will produce something akin to:{"symbols":"Language Models as Knowledge Bases?\nFabio Petroni1 Tim Rockt...","images":"...","rows":[...],"tokens":[...],"words":[...],"blocks":[...],"vila_span_groups":[...]}Note thatImagesare serialized tobase64if you includewith_imagesflag. Otherwise, it's left out of JSON serialization by default.7. How can I load myDocument?These can be used to reconstruct aDocumentagain via:withopen('filename.json')asf_in:doc_dict=json.load(f_in)doc=Document.from_json(doc_dict)