Package: ipypdf
iPyPDFA Jupyter-based tool to help parse out structured text from a PDF document and explore the contents.InstallationWindows Installerhttps://drive.google.com/drive/folders/1wmQisECMor04dgv9ZXFc07zq6zcHuija?usp=sharingThis will make a start-menu shortcut called "iPyPDF" which will open up the notebook for parsing documents.From SourceClone this repoInstall Anaconda or Miniconda if you do not already have itInstall mambaconda install mambaSolving the environment is impossibly slow without mambaCreate the environment and install ipypdf from sourcemambaenvcreate-fenvironment.yml-penv/ipypdf
condaactivateenv/ipypdf
pipinstall-e.Note: you can replace "mamba" with "conda" if you don't have mamba installed. It will just take longer to solve the environment.From pipCreate a conda environment withTesseractandJupyterlabcondacreate-nipypdfjupyterlabtesseract-cconda-forge`condaactivateipypdf
pipinstallipypdfGet a spacy model (the previous method accomplishes this automatically in theenvironment.ymlfile)python -m spacy download en_core_web_smOrconda install spacy-model-en_core_web_sm -c conda-forgeUsageipypdf is built for jupyter lab but should also work in jupyter notebooks.Launch jupyter lab withjupyter labfromipypdfimportAppapp=App("path/to/your/pdfs",bulk_render=False)appseenotebooksfor additional infoDevelopmentseeDEVELOPMENT.mdCommon IssuesAutoTools widget keeps saying layoutparser is not installedThis is usually a problem with pywin32.Tryconda install pywin32Also make sure that numpy is <1.19.3FeaturesWithin the GUI are 3 panels, Table of Contents, PDF viewer, and Tools.
In this section we are going over all of the various options available in the tools
panel.Auto-ToolsThis tab contains tools which will iterate through each page of the pdf.Text Only: Runs each page throughTesseractto obtain plain text.Parse Layout: Useslayoutparserto label portions of the document as either (title, text, image, or table). The sections
are then assembled together using a few simple rules in order to appoximate a shallow content hierarchy.Title and Text blocks are cropped out and sent through Tesseract to obtain the text.Tables are processed using a rule-based table parsing scheme describedhere.Image blocks have no additional processing.Notice that section 3 is missing. The process is not perfect. In this case, a section title was mislabled by layoutparser as standard text. Mistakes like this are fairly common. To correct them, you can edit the table of contents using the arrow keys (the cursor must be hovering over the table of contents).Table ParsingCytoscapeFolders,PDF Documents, andSectionshave a tab labeledCytoscape. This runs a tfidf similarity calculation over all nodes beneath the selected item. I.e. if you select the root node, then all defined nodes will be included in the calculation. However, only those with a link to another node will be drawn (this is for speed, may change this in the future).The color of each node denotes the pdf document it originated from.Selecting a node in the graph will highlight the node in theDocTree. Clicking the node in theDocTreewill render the first page of the node.SpacyExtracts named entities from the selected branch of the document tree. I.e.,
the raw text is compiled from a depth first search on whichever node is selected
in the table of contents. Then,spacy.nlp(text).entsreturns the named entities
found within the section.Digitizing UtilitiesI recommend turning offShow Boxesas this changes pages every time you add a node (working on a better solution)Each node has a specific set of tools available to use. Here are the tools provided when aSectionnode is selected.
Starting from the left:Add Section Nodeadds a sub-node of typeSectionand selects itAdd Text Nodeadds a sub-node of typeTextand selects itAdd Image Node...Delete NodeDelete the selected node and all of its childrenContent SelectorContent is extracted from the rendered image. Text is extracted using Optical Character Recognition (OCR). Images don't do any image analysis, they just denote coordinates and page number so that they can be retreived later if need be.When aSectionnode is selected, the selection tool will attempt to parse text from the portion of the page selected by the user. This text willoverwritethe label assigned to the node.When aTextnode is selected, the selection tool will attempt to parse text from the selected area andappendit to the node's content. This is because text blocks are not always perfectly rectangular, and often span multiple pages.When anImagenode is selected, the coordinates of the box are appended to the node's content.Save ButtonThis will generatejsonfiles for each document. When the tool is initialized, these are used to reconstruct the table of contents. You can also use the json file directly.