Package: dcr-core
DCR-CORE - Document Content Recognition API - READMEBased on the paper "Unfolding the Structure of a Document using Deep Learning" (Rahman and Finin, 2019), this software project aims to use various software techniques to automatically detect the structure in arbitrarypdfdocuments and thus make these documents more searchable.The computer linguistic methods used here assume that the documents to be processed are inpdfformat.
However, in order to be flexible in the selection of documents with respect to file format,DCR-COREincludes a sophisticated preprocessor mechanism that can convert many of the nonpdfformats topdfformat.From the documents inpdfformat, the next steps extract the text with the relevant metadata word by word, line by line, or page by page. In line-by-line extraction, an attempt is made to classify the individual lines and mark them accordingly, so that these line classifications can later be taken into account in token generation.In the currently last step qualified tokens can be generated, which contain on the one hand information about the localization of the token in the document and on the other hand token classification features like lemma, form, normalization etc..Please see theDocumentationfor more detailed information.1. Features1.1 GeneralSupport for documents in different languages - English as standard.1.2 PreprocessorIdentification of scannedpdfdocuments withPyMuPDF.Conversion of the scannedpdfdocuments into a set ofjpegorpngfiles withpdf2imageandPoppler.Conversion of the documents of typebmp,gif,jp2,jpeg,png,pnm,tif,tifforwebptopdfformat withTesseract OCR.Conversion ofcsv,docx,epub,html,odt,rstorrtftype documents topdfformat withPandocandTeX Live.1.3 Natural Language Processing (NLP)Extract text and metadata frompdfdocuments withPDFlib TET.Classification of lines in the document, e.g. body, footer, header lines, etc.Sentence-by-sentence determination of the token structure usingspaCy.Storage of the analysis results in JSON and XML flat files.2 Quick startIn addition to Python, the following software packages are required to useDCR-CORE:PDFlib TETPandocPopplerTeX LiveTesseract OCRNow, to avoid this installation effort, we recommend using the Docker image provided in DockerHubsee here.2.1 Docker Container AdministrationCreating and running a new container (Assuming the path prefix for the local data directory mapping is d:/TempMan):`docker run -it --name dcr-core -v d:/TempMan:/dcr-core/data/inbox_prod konnexionsgmbh/dcr-core:0.9.7`Restarting the container:docker start dcr-coreCheck the container is running:docker psTo access a running container:docker attach --detach-keys="ctrl-a" dcr-coreStopping a running container:docker stop dcr-core2.2 Docker Container UsageStarting Python in the Virtual Environment (inside thedcr-corecontainer):python3 -m pipenv run python3Make thedcr_coremodule available:from dcr_core import cls_processCreate an instance of theProcessclass:process = cls_process.Process()Process document files:process.document("data/inbox_prod/<file name>")3. Directory and File Structure of this Repository3.1 DirectoriesDirectoryContent.github/workflowsGitHub Actionworkflows.dataExample rule files for document line classification.docsDCR-COREdocumentation files.scriptsUbuntu and Windows Script for running the applicationsrcPython scripts andPDFlib TETfilestestsScripts and data for pytest.3.2 FilesFileFunctionality.gitignoreConfiguration of files and folders to be ignored..pylintrcConfiguration file forpylint.LICENSEText of the licence terms.logging_cfg.yamlConfiguration of the Logger functionality.MakefileDefinition of tasks to be excuted with themakecommand.MANIFEST.inSource distribution commands forPyPA.mkdocs.ymlConfiguration file forMkDocs.PipfileDefinition of the Python package requirements.Pipfile.lockDefinition of the specific versions of the Python packages.pyproject.tomlBuild system requirements according toPEP 518.README.mdThis file.setup.cfgSetup configuration file -see here.setup.cfg.referenceOriginal setup configuration file.4. SupportIf you need help withDCR-CORE, do not hesitate to get in contact with us!For questions and high-level discussions, useDiscussionson GitHub.To report a bug or make a feature request, open anIssueon GitHub.Please note that we may only provide support for problems / questions regarding core features ofDCR-CORE.
Any questions or bug reports about features of third-party themes, plugins, extensions or similar should be made to their respective projects.
But, such questions arenotbanned from theDiscussions.Make sure to stick around to answer some questions as well!5. LinksOfficial DocumentationRelease NotesDiscussions(Third-party themes, recipes, plugins and more)6. Contributing to DCR-CORETheDCR-COREproject welcomes, and depends on, contributions from developers and users in the open source community.
Please see theContributing Guidefor
information on how you can help.7. Code of ConductEveryone who interacts in theDCR-COREproject's codebase, issue trackers, and discussion forums is expected to follow theCode of Conduct.8. LicenseKonnexions Public License (KX-PL)