Package: dedoc
DedocDedoc is an open universal system for converting documents to a unified output format.
It extracts a document’s logical structure and content, its tables, text formatting and metadata.
The document’s content is represented as a tree storing headings and lists of any level.
Dedoc can be integrated in a document contents and structure analysis system as a separate module.WorkflowWorkflow description is givenhereFeatures and advantagesDedoc is implemented in Python and works with semi-structured data formats (DOC/DOCX, ODT, XLS/XLSX, CSV, TXT, JSON) and none-structured data formats like images (PNG, JPG etc.), archives (ZIP, RAR etc.), PDF and HTML formats.
Document structure extraction is fully automatic regardless of input data type.
Metadata and text formatting are also extracted automatically.In 2022, the system won a grant to support the development of promising AI projects from theInnovation Assistance Foundation (Фонд содействия инновациям).Dedoc provides:Extensibility due to a flexible addition of new document formats and to an easy change of an output data format.Support for extracting document structure out of nested documents having different formats.Extracting various text formatting features (indentation, font type, size, style etc.).Working with documents of various origin (statements of work, legal documents, technical reports, scientific papers) allowing flexible tuning for new domains.Working with PDF documents containing a textual layer:Support to automatically determine the correctness of the textual layer in PDF documents;Extract containing and formatting from PDF-documents with a textual layer using the developed interpreter of the virtual stack machine for printing graphics according to the format specification.Extracting table data from DOC/DOCX, PDF, HTML, CSV and image formats:Recognizing a physical structure and a cell text for complex multipage tables having explicit borders with the help of contour analysis.Working with scanned documents (image formats and PDF without text layer):Using Tesseract, an actively developed OCR engine from Google, together with image preprocessing methods.Utilizing modern machine learning approaches for detecting a document orientation, detecting single/multicolumn document page, detecting bold text and extracting hierarchical structure based on the classification of features extracted from document images.ImpactThis project may be useful as a first step of automatic document analysis pipeline (e.g. before the NLP part).
Dedoc is in demand for information analytic systems, information leak monitoring systems, as well as for natural language processing systems.
The library is intended for application use by developers of systems for automatic analysis and structuring of electronic documents, including for further search in electronic documents.Online-DocumentationRelevant documentation of the dedoc is availablehereInstallation instructionsThis project has REST Api and you can run it in Docker container.
Also, dedoc can be installed as a library viapip.
There are two ways to install and run dedoc as a web application or a library that are described below.Install and run dedoc using dockerYou should havegitanddockerinstalled for running dedoc by this method.
This method is more flexible because it doesn't depend on the operating system and other user's limitations,
still, the docker application should be installed and configured properly.If you don't need to change the application configuration, you may use the built docker image as well.Work with dedoc as service1. Pull the imagedockerpulldedocproject/dedoc2. Run the containerdockerrun-p1231:1231--rmdedocproject/dedocpython3/dedoc_root/dedoc/main.pyGo todockerhubto get more information about available dedoc images.If you need to change some application settings, you may updateconfig.pyaccording to your needs and re-build the image.
You can build and run image:1. Clone the repositorygitclonehttps://github.com/ispras/dedoc2. Go to thededocdirectorycddedoc3. Build the image and run the applicationdocker-composeup--build4. Run container with teststest="true"docker-composeup--buildIf you need to change some application settings, you may updateconfig.pyaccording to your needs and re-build the image.Install dedoc using pipIf you don't want to use docker for running the application, it's possible to run dedoc locally.
However, it isn't suitable for any operating system (Ubuntu 20+is recommended) and
there may be not enough machine's resources for its work.
You should havepython(python3.8,python3.9are recommended) andpipinstalled.1. Install necessary packages:sudoapt-getinstall-ylibreofficedjvulibre-binunzipunrarlibreofficeanddjvulibre-binpackages are used by converters (doc, odt to docx; xls, ods to xlsx; ppt, odp to pptx; djvu to pdf).
If you don't need converters, you can skip this step.unzipandunrarpackages are used in the process of extracting archives.2. InstallTesseract OCR 5framework:You can try any tutorial for this purpose or lookhereto get the example of Tesseract installing for dedoc container or use next commands for building Tesseract OCR 5 from sources:2.1. Install compilers and libraries required by the Tesseract OCR:sudoapt-getupdate
sudoapt-getinstall-yautomakebinutils-devbuild-essentialca-certificatesclangg++g++-multilibgcc-multiliblibcairo2libffi-dev\libgdk-pixbuf2.0-0libglib2.0-devlibjpeg-devlibleptonica-devlibpango-1.0-0libpango1.0-devlibpangocairo-1.0-0libpng-devlibsm6\libtesseract-devlibtoollibxext6makepkg-configpoppler-utilspstotextshared-mime-infosoftware-properties-commonswigzlib1g-dev2.2. Build Tesseract from sources:sudoadd-apt-repository-yppa:alex-p/tesseract-ocr-devel
sudoapt-getupdate--allow-releaseinfo-change
sudoapt-getinstall-ytesseract-ocrtesseract-ocr-rus
gitclone--depth1--branch5.0.0-beta-20210916https://github.com/tesseract-ocr/tesseract/cdtesseract&&./autogen.sh&&sudo./configure&&sudomake&&sudomakeinstall&&sudoldconfig&&cd..exportTESSDATA_PREFIX=/usr/share/tesseract-ocr/5/tessdata/Install the dedoc library via pip.You needtorch~=1.11.0andtorchvision~=0.12.0installed. If you already have torch and torchvision in your environment:pipinstalldedocOr you can install dedoc with torch and torchvision included:pipinstall"dedoc[torch]"Install and run dedoc from sourcesIf you want to run dedoc as a service from sources, it's possible to run dedoc locally.
However, it is suitable not for all operating systems (Ubuntu 20+is recommended) and
there may be not enough machine's resources for its work.
You should havepython(python3.8,python3.9are recommended) andpipinstalled.1. Install necessary packages: according to instructionsinstall necessary packages2. Build Tesseract from sources according to instructionsInstall Tesseract OCR-5 framework3. We recommend to install python's virtual environment (for example, viavirtualenvwrapper)Below are the instructions for installing the packagevirtualenvwrapper:sudopip3installvirtualenvvirtualenvwrapper
mkdir~/.virtualenvsexportWORKON_HOME=~/.virtualenvsecho"export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3.8">>~/.bashrcecho". /usr/local/bin/virtualenvwrapper.sh">>~/.bashrcsource~/.bashrc
mkvirtualenvdedoc_env4. Install python's requirements and launch dedoc service on default port1231:# clone dedoc projectgitclonehttps://github.com/ispras/dedoc.gitcddedoc# check on your's python environmentworkondedoc_envexportPYTHONPATH=$PYTHONPATH:$(pwd)pipinstall-rrequirements.txt
pipinstalltorch=1.11.0torchvision==0.12.0-fhttps://download.pytorch.org/whl/torch_stable.html
pythondedoc/main.py-c./dedoc/config.pyNow you can go to thelocalhost:1231and look at the docs and examples.Option: You can change the port of service:You need to change environmentDOCREADER_PORTFor local service launching onyour_port(e.g.1166). Install (installation instruction) and launch with environment:DOCREADER_PORT=1166pythondedoc/main.py-c./dedoc/config.pyFor service launching in docker-container you need to change port value inDOCREADER_PORTenv and fieldportsindocker-compose.ymlfile:...dedoc:...ports:-your_port_number:your_port_numberenvironment:DOCREADER_PORT:your_port_number...test:...environment:DOCREADER_PORT:your_port_numberGohereto get more details about dedoc installation.