Package: ocrd-cis
Content:ocrd_cisIntroductionInstallationProfilerUsageocrd-cis-postcorrectocrd-cis-alignocrd-cis-dataTraininingocrd-cis-ocropy-trainocrd-cis-ocropy-clipocrd-cis-ocropy-resegmentocrd-cis-ocropy-segmentocrd-cis-ocropy-deskewocrd-cis-ocropy-denoiseocrd-cis-ocropy-binarizeocrd-cis-ocropy-dewarpocrd-cis-ocropy-recognizeTesserocrWorkflow configurationTestingMiscellaneousOCR-D workspaceOCR-D linksocrd_cisCISOCR-Dcommand line
tools for the automatic post-correction of OCR-results.Introductionocrd_ciscontains different tools for the automatic post-correction
of OCR results.  It contains tools for the training, evaluation and
execution of the post-correction.  Most of the tools are following theOCR-D CLI conventions.Additionally, there is a helper tool to align multiple OCR results,
as well as an improved version ofOcropythat works with Python 3 and is also wrapped forOCR-D.InstallationThere are 2 ways to install theocrd_cistools:normal packaging:makeinstall# or equally: pip install -U pip .(Installsocrd_cisincluding its Python dependencies
from the current directory to the Python package directory.)editable mode:makeinstall-devel# or equally: pip install -e -U pip .(Installsocrd_cisincluding its Python dependencies
from the current directory.)It is possible (and recommended) to installocrd_cisin a custom user directory
(instead of system-wide) by usingvirtualenv(orvenv):# create venv:python3-mvenvvenv-dir# where "venv-dir" could be any path name# enter venv in current shell:sourcevenv-dir/bin/activate# install ocrd_cis:makeinstall# or any other way (see above)# use ocrd_cis:ocrd-cis-ocropy-binarize...# finally, leave venv:deactivateProfilerThe post correction is dependent on the languageprofilerand its laguage
configurations to generate corrections for suspicious words.  In order
to use the post correction a profiler with according language
configruations have to be present on the system.  You can refer to ourmanualsand ourlexical
resourcesfor more information.If you use docker you can use the preinstalled profiler from within
the docker-container.  The profiler is installed to/apps/profilerand the language configurations lie in/etc/profiler/languagesin
the container image.UsageMost tools follow theOCR-D specifications,
(which makes themOCR-Dprocessors,)
i.e. they accept the command-line options--input-file-grp,--output-file-grp,--page-id,--parameter,--mets,--log-level(each with an argument).
Invoke with--helpto get self-documentation.Some of the processors (most notably the alignment tool) expect a comma-seperated list
of multiple input file groups, or multiple output file groups.Theocrd-tool.jsoncontains a formal
description of all the processors along with the parameter config file
accepted by their--parameterargument.ocrd-cis-postcorrectThis processor runs the post correction using a pre-trained model.  If
additional support OCRs should be used, models for these OCR steps are
required and must be executed and aligned beforehand (seethe test
scriptfor an example).There is a basic model trained on the OCR-D ground truth.  It gets
installed allongside this module.  You can get the model's install
path usingocrd-cis-data -model(see below for a description ofocrd-cis-data).  To use this model (or any other model) themodelparameter in the configuration file must be set to the path of the
model to use.  Be aware that the models are trained with a specific
maximal number of OCR's (usally 2) and that is not possible to use
more OCR's than the number used for training (it is possible to use
less, though).Arguments:--parameterpath to configuration file--input-file-grpname of the master-OCR file group--output-file-grpname of the post-correction file group--log-levelset log level--metspath to METS file in workspaceAs mentioned above in order to use the postcorrection with input from
multiple OCR's, some preprocessing steps are needed: firstly the
additional OCR recognition has to be done and secondly the multiple
OCR's have to be aligned (you can also take a look to the functionocrd_cis_alignin thetests).  Assuming an
original recognition as file groupOCR1on the segmented document of
file groupSEG, the folloing commands can be used:ocrd-ocropus-recognize-ISEG-OOCR2...# additional OCRocrd-cis-align-IOCR1,OCR2-OALGN...# align OCR1 and OCR2ocrd-cis-postcorrect-IALGN-OPC...# post correctionocrd-cis-alignAligns tokens of multiple input file groups to one output file group.
This processor is used to align the master OCR with any additional support
OCRs.  It accepts a comma-separated list of input file groups, which
it aligns in order.Arguments:--parameterpath to configuration file--input-file-grpcomma seperated list of the input file groups;
first input file group is the master OCR; if there is a ground
truth (for evaluation) it must be the last file group in the list--output-file-grpname of the file group for the aligned result--log-levelset log level--metspath to METS file in workspaceocrd-cis-dataHelper tool to get the path of the installed data files. Usage:ocrd-cis-data [-h|-jar|-3gs|-model|-config]to get the path of the
jar library, the pre-trained post correction model, the path to the
default 3-grams language model file or the default training
configuration file.  This tool does not follow the OCR-D conventions.TraininingThere is no dedicated training script provided. Models are trained
using the java implementation directly (check out thetraining test
scriptfor an example).  Training a
model requires a workspace containing one or more file groups
consisting of aligned OCR and ground-truth documents (the last file
group has to be the ground truth).Arguments:--parameterpath to configuration file--input-file-grpname of the input file group to profile--output-file-grpname of the output file group where the profile
is stored--log-levelset log level--metspath to METS file in the workspaceocrd-cis-ocropy-trainTheocropy-traintool can be used to train LSTM models.
It takes ground truth from the workspace and saves (image+text) snippets from the corresponding pages.
Then a model is trained on all snippets for 1 million (or the given number of) randomized iterations from the parameter file.java-jar$(ocrd-cis-data-jar)\-ctrain\--input-file-grpOCR1,OCR2,GT\--log-levelDEBUG\-mmets.xml\--parameter$(ocrd-cis-data-config)ocrd-cis-ocropy-clipTheclipprocessor can be used to remove intrusions of neighbouring segments in regions / lines of a page.
It runs a connected component analysis on every text region / line of every PAGE in the input file group, as well as its overlapping neighbours, and for each binary object of conflict, determines whether it belongs to the neighbour, and can therefore be clipped to the background. It references the resulting segment image files in the output PAGE (viaAlternativeImage).
(Use this to suppress separators and neighbouring text.)ocrd-cis-ocropy-clip\-IOCR-D-SEG-REGION\-OOCR-D-SEG-REGION-CLIP\-p'{"level-of-operation": "region"}'Available parameters are:"level-of-operation"[string-"region"]PAGEXMLhierarchylevelgranularitytoannotateimagesforPossiblevalues:["region","line"]"dpi"[number--1]pixeldensityindotsperinch(overridesanymeta-dataintheimages);disabledwhennegative"min_fraction"[number-0.7]shareofforegroundpixelsthatmustberetainedbythelargestlabelocrd-cis-ocropy-resegmentTheresegmentprocessor can be used to remove overlap between neighbouring lines of a page.
It runs a line segmentation on every text region of every PAGE in the input file group, and for each line already annotated, determines the label of largest extent within the original coordinates (polygon outline) in that line, and annotates the resulting coordinates in the output PAGE.
(Use this to polygonalise text lines that are poorly segmented, e.g. via bounding boxes.)ocrd-cis-ocropy-resegment\-IOCR-D-SEG-LINE\-OOCR-D-SEG-LINE-RES\-p'{"extend_margins": 3}'Available parameters are:"dpi"[number--1]pixeldensityindotsperinch(overridesanymeta-dataintheimages);disabledwhennegative"min_fraction"[number-0.8]shareofforegroundpixelsthatmustberetainedbythelargestlabel"extend_margins"[number-3]numberofpixelstoextendtheinputpolygonshorizontallyandverticallybeforeintersectingocrd-cis-ocropy-segmentThesegmentprocessor can be used to segment (pages or) regions of a page into (regions and) lines.
It runs a line segmentation on every (page or) text region of every PAGE in the input file group, and adds (text regions containing)TextLineelements with the resulting polygon outlines to the annotation of the output PAGE.
(Doesnotdetect tables.)ocrd-cis-ocropy-segment\-IOCR-D-SEG-BLOCK\-OOCR-D-SEG-LINE\-p'{"level-of-operation": "page", "gap_height": 0.015}'Available parameters are:"dpi"[number--1]pixeldensityindotsperinch(overridesanymeta-dataintheimages);disabledwhennegative;whendisabledandnometa-dataisfound,300isassumed"level-of-operation"[string-"region"]PAGEXMLhierarchyleveltoreadimagesfromandaddelementstoPossiblevalues:["page","table","region"]"maxcolseps"[number-20](whenoperatingonthepage/tablelevel)maximumnumberofwhite/backgroundcolumnseparatorstodetect,countedpiece-wise"maxseps"[number-20](whenoperatingonthepage/tablelevel)numberofblack/foregroundcolumnseparatorstodetect(andsuppress),countedpiece-wise"maximages"[number-10](whenoperatingonthepagelevel)maximumnumberofblack/foregroundverylargecomponentstodetect(andsuppress),countedpiece-wise"csminheight"[number-4](whenoperatingonthepage/tablelevel)minimumheightofwhite/backgroundorblack/foregroundcolumnseparatorsinmultiplesofscale/capheight,countedpiece-wise"hlminwidth"[number-10](whenoperatingonthepage/tablelevel)minimumwidthofblack/foregroundhorizontalseparatorsinmultiplesofscale/capheight,countedpiece-wise"gap_height"[number-0.01](whenoperatingonthepage/tablelevel)largestminimumpixelaverageinthehorizontalorverticalprofiles(acrossthebinarizedimage)tostillberegardedasagapduringrecursiveX-Ycutfromlinestoregions;needstobelargerwhenmoreforegroundnoiseispresent,reducetoavoidmistakingtextfornoise"gap_width"[number-1.5](whenoperatingonthepage/tablelevel)smallestwidthinmultiplesofscale/capheightofavalleyinthehorizontalorverticalprofiles(acrossthebinarizedimage)tostillberegardedasagapduringrecursiveX-Ycutfromlinestoregions;needstobesmallerwhenmoreforegroundnoiseispresent,increasetoavoidmistakinginter-lineasparagraphgapsandinter-wordasinter-columngaps"overwrite_order"[boolean-true](whenoperatingonthepage/tablelevel)removeanyreferencesforexistingTextRegionelementswithinthetop(page/table)readingorder;otherwiseappend"overwrite_separators"[boolean-true](whenoperatingonthepage/tablelevel)removeanyexistingSeparatorRegionelements;otherwiseappend"overwrite_regions"[boolean-true](whenoperatingonthepage/tablelevel)removeanyexistingTextRegionelements;otherwiseappend"overwrite_lines"[boolean-true](whenoperatingontheregionlevel)removeanyexistingTextLineelements;otherwiseappend"spread"[number-2.4]distanceinpoints(pt)fromtheforegroundtoprojecttextline(ortextregion)labelsintothebackgroundforpolygonalcontours;ifzero,projecthalfascale/capheightocrd-cis-ocropy-deskewThedeskewprocessor can be used to deskew pages / regions of a page.
It runs a projection profile-based skew estimation on every segment of every PAGE in the input file group and annotates the orientation angle in the output PAGE.
(Doesnotinclude orientation detection.)ocrd-cis-ocropy-deskew\-IOCR-D-SEG-LINE\-OOCR-D-SEG-LINE-DES\-p'{"level-of-operation": "page", "maxskew": 10}'Available parameters are:"maxskew"[number-5.0]modulusofmaximumskewingangletodetect(largerwillbeslower,0willdeactivatedeskewing)"level-of-operation"[string-"region"]PAGEXMLhierarchylevelgranularitytoannotateimagesforPossiblevalues:["page","region"]ocrd-cis-ocropy-denoiseThedenoiseprocessor can be used to despeckle pages / regions / lines of a page.
It runs a connected component analysis and removes small components (black or white) on every segment of every PAGE in the input file group and references the resulting segment image files in the output PAGE (asAlternativeImage).ocrd-cis-ocropy-denoise\-IOCR-D-SEG-LINE-DES\-OOCR-D-SEG-LINE-DEN\-p'{"noise_maxsize": 2}'Available parameters are:"noise_maxsize"[number-3.0]maximumsizeinpoints(pt)forconnectedcomponentstoregardasnoise(0willdeactivatedenoising)"dpi"[number--1]pixeldensityindotsperinch(overridesanymeta-dataintheimages);disabledwhennegative"level-of-operation"[string-"page"]PAGEXMLhierarchylevelgranularitytoannotateimagesforPossiblevalues:["page","region","line"]ocrd-cis-ocropy-binarizeThebinarizeprocessor can be used to binarize (and optionally denoise and deskew) pages / regions / lines of a page.
It runs the "nlbin" adaptive whitelevel thresholding on every segment of every PAGE in the input file group and references the resulting segment image files in the output PAGE (asAlternativeImage). (If a deskewing angle has already been annotated in a region, the tool respects that and rotates accordingly.) Images can also be produced grayscale-normalized.ocrd-cis-ocropy-binarize\-IOCR-D-SEG-LINE-DES\-OOCR-D-SEG-LINE-BIN\-p'{"level-of-operation": "page", "threshold": 0.7}'Available parameters are:"method"[string-"ocropy"]binarizationmethodtouse(only'ocropy'willincludedeskewinganddenoising)Possiblevalues:["none","global","otsu","gauss-otsu","ocropy"]"threshold"[number-0.5]forthe'ocropy'and' global'method,black/whitethresholdtoapplyonthewhitelevelnormalizedimage(thelargerthemore/heavierforeground)"grayscale"[boolean-false]forthe'ocropy'method,producegrayscale-normalizedinsteadofthresholdedimage"maxskew"[number-0.0]modulusofmaximumskewingangle(indegrees)todetect(largerwillbeslower,0willdeactivatedeskewing)"noise_maxsize"[number-0]maximumpixelnumberforconnectedcomponentstoregardasnoise(0willdeactivatedenoising)"level-of-operation"[string-"page"]PAGEXMLhierarchylevelgranularitytoannotateimagesforPossiblevalues:["page","region","line"]ocrd-cis-ocropy-dewarpThedewarpprocessor can be used to vertically dewarp text lines of a page.
It runs the baseline estimation and center normalizer algorithm on every line in every text region of every PAGE in the input file group and references the resulting line image files in the output PAGE (asAlternativeImage).ocrd-cis-ocropy-dewarp\-IOCR-D-SEG-LINE-BIN\-OOCR-D-SEG-LINE-DEW\-p'{"range": 5}'Available parameters are:"dpi"[number--1]pixeldensityindotsperinch(overridesanymeta-dataintheimages);disabledwhennegative"range"[number-4.0]maximumverticaldispositionormaximummargin(willbemultipliedbymeancenterlinedeltastoyieldpixels)"max_neighbour"[number-0.05]maximumrateofforegroundpixelsintrudingfromneighbouringlines(linewillnotbeprocessedabovethat)ocrd-cis-ocropy-recognizeTherecognizeprocessor can be used to recognize the lines / words / glyphs of a page.
It runs LSTM optical character recognition on every line in every text region of every PAGE in the input file group and adds the resulting text annotation in the output PAGE.ocrd-cis-ocropy-recognize\-IOCR-D-SEG-LINE-DEW\-OOCR-D-OCR-OCRO\-p'{"textequiv_level": "word", "model": "fraktur-jze.pyrnn"}'Available parameters are:"textequiv_level"[string-"line"]PAGEXMLhierarchylevelgranularitytoaddtheTextEquivresultstoPossiblevalues:["line","word","glyph"]"model"[string]ocropymodeltoapply(e.g.fraktur.pyrnn)TesserocrInstall essential system packages for Tesserocrsudoapt-getinstallpython3-tk\tesseract-ocrlibtesseract-devlibleptonica-dev\libimage-exiftool-perllibxml2-utilsThen install Tesserocr from:https://github.com/OCR-D/ocrd_tesserocrpipinstall-rrequirements.txt
pipinstall.Download and move tesseract models from:https://github.com/tesseract-ocr/tesseract/wiki/Data-Filesor use your
own models and place them into: /usr/share/tesseract-ocr/4.00/tessdataWorkflow configurationA decent pipeline might look like this:image normalization/optimizationpage-level binarizationpage-level cropping(page-level binarization)(page-level despeckling)page-level deskewing(page-level dewarping)region segmentation, possibly subdivided intotext/non-text separationtext region segmentation (and classification)reading order detectionnon-text region classificationregion-level clipping(region-level deskewing)line segmentation(line-level clipping or resegmentation)line-level dewarpingline-level recognition(line-level alignment and post-correction)If GT is used, then cropping/segmentation steps can be omitted.If a segmentation is used which does not produce overlapping segments, then clipping/resegmentation can be omitted.TestingTo run a few basic tests typemake test(ocrd_cishas to be
installed in order to run any tests).MiscellaneousOCR-D workspaceCreate a new (empty) workspace:ocrd workspace init workspace-dircd intoworkspace-dirAdd new file to workspace:ocrd workspace add file -G group -i id -m mimetype -g pageIdOCR-D linksOCR-DGithubProject-pageGround-truth