Package: digital-eval
digital evalPython3 Tool to report evaluation outcomes from mass digitalization workflows.Featuresmatch automatically groundtruth (i.e. reference data) and candidates by filenameuse geometric information to evaluate only specific frame (i.e. specific column or region from large page) of
candidates (requires ALTO or PAGE format)aggregate evaluation outcome on domain range (with multiple subdomains)choose from textual metrics based on characters or words plus common Information Retrievalchoose between accuracy / error rate and different UTF-8 Python normsformats: ALTO, PAGE or plain text for both groundtruth and candidatesspeedup with parallel executionadditional OCR util:filter custom areas of single OCR filesInstallationpipinstalldigital-evalUsageMetricsCalculate similarity (acc) or difference (err) ratios between single reference/groundtruth and test/candidate item.Edit-Distance basedCharacter-based text string minus whitechars (Cs,Characters) or Letter-based (Ls,Letters) minus whites,
punctuation and digits.
Word/Token-based edit-distance of single tokens identified by whitespaces.Set basedCalculate union of sets of tokens/words (BoW,BagOfWords).
Operate on sets of tokens/words with respect to language specific stopwords usingnltk-framework for:Precision (IRPre,Pre,Precision): How many tokens from candidate are in groundtruth reference?Recall (IRRec,Rec,Recall): How many tokens from groundtruth reference should candidate include?F-Measure (IRFMeasure,FM): weighted ratio Precision / RecallUTF-8 NormalisationsUse standard Python Implementation of UTF-8 normalizations; default:NFKD.StatisticsStatistics calculated vianumpyinclude arithmetic mean, median and outlier detection with
interquartile range and are based on the specific groundtruth/reference (ref) for each metric, i.e. char, letters or
tokens.Evaluate treelike structuresTo evaluate OCR-candidate-data batch-like versus existing Groundtruth, please make sure that your structures fit this
way:groundtruthroot/
├──<domain>/│└──<subdomain>/
│└──<page-01>.gt.xml
candidateroot/
├──<domain>/│└──<subdomain>/
│└──<page-01>.xmlNow call via:digital-eval<path-candidate-root>/domain/-ref<path-groundtruth>/domain/for an aggregated overview on stdout. Feel free to increase verbosity via-v(or even-vv) to get detailed
information about each single data set which was evaluated.Structured OCR is considered to contain valid geometrical and textual data on word level, even though for recent PAGE
also line level is possible.Data problemsInconsistent OCR Groundtruth with empty texts (ALTO String elements missing CONTENT or PAGE without TextEquiv) or
invalid geometrical coordinates (less than 3 points or even empty) will lead to evaluation errors if geometry must be
respected.Additional OCR UtilsFilter AreaYou can filter a custom area of a page of an OCR file by providing the points of an arbitrary shape.
The format of the-p, --pointsargument is<pt_1_x>,<pt_1_y> <pt_2_x>,<pt_2_y> <pt_3_x>,<pt_3_y> ... <pt_n_x>,<pt_n_y>. For simple rectangular areas this can be expressed also with two points, with first point as top left and second point as bottom right:<pt_top_left_x>,<pt_top_left_y> <pt_bottom_right_x>,<pt_bottom_right_y>.The following example filters a rectangular area of 600x400 pixels of a page, which is described by an input ALTO file and saves the result to an output ALTO fileocr-utilframe-ipage_1.alto.xml-p"0,0 600,0 600,400 0,400"-opage_1_area.alto.xmlShort version with top left and bottom right:ocr-utilframe-ipage_1.alto.xml-p"0,0 600,400"-opage_1_area.alto.xmlDevelopmentPlattform: Intel(R) Core(TM) i5-6500CPU@3.20GHz, 16GB RAM, Ubuntu 20.04 LTS, Python 3.8.# clone localgitclone<repository-url><local-dir>cd<local-dir># enable virtual python environment (linux)# and install librariespython3-mvenvvenv
.venv/bin/activate
python-mpipinstall-Upip
python-mpipinstall-rrequirements.txt# installpipinstall.# optional:# install additional development dependenciespipinstall-rtests/test_requirements.txt
pytest-v# rundigital-eval--helpContributeContributions, suggestions and proposals welcome!LicenceUnder terms of theMIT license.NOTE: This software depends on other packages thatmaybe licensed under different open source licenses.