Package: cleval
CLEval: Character-Level Evaluation for Text Detection and Recognition TasksOfficial implementation of CLEval |paperOverviewWe propose a Character-Level Evaluation metric (CLEval). To perform fine-grained assessment of the results,instance matchingprocess handles granularity difference andscoring processconducts character-level evaluation. Please refer to the paper for more details. This code is based onICDAR15 official evaluation code.2023.10.16 Huge UpdateMuch More Faster Versionof CLEval has been Uploaded!!Support CLISupport torchmetricSupport scale-wise evaluationSimplified Method DescriptionSupported annotation typesLTRB(xmin, ymin, xmax, ymax)QUAD(x1, y1, x2, y2, x3, y3, x4, y4)POLY(x1, y1, x2, y2, ..., x_2n, y_2n)Supported datasetsICDAR 2013 Focused Scene TextLinkICDAR 2015 Incidental Scene TextLinkTotalTextLinkAny other datasets that have a similar format with the datasets mentioned aboveInstallationBuild from pipdownload from Clova OCR pypi$pipinstallclevalor build with url$pipinstallgit+https://github.com/clovaai/CLEval.git--userBuild from source$gitclonehttps://github.com/clovaai/CLEval.git
$cdcleval
$pythonsetup.pyinstall--userHow to useYou can replaceclevalwithPYTHONPATH=$PWD python cleval/main.pyfor evaluation using source.$PYTHONPATH=$PWDpythoncleval/main.py-g=gt/gt_IC13.zip-s=[result.zip]--BOX_TYPE=LTRBDetection evaluation (CLI)$cleval-g=gt/gt_IC13.zip-s=[result.zip]--BOX_TYPE=LTRB# IC13$cleval-g=gt/gt_IC15.zip-s=[result.zip]# IC15$cleval-g=gt/gt_TotalText.zip-s=[result.zip]--BOX_TYPE=POLY# TotalTextNotesThe default value ofBOX_TYPEis set toQUAD. It can be explicitly set to--BOX_TYPE=QUADwhen running evaluation on IC15 dataset.Add--TANSCRIPTIONoption if the result file contains transcription.Add--CONFIDENCESoption if the result file contains confidence.End-to-end evaluation (CLI)$cleval-g=gt/gt_IC13.zip-s=[result.zip]--E2E--BOX_TYPE=LTRB# IC13$cleval-g=gt/gt_IC15.zip-s=[result.zip]--E2E# IC15$cleval-g=gt/gt_TotalText.zip-s=[result.zip]--E2E--BOX_TYPE=POLY# TotalTextNotesAdding--E2Ealso automatically adds--TANSCRIPTIONoption. Make sure that the transcriptions are included in the result file.Add--CONFIDENCESoption if the result file contains confidence.TorchMetricfromclevalimportCLEvalMetricmetric=CLEvalMetric()forgt,detinzip(gts,dets):# your fancy algorithm# ...# gt_quads = ...# det_quads = ...# ..._=metric(det_quads,gt_quads,det_letters,gt_letters,gt_is_dcs)metric_out=metric.compute()metric.reset()Profiling$cleval-g=resources/test_data/gt/gt_eval_doc_v1_kr_single.zip-s=resources/test_data/pred/res_eval_doc_v1_kr_single.zip--E2E-v--DEBUG--PPROFILE>profile.txt
$PYTHONPATH=$PWDpythoncleval/main.py-gresources/test_data/gt/dummy_dataset_val.json-sresources/test_data/pred/dummy_dataset_val.json--SCALE_WISE--DOMAIN_WISE--ORIENTATION--E2E--ORIENTATION-v--PROFILE--DEBUG>profile.txtParamters for evaluation scriptnametypedefaultdescription-gstringpath to ground truth zip file-sstringpath to result zip file-ostringpath to save per-sample result file 'results.zip'nametypedefaultdescription--BOX_TYPEstringQUADannotation type of box (LTRB, QUAD, POLY)--TRANSCRIPTIONbooleanFalseset True if result file has transcription--CONFIDENCESbooleanFalseset True if result file has confidence--E2EbooleanFalseto measure end-to-end evaluation (if not, detection evalution only)--CASE_SENSITIVEbooleanTrueset True to evaluate case-sensitively. (only used in end-to-end evaluation)Note : Please refer toarg_parser.pyfile for additional parameters and default settings used internally.Note : For scalewise evaluation, we measure the ratio of the shorter length (text height) of the text-box to the longer length of the image.
Through this, evaluation for each ratio can be performed. To adjust the scales, please use SCALE_BINS argument.Citation@article{baek2020cleval,
  title={CLEval: Character-Level Evaluation for Text Detection and Recognition Tasks},
  author={Youngmin Baek, Daehyun Nam, Sungrae Park, Junyeop Lee, Seung Shin, Jeonghun Baek, Chae Young Lee and Hwalsuk Lee},
  journal={arXiv preprint arXiv:2006.06244},
  year={2020}
}Contact usCLEval has been proposed to make fair evaluation in the OCR community, so we want to hear from many researchers. We welcome any feedbacks to our metric, and appreciate pull requests if you have any comments or improvements.LicenseCopyright (c) 2020-present NAVER Corp.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.ContributePlease use pre-commit which uses Black and Isort.$ pip install pre-commit
$ pre-commit installStep By StepWrite an issue.Match code style (black, isort)Wirte test code.Delete branch after Squash&Merge.Required Approve: 1Code MaintainerDonghyun Kim (artit.anthony@gmail.com)