Package: pyEdgeEval
Python Edge Evaluation ToolsEdge detection tasks heavily rely on the original codes introduced in theBSDS300/500 benchmarkwhich were targeted for MATLAB users.
In the field of computer vision, various edge detection algorithms are now resorting to Python which supports various machine learning libraries such as PyTorch and Tensorflow.
However, not everyone has access to MATLAB and the original benchmark codes are outdated.
I created this open-source library,pyEdgeEval, to make it easier to evaluate and reproduce recent deep learning models for edge and boundary detection.
The original C++ codes used in the MATLAB benchmarks are ported with Cython and the evaluation scripts are rewritten in Python3.
This is especially useful for evaluating algorithms on remote linux servers (just runpip install pyEdgeEval) and docker containers, which have been difficult before due to MATLAB's constrains.
The codebase is designed to be extensible and supports various tasks and datasets as well as different evaluation protocols.
To test the validity of the evaluation code,pyEdgeEval's results are compared with the results of the original MATLAB codes.
Besides benchmarking,pyEdgeEvaladds various tools for edge detection such asmask2edgetransformation.pyEdgeEvalis:a Python alternative to the original MATLAB benchmarklight with minimal dependenciesmodular architecture and easily customizablerelatively fast (uses multiprocessing and Cython)implements common preprocessing algorithmssupports various tasks and datasets (extensible to other datasets)supports various evaluation protocolsedge generation toolsetc...Supported tasks:Edge DetectionSemantic Boundary DetectionSupported datasets:BSDS500SBDCityscapes(semantic boundary detection)Disclaimers:The evaluation code does not output results that exactly match the original MATLAB benchmark. This could be for various reasons such as random seeds for matching algorithm. The results are, for the most part, close enough (around 0.01% difference).The codes and algorithms are not perfect. I will not take responsibility for how the code is used (check the license(s)).If you find some bugs or want to improve this project, please submit issues or pull requests.InstallationDependenciespython >= 3.8(tested on 3.8.x)cv2Installation guide# Install dependenciespipinstall-rrequirements.txt# install cv2 (e.g. use pip)pipinstallopencv-python# Option 1. install without cloning the project (only tested on ubuntu with python 3.8)pipinstallpyEdgeEval# Option 2. install as a pip package (install as a package)gitclonehttps://github.com/haruishi43/py-edge-eval.git
pipinstall-e.Converting Cityscapes Dataset for SBDScript:pythonscripts/convert_datasets/cityscapes.py--nproc=8NOTE:Beaware that using multi-processing will consume at most 10GB per process (I'm working on debugging memory allocation issues).--nonISwill generate non-IS boundaries.The script will generate full resolution training dataset, full resolution validation dataset, and half resolution validation dataset (both raw/thin for validation).Evaluation for each datasetsBSDS500# get complete list of options$pythonscripts/evaluate/bsds500.py-hOptions:Evaluate BSDS output

positional arguments:
  bsds_path             the root path of the BSDS-500 dataset
  pred_path             the root path of the predictions

optional arguments:
  -h, --help            show this help message and exit
  --output-path OUTPUT_PATH
                        the root path of where the results are populated
  --use-val             val or test
  --max-dist MAX_DIST   tolerance distance (default: 0.0075)
  --thresholds THRESHOLDS
                        the number of thresholds (could be a
                        list of floats); use 99 for eval
  --raw                 option to remove the thinning process (i.e. uses raw predition)
  --apply-nms           applies NMS before evaluation
  --nproc NPROC         the number of parallel threads# Example:pythonscripts/evaluate/bsds500.py<path/to/bsds500><path/to/pred><path/to/output>\--thresholds=5--nproc=8Tested with@xwjabc's HED implementation.Due to the randomness in the original MATLAB (C++) codebase, the results will be different (at most +-0.001 difference).Setting--nprocto the number of available cores will drastically improve the speed of evaluation.SBDpythonscripts/evaluate/sbd.py<path/to/sbd><path/to/pred><path/to/output>--categories=15--thresholds=5--nproc=8CityScapesFirst, create GT data using this script:# if you plan on evaluating with instance-sensitive edges (IS edges)pythonscripts/convert_dataset/cityscapes.py--nproc8# if you plan on evaluating with non-instance-sensitive edgespythonscripts/convert_dataset/cityscapes.py--nonIS--nproc8The scripts will create two types of edges (raw and thin) for two different scales (half and full).Evaluation script:# thin protocolpythonscripts/evaluate/cityscapes_thin.py<path/to/cityscapes><path/to/predictions><path/to/output>--categories='[1, 14]'--thresholds99--nproc8# raw protocolpythonscripts/evaluate/cityscapes_raw.py<path/to/cityscapes><path/to/predictions><path/to/output>--categories='[1, 14]'--thresholds99--nproc8For instance-insensitive edges, you would need to supply--pre-sealargument.You can also preprocess the predictions by passing--apply-thinningand/or--apply-nmsfor thinning and NMS respectively.LicenseThe code is released under the MIT License (please refer to the LICENSE file for details).I modified codes from other projects and their licenses applies to those files (please refer toLicenses).Development and ContributionSeedev.md.