Package: py-ssvad-metrics
Single-Scene Video Anomaly Detection (SSVAD) MetricsThis project contains evaluation protocol (metrics) for benchmarking single-scene video anomaly detection (SSVAD).Evaluation ProtocolThis is an unofficial implementation of Sec. 2.2 ofA Survey of Single-Scene Video Anomaly Detection.
This metric is intendedonlyfor single-scene video anomaly detection methods and is untested for other purposes.There are 3 kind of supported outputs from SSVAD methods/ground-truths:Pixel-level anomaly scores maps.Bounding-boxes based outputs, it will be implicitly converted into pixel-level anomaly scores maps. Overlapping bounding-boxes will be averaged by its scores.Frame-level anomaly scores.The metrics can be categorized into 4:Track-based metric: requires pixel-level anomaly scores maps predictions and ground-truths, plus anomaly track ID for each frame in the ground-truths that contains anomalous regions (predictions does not require anomaly track ID and is ignored in the process).Region-based metric: requires pixel-level anomaly scores maps predictions and ground-truths.Pixel-level traditional metric: requires pixel-level anomaly scores maps predictions and ground-truths.Frame-level traditional metric: only require frame-level anomaly scores; does not require pixel-level anomaly scores maps predictions and ground-truths.Each prediction output and ground-truth annotation must be a JSON file that follows data structure defined inssvad_metrics.data_schema.VADAnnotation.
Pixel-level anomaly scores maps arrays must be provided using.tiffor.npyformat,
containing single-precision (32-bit) floating point values ranging from 0.0 to 1.0. For ground-truths, it can be same as pixel-level predictions, or using boolean values instead. But only.npyfiles supported if using boolean values (as.tiffrequires 32-bit floating point values).InstallationThis metrics is available via PyPI.pipinstallpy-ssvad-metricsUsagePrepare ground-truth JSON files and prediction JSON files (also the pixel anomaly score map linked files for pixel-level predictions and groundtruths).ssvad_metrics.data_schema.VADAnnotationcan be used for the data structure reference and validator of the JSON file.JSON file examples are in thesamples folder.For UCSD Pedestrian 1 and 2 datasets, CUHK Avenue dataset, and Street Scene dataset, we provided scripts for converting ground-truth annotation files from Street Scene dataset provided by the paper (txt files, each row contains: <filename> <track_id> <x_center> <y_center> <width> <height>). Download link is provided in the paperhttp://www.merl.com/demos/video-anomaly-detection.Example usage for single groundtruth and prediction file pair:importssvad_metricsresult=ssvad_metrics.evaluate("tests/gt_examples/Test001_gt.json","tests/pred_examples/Test001_pred.json")Example usage for multiple groundtruth and prediction file pairs:importssvad_metricsresult=ssvad_metrics.accumulated_evaluate("tests/gt_examples","tests/pred_examples",gt_name_suffix="_gt",pred_name_suffix="_pred")For more examplles, seesamples folder.Visual InspectionWe also provide tools for visual inspection for checking thequality of false positives.
After installingpy-ssvad-metrics, the visualizer can be used by executingssvad-visualizeorssvad-visualize-dir.
Seessvad-visualize --helporssvad-visualize-dir --helpfor details and usage.
Also, seessvad_metrics.visualizeorssvad_metrics.visualize_dirfor the Python API details and usage.
Requires FFMPEG installation on the system andffmpeg-pythonpackage (NOTpython-ffmpeg).
FFMPEG is used instead of OpenCV VideoWriter, since the OpenCV packages that are distributed in the PyPI usually does not embed FFMPEG that uis compiled with H264 codec.ReferencesB. Ramachandra, M. Jones and R. R. Vatsavai, "A Survey of Single-Scene Video Anomaly Detection," in IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109/TPAMI.2020.3040591.LicenseGPL-3.0 License. Brought to open-source by PT Qlue Performa Indonesia.ContributingFeel free to contribute for improvements.