Package: robustness-benchmark
Metrics Robustness BenchmarkRepository for "Comparing the robustness of modern image- and video-quality metrics to adversarial attacks" paperGeneral requirementsPython3, Jypyter, GPU (with CUDA), Docker (if you want to launch adversarial attacks)Repository structurerobustness_benchmark/methods/- adversarial attacks and utilsrobustness_benchmark/methods/utils/- supportive functions for attacksrobustness_benchmark/models/- domain transformation model weightsrobustness_benchmark/interface.py- benchmark module interface. Main functions arerun_attacks(), collect_results(), domain_transform(), evaluate_robustness(), run_full_pipeline(). More details on usage in functions' docstrings and demo Notebooklib_demo.ipynb.subjects/- metrics code (only MANIQA metric for demo)res/- precomputed results (only MANIQA metric for demo)test_dataset/- small test set of images to test module functionalitytest_results/- results fortest_dataset(MANIQA)Demo code:lib_demo.ipynb- benchmark module usagedemo_scoring.ipynb- calculate robustness scores for MANIQA (using precomputed results from /res/)Supplementary code:robustness_benchmark/score_methods.py- functions to calculate attack efficiency scores (described in paper in "Methodology" section)robustness_benchmark/NOT.py- functions to perform Neural Optimal Transport for mapping metrics values to one domainRunning the demo codeRobusness Benchmark pip moduleModule installationNote: It is recommended to install PyTorch version suitable for your Python/CUDA installation fromofficial websitebefore installing the library.Direct install viapip install robustness_benchmarkwill be available soon. To install the latest version of the module you can clone the repo and pip install it:git clone https://github.com/msu-video-group/MSU_Metrics_Robustness_Benchmark/tree/mainpip install -r requirements.txtpip install -e .Demo metric setupTo install demo metric MANIQA you can use following commands in benchmark's root directory:git submodule update --init --recursivecd subjects/maniqa/src && git apply ../patches/maniqa.patch -vcd subjects/maniqa && cp model.py srcLaunch adversarial attacksExample usage of module can be found inlib_demo.ipynbNotebook. To run the attacks and evaluate metric's robustness you can run it cell-by-cell. You can also test other metrics and attacks if they follow the module interface. Main library funtions are listed below, for more details check functions' docstrings.robustness_benchmark.interface functionsrun_attacks()- Run given attacks on metric on specified datasets.collect_results()- Given the path to directory with raw results produced byrun_attack(), collects them into single DataFrame.domain_transform()- Apply domain transformation to collected attack results fromcollect_results(). Works only with supported metrics.evaluate_robustness()- Evaluate metric's robustness to attacks and return table with results on each type of attack.run_full_pipeline()- Run full benchmark pipeline: run attacks, save results, collect them, apply domain transform (if needed), evaluate.Using precomputed resultsDownload precomputed results used in article fromhere(password: 'neurips_benchmark_2023')Clone this repo:git clone https://github.com/msu-video-group/MSU_Metrics_Robustness_Benchmark/tree/mainInstall requirements:pip install -r requirements.txtLaunch Jupyter noteboor or jupyter labLaunch demo_scoring.ipynb cell-by-cell