Package: insynth
InSynthRobustness testing of Keras models using domain-specific input generation in PythonExplore the docs »View Demo·Report Bug·Request FeatureTable of ContentsAbout The ProjectBuilt WithGetting StartedPrerequisitesInstallationUsageReproduce Experimental ResultsContributingLicenseContactAbout The ProjectThe robustness of machine learning models is crucial to their safe and reliable operation in real-world applications.
However, conducting robustness tests is hard as it requires evaluating the model under test repeatedly on different
datasets.InSynthprovides an easy-to-use, efficient and reliable framework for conducting robustness tests.It works by applying a set of domain-specific input generation techniques (image, audio or text) to the seed dataset,
and then evaluating the model under test on the generated inputs. Then, a set of coverage criteria are evaluated to
determine how well each dataset covers the model. Finally, a report is generated comparing the models' performance and
coverage on different generated datasets.(back to top)Built WithPythonNumpyKerasPillowlibrosaaudiomentationsalbumentationsscipypandaspydubtqdm(back to top)Getting StartedThis section describes the steps to follow when you want to get started with theInSynthproject.PrerequisitesBefore installingInSynth, make sure you have the following software applications installed and updated to the latest
version.PythonpipffmpegInstallationTo installInSynth, only one step is required.Run the following command to install the python package from the PyPI repository:pipinstallinsynth(back to top)UsageInSynthcan be used in a variety of ways depending on the goal you are trying to achieve.Data GenerationTo mutate an existing dataset using any of the perturbators provided in the framework, follow the steps below.Import the perturbator (e.g. theImageNoisePerturbator) from the respective module.frominsynth.perturbators.imageimportImageNoisePerturbatorCreate an instance of the perturbator.perturbator=ImageNoisePerturbator()Create aPILimage object from a file stored on disk and apply the perturbator to it.seed_image=Image.open('path/to/image.jpg')mutated_image=perturbator.apply(seed_image)For audio perturbators, the same procedure applies but using thelibrosa.loadmethod.
Similarly, text perturbators expect the seed text to be provided as a string.Save the mutated image to disk or display it.mutated_image.save('path/to/mutated_image.jpg')mutated_image.show()Coverage Criteria CalculationTo calculate the coverage criteria for a model, follow the steps below.Import the coverage criteria (e.g. theCoverageCriteria) from the respective module.frominsynth.calculators.neuronimportStrongNeuronActivationCoverageCalculatorCreate an instance of the coverage criteria and pass the model to be tested to the constructor.coverage_calculator=StrongNeuronActivationCoverageCalculator(model)If applicable, run theupdate_neuron_boundsmethod to determine the neuron bounds of the model.coverage_calculator.update_neuron_bounds(training_dataset)Run theupdate_coveragemethod to update model coverage for the given input.coverage_calculator.update_coverage(input_data)Run theget_coveragemethod to retrieve the current model coverage.coverage=coverage_calculator.get_coverage()Print the coverage to the console.print(coverage)Robustness TestingThe previous two sections describe how to generate a mutated dataset and calculate the coverage criteria for a model.
These are prerequisites for testing the robustness of a model.
In order to conduct a full end-to-end robustness test, the runner class is provided inInSynth.Import the runner class from the respective module.frominsynth.runnersimportBasicImageRunnerCreate an instance of the runner class and pass the list of perturbators, the list of coverage calculators and the model to the constructor in addition to the dataset inputs and target variables.runner=BasicImageRunner(list_of_perturbators,list_of_coverage_calculators,dataset_x,dataset_y,model)Note that thedataset_xparameter should be a method returning a python generator iterating over all samples to enable the processing of large datasets which do not fit into memory.dataset_x=lambda:(xforxindataset)Run therunmethod to conduct the end-to-end robustness test.report,robustness_score=runner.run()Use thereportvariable to analyse the test results or use therobustness_scorevariable to retrieve a single robustness measure of the model.print(report)print(robustness_score)If you want to apply all available perturbators and coverage calculators for a given domain, utilize the respectiveComprehensiveRunnerclasses.For more examples, please refer to theDocumentation(back to top)Reproduce Experimental ResultsThe experimental results from the thesis can be reproduced by running the corresponding scripts in theexperimentsdirectory.The performance comparison experiment is conducted in thereproduce_coverage_speed_comparison.pyscript.The performance and coverage experiment is conducted in thereproduce_imagenet_results.py,reproduce_speaker_recognition_results.pyandreproduce_sentiment_analysis_results.pyscripts.
To generate the speaker recognition and sentiment analysis models, first thegenerate_model_speaker_recognition.pyandgenerate_model_sentiment_analysis.pyscripts have to be run.The perturbation strength experiment is conducted in thereproduce_imagenet_sensitivity_results.py,reproduce_speaker_recognition_sensitivity_results.pyandreproduce_sentiment_analysis_sensitivity_results.pyscripts.Lastly, the diagrams used in the thesis can be generated by running theresult_analysis.ipynbnotebook.(back to top)ContributingContributions are what make the open source community such an amazing place to learn, inspire, and create. Any
contributions you make aregreatly appreciated.If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also
simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!Fork the ProjectCreate your Feature Branch (git checkout -b feature/AmazingFeature)Commit your Changes (git commit -m 'Add some AmazingFeature')Push to the Branch (git push origin feature/AmazingFeature)Open a Pull Request(back to top)LicenseDistributed under the MIT License. SeeLICENSEfor more information.(back to top)ContactMarian LambertProject Link:https://github.com/mlxyz/insynth(back to top)