Package: python-tool-competition-2024
SBFT Python Tool Competition RunnerThe Workshop onThe 17th International Search-Based and Fuzz Testing (SBFT)offers a Python tool competition on automatic test generation for the Python programming
language.Important SBFT'24 Python Tool Compeetition DatesAll dates are anywhere on earth (AoE, UTC-12h):Tool Submission: 1. December 2023Results provided to tool authors: 9. January 2024Competition report: 16. January 2024Notification to authors: 21. December 2023Camera ready due: 25. January 2024Author's registration due: TBAWorkshop: TBA April 2024SubmissionSubmit your test generator by sending an email all organizers:Nicolas Erni (ZHAW) -gann@zhaw.chAl-Ameen Mohammed Ali Mohammed (ZHAW) -alam@zhaw.chChristian Birchler (ZHAW) -birc@zhaw.chPouria Derakhshanfar (JetBrains) -pouria.derakhshanfar@jetbrains.comStephan Lukasczyk (University of Passau) -stephan.lukasczyk@uni-passau.deSebastiano Panichella (ZHAW) -panc@zhaw.chYour email should include:A link to the repositoryIf the repository is private then invite the competition organizers to the repositoryThe name of the version tagThe commit hash of the version tagOverviewIn general, manual testing of software systems is costly.
Automatic test generation for Python lowers barriers to
test and ensure quality for Python software.
Therefore, the SBFT initiates a competition on Python test generation,
which enhances research and implementation for novel test generators for Python.
This repository provides the necessary facilities and guidelines
for participating in the SBFT Python tool competition.Test Generator ImplementationThe platform provides an API for writing test cases.
You simply need to generate astringthat represent the body of a valid Python
testing module that tests a given target withpytest.
Furthermore, your test code needs to be compatible withcosmic-rayso that your test suite can be assessed through mutation scores.
Below you see how to use the platform to implement a generator.UsageInstallationThis tool requires at least Python 3.9 andpoetry.pip install python-tool-competition-2024Creating a New Test Generator ProjectRunpython-tool-competition-2024 initand follow the instructions on the screen.
During the setup process, you need to provide the following information:Human-readable name of the test generator (mandatory)Directory in which the project will be created (by default: the current directory)Your full name (mandatory)Your email address (mandatory)Information required to createpyproject.toml:Project's version (by default 0.1.0)Author (by default, it will use the information provided in the prior steps)Licence (optional)Compatible Python versions for your generatorProject's main dependencies (can be defined during the setup or afterward)Project's development dependencies (can be defined during the setup or afterward)This will create apoetryproject with all
required files.
It creates a template of the generator and exposes it as a plugin of the
competition runner.Additionally, some very basic example targets are created as well.
You can add your own files to thetargetsdirectory.Now you can implement thebuild_testfunction of your generator.
This will be called for each target file and it expects that either aTestGenerationSuccessorTestGenerationFailureis returned.
The success contains the body of the generated test file.
Storing the file is handled by the runner that runsbuild_test.
The failure contains a reason and lines that describe the failure.For examples see:https://github.com/ThunderKey/python-tool-competition-2024-klarahttps://github.com/ThunderKey/python-tool-competition-2024-hypothesis-ghostwriterhttps://github.com/ThunderKey/python-tool-competition-2024-pynguinRunning inside of the ProjectInside of the create project runpoetry run python-tool-competition-2024 run <generator name>.
This will start the test generation process.Withpoetry run python-tool-competition-2024 run -hyou can find out what
generators were detected.The tool does not only execute the test generator, it also runs the generated tests
against the code to measure different metrics: it measures line and branch coverage
using thecoverageframework;
it furthermore computesmutation scoreutilizing thecosmic-raytool.After finishing the test generation process, the script will print the
information regarding the coverage achieved by the tests generated by your tool.
It will also save these data asXMLfiles in the<generator name>/coveragesdirectory.LicenseCLI tool for developing python test generators
Copyright (C) 2023  Nicolas Erni

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <https://www.gnu.org/licenses/>.Improving the Competition RunnerInstallation:poetry installTesting:toxUsepre-commitif possible.