Package: fauxpy
1. FauxPyThe current repository contains the source code of FauxPy,
an automated fault localization tool for Python programs.
FauxPy supports seven well-known fault-localization
techniques in four families (spectrum-based,
mutation-based, predicate switching, and stack-trace based).
The techniques FauxPy currently supports are as follows:TarantulaOchiaiDStarMetallaxisMusePredicate switchingStack traceFauxPy has been tested on 13 real-world Python
programs (e.g., Keras, Pandas, ...), detailed in
our paper"An Empirical Study of Fault Localization in Python Programs"by Mohammad Rezaalipour and Carlo A. Furia.In the following video, we demonstrate some of
FauxPy's fault localization techniques in action.1.1 InstallationTo install FauxPy, run the following command:pip install fauxpyWe have tested FauxPy on Python 3.6, 3.7, and 3.8.
But it should work on other versions of Python as well.1.2 Using FauxPyFauxPy is a Pytest plugin, installing which adds new
command line options to Pytest, allowing users to
control FauxPy's behaviour.
Thus, using FauxPy is very similar to using Pytest.
For instance, imagine we have a Python projectmy_projectthe has two packages:code, andtests.
Packagecodecontains all the source code of the project, and
packagetestsincludes the project's test suite.
Executing the following commands run FauxPy
on packagecode, using all the test cases
inside packagetests, resulting in a list of
line numbers along with their suspiciousness scores.
This list is shown in the command line, and also, saved as
a csv file in a directory next to the project's directory.cd my_projectpython -m pytest --src codeThe following repository provides step-by-step instructions
on how to use FauxPy on some simple projects.https://github.com/atom-sw/fauxpy-examplesIt is advised to first take a look at the repository mentioned above,
before going any further with the instructions in the current repository,
especially, if you do not know much about fault localization in general.1.3 Command Line OptionsFauxPy adds 7 command line options to Pytest,
explained in the rest of this section.The following command shows an example of using all
of FauxPy's options at the same time. However, many of
these options are not mandatory and can be skipped
based on the usage scenario users have in mind.python -m pytest tests \
  --src code \
  --family sbfl \
  --exclude "[code/package_x, code/module_x.py]" \
  --granularity statement \
  --top-n 3 \
  --failing-list "[tests/test_module_y.py::test_function_z]"1.3.1 Targeted Source CodeName:--srcType:MandatoryOption--srcis the only mandatory command line option of FauxPy.
It takes a relative path to a Python package or module
within the project at hand. FauxPy only considers
program entities in the package or module passed
as--srcwhile
performing fault localization.
If users want to involve the whole project in
FauxPy's fault localization session, they can pass.(current directory's path) as the argument.1.3.2 Fault Localization FamilyName:--familyChoices:{sbfl,mbfl,ps,st}Type:OptionalDefault:sbflFauxPy supports four fault localization families: SBFL, MBFL, PS, and ST.
Using option--family, users can decided which fault localization
family should be used during their fault localization session.
Option--familyis an optional argument. If it is not provided,
FauxPy pickssbflby default.1.3.3 Fault Localization GranularityName:--granularityChoices:{statement,function}Type:OptionalDefault:statementThe current version of FauxPy supports two different
granularity levels: statement, and function.
Using statement-level granularity, FauxPy's default choice,
FauxPy outputs a list of line numbers and their suspiciousness scores.
Using the function-level granularity, FauxPy returns suspicious functions
instead of line numbers.1.3.4 Excluded ItemsName:--excludeType:OptionalDefault:No packages and modules are excludedSometimes, users need to exclude a package or module from
a fault localization session. For instance, if.is passed to
option--src, users might want to exclude packagetests(project's test suite) or directoryenv(virtual environment of the project) from their
fault localization session. If such items are not excluded,
FauxPy includes in its output the program entities in these
directories, as well. In such cases, users can use option--exclude.
This option takes a comma separated list of relative paths to
directories that are supposed to be excluded.For instance, the following command runs FauxPy on
the whole project but the two directoriesenvandtests.python -m pytest tests \
  --src . \
  --exclude "[env, tests]"1.3.5 Targeted Failing TestsName:--failing-listType:OptionalDefault:all the failing tests in the test suite.Technically, FauxPy can perform fault localization
when the project at hand has multiple bugs, revealed
by different failing tests. How fault localization techniques
behave in this situation requires further research.
In any case, FauxPy can be configured to only consider
certain failing tests during a fault localization session, providing
option--failing-list. We call these tests"targeted failing tests".
If this option is used, FauxPy only considers the targeted failing tests
while computing suspiciousness scores, avoiding the noise
caused by other bugs and their corresponding failing tests.
If this option is not used, FauxPy includes in
the fault localization session all the failing tests
in the given test suite.The argument passed to this option is a comma seperated list
of test functions in the Pytest format. A test in Pytest is identified
by its path, class name, and function name in the following format:[FILE_PATH]::[CLASS_NAME]::[FUNCTION_NAME]For example, Pytest identifies a test function namedtest_read_file, whose
relative path istest/test_common/test_file.py,
and is inside classTest_IOas follows:test/test_common/test_file.py::Test_IO::test_read_fileIf the targeted failing test is parametrized, the
parameter part of the name must be
removed while it is passed to--failing-list.
For instance, if testtest/test_tuils.py::test_addition[1-2-3]is a targeted failing test,test/test_tuils.py::test_additionmust be
one of the elements in the list passed to--failing-list.FauxPy provides another option--failing-file.
The functionality of this option is the same as
that of--failing-list; but option--failing-filetakes a path to a file that includes the targeted failing
tests, each in a separate line.1.3.6 Top n ResultsName:--top-nChoices:{-1} ∪ Integer[1, Inf]Type:OptionalDefault:-1Sometimes the list of program entities FauxPy outputs has too many records.
In this case, option--top-ncan be used to shorten the list.
This option takes an integer greater than 0, indicating the number of records
that FauxPy is expected to include in its output.
Passing -1, the default option, means FauxPy must return all the records.1.3.7 Test-suite SubsetFauxPy allows users to run the whole test suite
in a fault localization session, or just a subset of it such as
a certain test package, test module, test function
or even a combination of them.
For instance, the following command runs FauxPy while using only tests
in packagetests/package_x, moduletests/test_y.py,
and test functiontests/test_z.py::test_function_t.
In this case, FauxPy only considers these tests in the
fault localization session.python -m pytest tests/package_x \
                 tests/test_y.py \
                 tests/test_z.py::test_function_t \
                 --src code1.4 GuidelinesAs the first step of every fault localization session, FauxPy
runs the whole test suite passed to it.
Unlike SBFL and MBFL techniques, the two families ST and PS
only use failing tests to perform fault localization.
These two techniques simply ignore the information provided
by running passing tests.
Thus, it is better to run PS and ST using only the targeted failing
tests in the test suite. In this way, FauxPy performs more efficiently;
however, the result is going to be the same even if the passing tests are also
involved. Section1.3.7 Test-suite Subsetexplains
how the test suite size can be controlled.1.5 LimitationsBoth SBFL and MBFL families useCoverage.py,
through its API,
to collect execution
traces. So, any limitation of Coverage.py is a
limitation of these two families.The Stack Trace (ST) family does not work properly
if Pytest's option--tb=nativeis used.
Do not use--tb=nativeneither as a command line argument
nor as included in apytest.inifile.FauxPy is not compatible withpytest-sugar.
So, in order to use FauxPy,pytest-sugarmust be uninstalled
(runpip uninstall pytest-sugar) from your Python environment.1.6 CitationsIn the following paper, we used FauxPy
to conduct an empirical
study of fault localization:An Empirical Study of Fault Localization in Python Programs:@misc{Rezaalipour:2023,
      title={An Empirical Study of Fault Localization in Python Programs}, 
      author={Mohammad Rezaalipour and Carlo A. Furia},
      year={2023},
      eprint={2305.19834},
      archivePrefix={arXiv},
      primaryClass={cs.SE}
}The following repository contains the replication package of
this paper. This replication package contains 540 bash scripts,
each running FauxPy on a real-world program.
You can run some of them to see FauxPy's behaviour
on real-world programs (see Section3. Running
the experimentsin
the following repository).https://github.com/atom-sw/fauxpy-experiments2. MirrorsThe current repository is a public mirror of
our internal private repository.
We have two public mirrors, which are as follows:https://github.com/atom-sw/fauxpyhttps://github.com/mohrez86/fauxpy