Repository: boxed/mutmut

Issue:
Add option to take coverage contexts into account
``coverage.py`` is able to capture [dynamic contexts](https://coverage.readthedocs.io/en/coverage-5.5/contexts.html#dynamic-contexts), which means it can track which test exercised which parts of the code under test.

As ``mutmut`` is already able to read the coverage database, it could significantly improve the execution times if it would take this context data into account and only run the tests that actually cover the mutated line.

If no context was recorded or the context is not the name of a test function (e.g. a user-defined static context for example), ``mutmut`` should run the whole test suite as a fallback. 
This feature should probably also be optional. 

Discussion:

Patch:
diff --git a/HISTORY.rst b/HISTORY.rst
index e2c35e3..0415fb9 100644
--- a/HISTORY.rst
+++ b/HISTORY.rst
@@ -12,6 +12,9 @@ Changelog
   to check which parameters are relevant to this specific subcommand. The change is backwards compatible, and all existing commands
   work the same as before, with the exception of ``mutmut --version``, which now has to be ``mutmut version``.
 
+* You can now set the ``context.config.test_command`` in the ``mutmut_config.pre_mutation(context)`` hook to select the relevant subset of tests.
+  Coverage contexts are now accessible in ``config.coverage_data`` to help with the selection.
+
 2.2.0
 ~~~~~
 
diff --git a/README.rst b/README.rst
index baf9301..542711c 100644
--- a/README.rst
+++ b/README.rst
@@ -191,6 +191,154 @@ Note that ``--disable-mutation-types`` and ``--enable-mutation-types`` are exclu
 be combined.
 
 
+Selecting tests to run
+----------------------
+
+If you have a large test suite or long running tests, it can be beneficial to narrow the set of tests to
+run for each mutant down to the tests that have a chance of killing it.
+Determining the relevant subset of tests depends on your project, its structure, and the metadata that you
+know about your tests.
+``mutmut`` provides information like the file to mutate and `coverage contexts <https://coverage.readthedocs.io/en/coverage-5.5/contexts.html>`_
+(if used with the ``--use-coverage`` switch).
+You can set the ``context.config.test_command`` in the ``pre_mutation(context)`` hook of ``mutmut_config.py``.
+The ``test_command`` is reset after each mutant, so you don't have to explicitly (re)set it for each mutant.
+
+This section gives examples to show how this could be done for some concrete use cases.
+All examples use the default test runner (``python -m pytest -x --assert=plain``).
+
+Selection based on source and test layout
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If the location of the test module has a strict correlation with your source code layout, you can simply
+construct the path to the corresponding test file from ``context.filename``.
+Suppose your layout follows the following structure where the test file is always located right beside the
+production code:
+
+.. code-block:: console
+
+    mypackage
+    ├── production_module.py
+    ├── test_production_module.py
+    └── subpackage
+        ├── submodule.py
+        └── test_submodule.py
+
+Your ``mutmut_config.py`` in this case would look like this:
+
+.. code-block:: python
+
+    import os.path
+
+    def pre_mutation(context):
+        dirname, filename = os.path.split(context.filename)
+        testfile = "test_" + filename
+        context.config.test_command += ' ' + os.path.join(dirname, testfile)
+
+Selection based on imports
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you can't rely on the directory structure or naming of the test files, you may assume that the tests most likely
+to kill the mutant are located in test files that directly import the module that is affected by the mutant.
+Using the ``ast`` module of the Python standard library, you can use the ``init()`` hook to build a map which test file
+imports which module, and then lookup all test files importing the mutated module and only run those:
+
+.. code-block:: python
+
+    import ast
+    from pathlib import Path
+
+    test_imports = {}
+
+
+    class ImportVisitor(ast.NodeVisitor):
+        """Visitor which records which modules are imported."""
+        def __init__(self) -> None:
+            super().__init__()
+            self.imports = []
+
+        def visit_Import(self, node: ast.Import) -> None:
+            for alias in node.names:
+                self.imports.append(alias.name)
+
+        def visit_ImportFrom(self, node: ast.ImportFrom) -> None:
+            self.imports.append(node.module)
+
+
+    def init():
+        """Find all test files located under the 'tests' directory and create an abstract syntax tree for each.
+        Let the ``ImportVisitor`` find out what modules they import and store the information in a global dictionary
+        which can be accessed by ``pre_mutation(context)``."""
+        test_files = (Path(__file__).parent / "tests").rglob("test*.py")
+        for fpath in test_files:
+            visitor = ImportVisitor()
+            visitor.visit(ast.parse(fpath.read_bytes()))
+            test_imports[str(fpath)] = visitor.imports
+
+
+    def pre_mutation(context):
+        """Construct the module name from the filename and run all test files which import that module."""
+        tests_to_run = []
+        for testfile, imports in test_imports.items():
+            module_name = context.filename.rstrip(".py").replace("/", ".")
+            if module_name in imports:
+                tests_to_run.append(testfile)
+        context.config.test_command += f"{' '.join(tests_to_run)}"
+
+Selection based on coverage contexts
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you recorded `coverage contexts <https://coverage.readthedocs.io/en/coverage-5.5/contexts.html>`_ and use
+the ``--use-coverage`` switch, you can access this coverage data inside the ``pre_mutation(context)`` hook
+via the ``context.config.coverage_data`` attribute. This attribute is a dictionary in the form
+``{filename: {lineno: [contexts]}}``.
+
+Let's say you have used the built-in dynamic context option of ``Coverage.py `` by adding the following to
+your ``.coveragerc`` file:
+
+.. code-block:: console
+
+    [run]
+    dynamic_context = test_function
+
+``coverage`` will create a new context for each test function that you run in the form ``module_name.function_name``.
+With ``pytest``, we can use the ``-k`` switch to filter tests that match a given expression.
+
+.. code-block:: python
+    import os.path
+
+    def pre_mutation(context):
+        """Extract the coverage contexts if possible and only run the tests matching this data."""
+        if not context.config.coverage_data:
+            # mutmut was run without ``--use-coverage``
+            return
+        fname = os.path.abspath(context.filename)
+        contexts_for_file = context.config.coverage_data.get(fname, {})
+        contexts_for_line = contexts_for_file.get(context.current_line_index, [])
+        test_names = [
+            ctx.rsplit(".", 1)[-1]  # extract only the final part after the last dot, which is the test function name
+            for ctx in contexts_for_line
+            if ctx  # skip empty strings
+        ]
+        if not test_names:
+            return
+        context.config.test_command += f' -k "{" or ".join(test_names)}"'
+
+Pay attention that the format of the context name varies depending on the tool you use for creating the contexts.
+For example, the ``pytest-cov`` plugin uses ``::`` as separator between module and test function.
+Furthermore, not all tools are able to correctly pick up the correct contexts. ``coverage.py`` for instance is (at the time of writing)
+unable to pick up tests that are inside a class when using ``pytest``.
+You will have to inspect your ``.coverage`` database using the `Coverage.py API <https://coverage.readthedocs.io/en/coverage-5.5/api.html>`_
+first to determine how you can extract the correct information to use with your test runner.
+
+Making things more robust
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Despite your best efforts in picking the right subset of tests, it may happen that the mutant survives because the test which is able
+to kill it was not included in the test set. You can tell ``mutmut`` to re-run the full test suite in that case, to verify that this
+mutant indeed survives.
+You can do so by passing the ``--rerun-all`` option to ``mutmut run``. This option is disabled by default.
+
+
 JUnit XML support
 -----------------
 
diff --git a/mutmut/__init__.py b/mutmut/__init__.py
index 71e2c0b..003b15a 100644
--- a/mutmut/__init__.py
+++ b/mutmut/__init__.py
@@ -782,6 +782,10 @@ def run_mutation(context: Context, callback) -> str:
         start = time()
         try:
             survived = tests_pass(config=config, callback=callback)
+            if survived and config.test_command != config._default_test_command and config.rerun_all:
+                # rerun the whole test suite to be sure the mutant can not be killed by other tests
+                config.test_command = config._default_test_command
+                survived = tests_pass(config=config, callback=callback)
         except TimeoutError:
             return BAD_TIMEOUT
 
@@ -798,6 +802,7 @@ def run_mutation(context: Context, callback) -> str:
 
     finally:
         move(context.filename + '.bak', context.filename)
+        config.test_command = config._default_test_command  # reset test command to its default in the case it was altered in a hook
 
         if config.post_mutation:
             result = subprocess.check_output(config.post_mutation, shell=True).decode().strip()
@@ -810,9 +815,9 @@ class Config(object):
                  baseline_time_elapsed, test_time_multiplier, test_time_base,
                  dict_synonyms, total, using_testmon,
                  tests_dirs, hash_of_tests, pre_mutation, post_mutation,
-                 coverage_data, paths_to_mutate, mutation_types_to_apply, no_progress):
+                 coverage_data, paths_to_mutate, mutation_types_to_apply, no_progress, rerun_all):
         self.swallow_output = swallow_output
-        self.test_command = test_command
+        self.test_command = self._default_test_command = test_command
         self.covered_lines_by_filename = covered_lines_by_filename
         self.baseline_time_elapsed = baseline_time_elapsed
         self.test_time_multipler = test_time_multiplier
@@ -828,6 +833,7 @@ class Config(object):
         self.paths_to_mutate = paths_to_mutate
         self.mutation_types_to_apply = mutation_types_to_apply
         self.no_progress = no_progress
+        self.rerun_all = rerun_all
 
 
 def tests_pass(config: Config, callback) -> bool:
@@ -1173,7 +1179,8 @@ def run_mutation_tests(config, progress, mutations_by_file):
 
 def read_coverage_data():
     """
-    :rtype: CoverageData or None
+    Reads the coverage database and returns a dictionary which maps the filenames to the covered lines and their contexts.
+    :rtype: dict[str, dict[int, list[str]]]
     """
     try:
         # noinspection PyPackageRequirements,PyUnresolvedReferences
@@ -1183,7 +1190,7 @@ def read_coverage_data():
     cov = Coverage('.coverage')
     cov.load()
     data = cov.get_data()
-    return {filepath: data.lines(filepath) for filepath in data.measured_files()}
+    return {filepath: data.contexts_by_lineno(filepath) for filepath in data.measured_files()}
 
 
 def read_patch_data(patch_file_path):
diff --git a/mutmut/__main__.py b/mutmut/__main__.py
index add5330..09b161c 100644
--- a/mutmut/__main__.py
+++ b/mutmut/__main__.py
@@ -106,6 +106,9 @@ def version():
 @click.option('--runner')
 @click.option('--use-coverage', is_flag=True, default=False)
 @click.option('--use-patch-file', help='Only mutate lines added/changed in the given patch file')
+@click.option('--rerun-all', is_flag=True, default=False, help='If you modified the test_command in the pre_mutation hook, '
+                                                               'the default test_command (specified by the "runner" option) '
+                                                               'will be executed if the mutant survives with your modified test_command.')
 @click.option('--tests-dir')
 @click.option('-m', '--test-time-multiplier', default=2.0, type=float)
 @click.option('-b', '--test-time-base', default=0.0, type=float)
@@ -127,7 +130,7 @@ def version():
 def run(argument, paths_to_mutate, disable_mutation_types, enable_mutation_types, runner,
         tests_dir, test_time_multiplier, test_time_base, swallow_output, use_coverage,
         dict_synonyms, pre_mutation, post_mutation, use_patch_file, paths_to_exclude,
-        simple_output, no_progress):
+        simple_output, no_progress, rerun_all):
     """
     Runs mutmut. You probably want to start with just trying this. If you supply a mutation ID mutmut will check just this mutant.
     """
@@ -139,7 +142,7 @@ def run(argument, paths_to_mutate, disable_mutation_types, enable_mutation_types
     sys.exit(do_run(argument, paths_to_mutate, disable_mutation_types, enable_mutation_types, runner,
                     tests_dir, test_time_multiplier, test_time_base, swallow_output, use_coverage,
                     dict_synonyms, pre_mutation, post_mutation, use_patch_file, paths_to_exclude,
-                    simple_output, no_progress))
+                    simple_output, no_progress, rerun_all))
 
 
 @climain.command(context_settings=dict(help_option_names=['-h', '--help']))
@@ -237,7 +240,7 @@ def html(dict_synonyms):
 def do_run(argument, paths_to_mutate, disable_mutation_types,
            enable_mutation_types, runner, tests_dir, test_time_multiplier, test_time_base,
            swallow_output, use_coverage, dict_synonyms, pre_mutation, post_mutation,
-           use_patch_file, paths_to_exclude, simple_output, no_progress):
+           use_patch_file, paths_to_exclude, simple_output, no_progress, rerun_all):
     """return exit code, after performing an mutation test run.
 
     :return: the exit code from executing the mutation tests
@@ -387,7 +390,8 @@ Legend for output:
         post_mutation=post_mutation,
         paths_to_mutate=paths_to_mutate,
         mutation_types_to_apply=mutation_types_to_apply,
-        no_progress=no_progress
+        no_progress=no_progress,
+        rerun_all=rerun_all
     )
 
     parse_run_argument(argument, config, dict_synonyms, mutations_by_file, paths_to_exclude, paths_to_mutate, tests_dirs)


Test patch:
diff --git a/tests/test_main.py b/tests/test_main.py
index f968a0d..586900d 100644
--- a/tests/test_main.py
+++ b/tests/test_main.py
@@ -318,6 +318,51 @@ def test_full_run_no_surviving_mutants_junit(filesystem):
     assert int(root.attrib['disabled']) == 0
 
 
+def test_mutant_only_killed_after_rerun(filesystem):
+    mutmut_config = filesystem / "mutmut_config.py"
+    mutmut_config.write("""
+def pre_mutation(context):
+    context.config.test_command = "echo True"
+""")
+    CliRunner().invoke(climain, ['run', '--paths-to-mutate=foo.py', "--test-time-base=15.0", "--rerun-all"], catch_exceptions=False)
+    result = CliRunner().invoke(climain, ['results'], catch_exceptions=False)
+    print(repr(result.output))
+    assert result.exit_code == 0
+    assert result.output.strip() == u"""
+To apply a mutant on disk:
+    mutmut apply <id>
+
+To show a mutant:
+    mutmut show <id>
+""".strip()
+
+
+def test_no_rerun_if_not_specified(filesystem):
+    mutmut_config = filesystem / "mutmut_config.py"
+    mutmut_config.write("""
+def pre_mutation(context):
+    context.config.test_command = "echo True"
+""")
+    CliRunner().invoke(climain, ['run', '--paths-to-mutate=foo.py', "--test-time-base=15.0"], catch_exceptions=False)
+    result = CliRunner().invoke(climain, ['results'], catch_exceptions=False)
+    print(repr(result.output))
+    assert result.exit_code == 0
+    assert result.output.strip() == u"""
+To apply a mutant on disk:
+    mutmut apply <id>
+
+To show a mutant:
+    mutmut show <id>
+
+
+Survived 🙁 (14)
+
+---- foo.py (14) ----
+
+1-14
+""".strip()
+
+
 def test_full_run_one_surviving_mutant(filesystem):
     with open(os.path.join(str(filesystem), "tests", "test_foo.py"), 'w') as f:
         f.write(test_file_contents.replace('assert foo(2, 2) is False', ''))