Repository: avocado-framework/avocado

Issue:
Support for codecoverage when running Python tests
Avocado doesn't have a documented and/or supported way for generating code coverage information when running Python based tests.

The first objective is to drop the use of the custom `selftests/run_coverage` script and the additional job, in favor of reusing one already existing CI job with Avocado:

 * running code coverage tools
 * keeping the information
 * pushing the information

Even though Python based tests and code coverage is the primary goal of this issue, it's probable that the solution will be about providing a flexible way to add a "wrapper" of sorts before the actual execution of tests.
 


Discussion:
Documentation about how to use Coverage.py with Avocado: https://github.com/avocado-framework/avocado/pull/4791
With the nrunner now being the default runner, we should either make it clear that this is a legacy runner only feature, or attempt to evolve the coverage support + nrunner.
> With the nrunner now being the default runner, we should either make it clear that this is a legacy runner only feature, or attempt to evolve the coverage support + nrunner.

I remember I have tested the documentation with nrunner, and it worked. I tested again today with a test code I have, and the documentation is still valid for avocado using nrunner.

But I agree, having native support would make things easier.
> > With the nrunner now being the default runner, we should either make it clear that this is a legacy runner only feature, or attempt to evolve the coverage support + nrunner.
> 
> I remember I have tested the documentation with nrunner, and it worked. I tested again today with a test code I have, and the documentation is still valid for avocado using nrunner.
> 
> But I agree, having native support would make things easier.

@willianrampazzo maybe then closing this issue and creating one (for longer term) about native support?
> > > With the nrunner now being the default runner, we should either make it clear that this is a legacy runner only feature, or attempt to evolve the coverage support + nrunner.
> > 
> > 
> > I remember I have tested the documentation with nrunner, and it worked. I tested again today with a test code I have, and the documentation is still valid for avocado using nrunner.
> > But I agree, having native support would make things easier.
> 
> @willianrampazzo maybe then closing this issue and creating one (for longer term) about native support?

That is a good idea. I opened https://github.com/avocado-framework/avocado/issues/4957, feel free to edit it if needed.
Hiya, I'm looking at this again because of a shortcoming with the legacy runner in 94.0; so I'm trying out the nrunner for the QEMU source tree.

First, if you check out the QEMU source tree master and do the following:
```
> cd python/
> make distclean
> make dev-venv
> source .dev-venv/bin/activate.fish  # (Or your shell of choice ...)
> pip install coverage
> make check-coverage
# Makefile invocation above is equivalent to:
# > coverage run -m avocado --config avocado.cfg run tests/*.py
# > coverage combine
# > coverage report
```
You *should* be able to see that the coverage report printed to the console shows something similar to this: (This report from my working branch, don't worry about it.) the important thing is that protocol.py is at 92%, events, message, models etc are non-zero.
```
Name                             Stmts   Miss  Cover
----------------------------------------------------
qemu/machine/__init__.py             3      3     0%
qemu/machine/console_socket.py      72     72     0%
qemu/machine/machine.py            342    342     0%
qemu/machine/qtest.py               64     64     0%
qemu/qmp/__init__.py                 8      0   100%
qemu/qmp/error.py                    5      2    60%
qemu/qmp/events.py                  80     51    36%
qemu/qmp/legacy.py                  89     89     0%
qemu/qmp/message.py                 68     42    38%
qemu/qmp/models.py                  56     36    36%
qemu/qmp/protocol.py               330     25    92%
qemu/qmp/qmp_client.py             226    155    31%
qemu/qmp/qmp_shell.py              280    280     0%
qemu/qmp/qmp_tui.py                269    269     0%
qemu/qmp/util.py                    64     21    67%
qemu/utils/__init__.py              11     11     0%
qemu/utils/accel.py                 28     28     0%
qemu/utils/qemu_ga_client.py       197    197     0%
qemu/utils/qom.py                  109    109     0%
qemu/utils/qom_common.py            64     64     0%
qemu/utils/qom_fuse.py              95     95     0%
----------------------------------------------------
TOTAL                             2460   1955    21%
```

Switching to the nrunner by doing the following:
1. edit python/avocado.cfg

```
diff --git a/python/avocado.cfg b/python/avocado.cfg
index c7722e7ecd..10dc6fb605 100644
--- a/python/avocado.cfg
+++ b/python/avocado.cfg
@@ -1,6 +1,3 @@
-[run]
-test_runner = runner
-
 [simpletests]
 # Don't show stdout/stderr in the test *summary*
 status.failure_fields = ['status']
```

and then adding a `.coveragerc` file as per the Avocado documentation: https://avocado-framework.readthedocs.io/en/94.0/guides/writer/chapters/integrating.html#coverage-py

(Note: this same config is already in setup.cfg, but just in case, I added a standalone file to try and match the avocado docs more closely.)

```
[run]
concurrency = multiprocessing
source = qemu/
parallel = true
```

and then running:

```
> coverage run -m avocado --config avocado.cfg run tests/*.py
> coverage combine
> coverage report
```

produces a report with 0% coverage.

My suspicion is that the python subprocesses being launched by the nrunner aren't using the Coverage runner, so they go uninspected. Using the nrunner, I see only one `.coverage.foo.bar.baz` file generated, whereas with using the legacy runner I'll see 30 or so of the same.

If Avocado had a hook I could configure per-each subprocess to run some custom code on start for each child, I could probably take advantage of the instructions in https://coverage.readthedocs.io/en/latest/subprocess.html in order to call `coverage.process_startup()`, but as it stands I would need to manually edit all of my tests to explicitly initialize coverage.py, I think -- or I'd need to automate some kind of editing/mucking around with the venvs to run some custom code, which seems too hacky/fragile.
We are taking a look at this.
Hi @jnsnow, I found something that is missing in our documentation but present in the [Coverage's docs](https://coverage.readthedocs.io/en/coverage-4.3.3/subprocess.html) and could help here:

https://gitlab.com/beraldoleal/qemu/-/commit/c7aeb34505aab4357421890575cc9ddd7e83d04f

This branch has 2 commits, but this one is the main one. Now, I'm getting 19% of coverage:

```
Name                             Stmts   Miss  Cover
----------------------------------------------------
qemu/aqmp/__init__.py                8      0   100%
qemu/aqmp/aqmp_tui.py              269    269     0%
qemu/aqmp/error.py                   5      2    60%
qemu/aqmp/events.py                 81     52    36%
qemu/aqmp/legacy.py                 56     56     0%
qemu/aqmp/message.py                68     42    38%
qemu/aqmp/models.py                 56     36    36%
qemu/aqmp/protocol.py              328     26    92%
qemu/aqmp/qmp_client.py            222    151    32%
qemu/aqmp/util.py                   64     21    67%
qemu/machine/__init__.py             3      3     0%
qemu/machine/console_socket.py      72     72     0%
qemu/machine/machine.py            345    345     0%
qemu/machine/qtest.py               64     64     0%
qemu/qmp/__init__.py               165    165     0%
qemu/qmp/qemu_ga_client.py         198    198     0%
qemu/qmp/qmp_shell.py              282    282     0%
qemu/qmp/qom.py                    109    109     0%
qemu/qmp/qom_common.py              63     63     0%
qemu/qmp/qom_fuse.py                95     95     0%
qemu/utils/__init__.py              11     11     0%
qemu/utils/accel.py                 28     28     0%
----------------------------------------------------
TOTAL                             2592   2090    19%

```

Acording to the Coverage' manual, we need to create 1) a `sitecustomize.py` and 2) export `COVERAGE_PROCESS_START` pointing to `.coveragerc`.

Yes, I know that this is more like a "workaround", but maybe could be one alternative while Avocado is not calling this inside the subprocess.

Let us know if this is a viable alternative for you and sorry for the inconvenience.
Yeah, there's workarounds available -- I don't think I can direct people to edit their sitecustomize.py, though!

A workaround more in my control would be to edit the tests to invoke the Coverage startup instead -- I think that's something I could at least check in to the repo. (But, well. I probably won't.)

(No worries about the inconvenience, it's thankfully nothing I was relying on in a pipeline!)
> I don't think I can direct people to edit their sitecustomize.py, though!

@jnsnow, This file it was added to `qemu/python/sitecustomize.py`. We just need it there. But of course, if we could import this on Avocado side would be better. 

Resolving patch:
diff --git a/avocado/core/runners/avocado_instrumented.py b/avocado/core/runners/avocado_instrumented.py
--- a/avocado/core/runners/avocado_instrumented.py
+++ b/avocado/core/runners/avocado_instrumented.py
@@ -1,4 +1,5 @@
 import multiprocessing
+import os
 import tempfile
 import time
 import traceback
@@ -70,11 +71,22 @@ def _run_avocado(runnable, queue):
                              }]
 
             messages.start_logging(runnable.config, queue)
+
+            if 'COVERAGE_RUN' in os.environ:
+                from coverage import Coverage
+                coverage = Coverage()
+                coverage.start()
+
             instance = loader.load_test(test_factory)
             early_state = instance.get_state()
             early_state['type'] = "early_state"
             queue.put(early_state)
             instance.run_avocado()
+
+            if 'COVERAGE_RUN' in os.environ:
+                coverage.stop()
+                coverage.save()
+
             state = instance.get_state()
             fail_reason = state.get('fail_reason')
             queue.put(messages.WhiteboardMessage.get(state['whiteboard']))


Test patch: