Repository: avocado-framework/avocado

Issue:
Job: allow introspection of suites and selection of suites for execution
An Avocado job contains test suites that usually group tests together either for categorization or for configuration purposes.  In either case, and especially in the "categorization" example, it may be useful to only run a subset of a job's suites.

This use case can be seen in Avocado itself in the `selftests/check.py` script, which could benefit from, instead of adding command line options for including/excluding suites, would get that information straight from the job's suite.

Definition of done:

1. an API that allows users to list the suites (may already be satisfied by the `test_suites` attribute, or may add another API that returns suite names)
2. an API that allows the job to selectively execute some of the suites (preferably by using the test suite's name(s))


Discussion:
@clebergnu I was thinking about this, and I think 1 is already covered by `job.test_suites`.

With regards 2: what do you think that we could have a `suite.enabled = True` (by default) flag? (That would be part of the suite's config dict as well. (Or even `job.run.disabled`). In that way we could filter the test_suites that has this enabled or disabled.


@clebergnu, about 1, IMO I would not push a different property/object just to get the names here.

Each suite has already a name, delegating this to the Job class would increase the coupling level between classes. 

```python
[ suite.name for suite in job.test_suites ]
```
> @clebergnu, about 1, IMO I would not push a different property/object just to get the names here.
> 
> Each suite has already a name, delegating this to the Job class would increase the coupling level between classes.
> 
> ```python
> [ suite.name for suite in job.test_suites ]
> ```

I'm fine with this.
@clebergnu what about 2? my first comment.
> @clebergnu I was thinking about this, and I think 1 is already covered by `job.test_suites`.
> 
> With regards 2: what do you think that we could have a `suite.enabled = True` (by default) flag? (That would be part of the suite's config dict as well. (Or even `job.run.disabled`). In that way we could filter the test_suites that has this enabled or disabled.

Yes, this is exactly what this issue is about.  About the API, I slightly prefer the `suite.enabled` approach.

But the real answer, IMO, will come from trying to hook it into `check.py`.
Let me expand a bit on the application of this feature (the mention about hooking it into `check.py`).  There are many projects that use custom scripts or `Makefiles` to manage their test selection and execution, and that includes projects that use Avocado.

For instance, this is QEMU's `make check-help`:

```
$ make check-help
Regression testing targets:
 make check                Run block, qapi-schema, unit, softfloat, qtest and decodetree tests
 make bench                Run speed tests

Individual test suites:
 make check-qtest-TARGET   Run qtest tests for given target
 make check-qtest          Run qtest tests
 make check-unit           Run qobject tests
 make check-qapi-schema    Run QAPI schema tests
 make check-block          Run block tests
 make check-tcg            Run TCG tests
 make check-softfloat      Run FPU emulation tests
 make check-acceptance     Run acceptance (functional) tests for currently configured targets

 make check-report.tap     Generates an aggregated TAP test report
 make check-venv           Creates a Python venv for tests
 make check-clean          Clean the tests and related data

The following are useful for CI builds
 make check-build          Build most test binaris
 make get-vm-images        Downloads all images used by acceptance tests, according to configured targets (~350 MB each, 1.5 GB max)


The variable SPEED can be set to control the gtester speed setting.
Default options are -k and (for make V=1) --verbose; they can be
changed with variable GTESTER_OPTIONS.
```

Most of these targets are subset of tests which map well into a Job's test suites.  The opportunity here is to have either job code examples and documentation that show how a script can be easily written that allows for the selection of suites.  The user may have to deal with `argparse`, but it would be awesome if that could be made as lightly of a use as possible.

Resolving patch:
diff --git a/avocado/core/suite.py b/avocado/core/suite.py
--- a/avocado/core/suite.py
+++ b/avocado/core/suite.py
@@ -84,10 +84,11 @@ def resolutions_to_runnables(resolutions, config):
 
 class TestSuite:
     def __init__(self, name, config=None, tests=None, job_config=None,
-                 resolutions=None):
+                 resolutions=None, enabled=True):
         self.name = name
         self.tests = tests
         self.resolutions = resolutions
+        self.enabled = enabled
 
         # Create a complete config dict with all registered options + custom
         # config
diff --git a/avocado/plugins/runner_nrunner.py b/avocado/plugins/runner_nrunner.py
--- a/avocado/plugins/runner_nrunner.py
+++ b/avocado/plugins/runner_nrunner.py
@@ -325,6 +325,10 @@ def _abort_if_missing_runners(runnables):
     def run_suite(self, job, test_suite):
         summary = set()
 
+        if not test_suite.enabled:
+            job.interrupted_reason = f"Suite {test_suite.name} is disabled."
+            return summary
+
         test_suite.tests, missing_requirements = nrunner.check_runnables_runner_requirements(
             test_suite.tests)
         self._abort_if_missing_runners(missing_requirements)
diff --git a/examples/jobs/passjob_custom.py b/examples/jobs/passjob_custom.py
--- a/examples/jobs/passjob_custom.py
+++ b/examples/jobs/passjob_custom.py
@@ -13,6 +13,8 @@
 
 suite1 = TestSuite(name='suite1', tests=[Runnable("noop", "noop")])
 suite2 = TestSuite(name='suite2', tests=[Runnable("noop", "noop")])
+suite3 = TestSuite(name='suite3', enabled=False,
+                   tests=[Runnable("noop", "noop")])
 
-with Job(test_suites=[suite1, suite2]) as j:
+with Job(test_suites=[suite1, suite2, suite3]) as j:
     sys.exit(j.run())


Test patch: