Repository: avocado-framework/avocado

Issue:
make check missing some results
When you run make check the avocado job will stop without results from some tests: 
```
make check
...
 (check-883/998) selftests/functional/plugin/test_diff.py:DiffTests.test_diff: PASS (6.32 s)
 (check-894/998) selftests/lint.sh: STARTED
 (check-895/998) selftests/modules-boundaries.sh: STARTED
 (check-896/998) selftests/inspekt-style.sh: STARTED
 (check-895/998) selftests/modules-boundaries.sh: PASS (0.18 s)
 (check-884/998) selftests/functional/plugin/test_diff.py:DiffTests.test_diff_nocmdline: PASS (6.48 s)
 (check-898/998) selftests/inspekt-indent.sh: STARTED
 (check-897/998) selftests/signedoff-check.sh: STARTED
 (check-897/998) selftests/signedoff-check.sh: PASS (0.04 s)
 (check-899/998) selftests/spell.sh: STARTED
 (check-900/998) selftests/isort.sh: STARTED
 (check-901/998) optional_plugins/robot/tests/test_robot.py:Loader.test_discover: STARTED
 (check-901/998) optional_plugins/robot/tests/test_robot.py:Loader.test_discover: PASS (0.21 s)
RESULTS    : PASS 894 | ERROR 0 | FAIL 0 | SKIP 104 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/jarichte/avocado/job-results/job-2021-05-14T15.23-b692048/results.html
JOB TIME   : 505.51 s
```
You can see that tests like `lint.sh`, `inspekt-style.sh`, `inspekt-indent.sh`, `spell.sh` and `isort.sh` started but never finished. And the last test is with index 901, but we have 998 tests, the summary results are correct, but we're missing the information about those 97 tests.



Discussion:
I thought it would be a good idea to debug using the static checks only, so I stripped the `check.py` file:

```python
#!/usr/bin/env python3

import argparse
import glob
import sys

from avocado.core.job import Job
from avocado.core.suite import TestSuite


def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument('--disable-static-checks',
                        help='Disable the static checks (isort, lint, etc)',
                        action='store_true')
    return parser.parse_args()


def create_suites(args):
    suites = []
    config_check = {
        'run.references': [],
        'run.test_runner': 'nrunner',
        'run.ignore_missing_references': True,
        'job.output.testlogs.statuses': ['FAIL']
    }

    if not args.disable_static_checks:
        config_check['run.references'] += glob.glob('selftests/*.sh')

    suites.append(TestSuite.from_config(config_check, "check"))
    return suites


def print_failed_tests(tests):
    if tests:
        print("Failed tests:")
        for test in tests:
            print(test.get('name'), test.get('status'))


def main():
    args = parse_args()
    suites = create_suites(args)
    config = {'core.show': ['app'],
              'run.test_runner': 'nrunner'}
    with Job(config, suites) as j:
        exit_code = j.run()
    print_failed_tests(j.get_failed_tests())
    return exit_code


if __name__ == '__main__':
    sys.exit(main())
```

But, this works as expected:

```
JOB ID     : e7e9847b392c086656afc22468bf69b5124dc57a
JOB LOG    : /home/wrampazz/avocado/job-results/job-2021-05-14T12.24-e7e9847/job.log
 (check-4/7) selftests/spell.sh: STARTED
 (check-2/7) selftests/inspekt-style.sh: STARTED
 (check-3/7) selftests/lint.sh: STARTED
 (check-1/7) selftests/isort.sh: STARTED
 (check-5/7) selftests/signedoff-check.sh: STARTED
 (check-6/7) selftests/inspekt-indent.sh: STARTED
 (check-5/7) selftests/signedoff-check.sh: PASS (0.01 s)
 (check-7/7) selftests/modules-boundaries.sh: STARTED
 (check-7/7) selftests/modules-boundaries.sh: PASS (0.07 s)
 (check-1/7) selftests/isort.sh: PASS (0.25 s)
 (check-6/7) selftests/inspekt-indent.sh: PASS (4.88 s)
 (check-2/7) selftests/inspekt-style.sh: PASS (21.74 s)
 (check-3/7) selftests/lint.sh: FAIL (24.64 s)
 (check-4/7) selftests/spell.sh: PASS (25.93 s)
RESULTS    : PASS 6 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/wrampazz/avocado/job-results/job-2021-05-14T12.24-e7e9847/results.html
JOB TIME   : 27.46 s
Failed tests:
check-3-selftests/lint.sh FAIL
selftests/check_tmp_dirs
No temporary avocado dirs lying around in /tmp
```
I cant reproduce. And the issue seems to be related with 40c84848a1aa6fc3c41ad51d62e477cf25b2df1f.

Still doing some investigation here...

Resolving patch:
diff --git a/selftests/check.py b/selftests/check.py
--- a/selftests/check.py
+++ b/selftests/check.py
@@ -9,6 +9,7 @@
 from avocado.core import exit_codes
 from avocado.core.job import Job
 from avocado.core.suite import TestSuite
+from avocado.utils.network.ports import find_free_port
 
 BOOLEAN_ENABLED = [True, 'true', 'on', 1]
 BOOLEAN_DISABLED = [False, 'false', 'off', 0]
@@ -497,11 +498,14 @@ def create_suites(args):
     # ========================================================================
     # Run all static checks, unit and functional tests
     # ========================================================================
+    status_server = '127.0.0.1:%u' % find_free_port()
     config_check = {
         'run.references': ['selftests/jobs/',
                            'selftests/unit/',
                            'selftests/functional/'],
         'run.test_runner': 'nrunner',
+        'nrunner.status_server_listen': status_server,
+        'nrunner.status_server_uri': status_server,
         'run.ignore_missing_references': True,
         'job.output.testlogs.statuses': ['FAIL']
     }


Test patch: