Repository: avocado-framework/avocado

Issue:
Categorize the test results data by status
**Is your feature request related to a problem? Please describe.**
It is not clear which tests passed and which failed by merely looking at the test results data dir. Consider a local development environment (aka not a cloud-based CI infrastructure). Now, consider a scenario where test results are exposed as some kind of artifacts which are then presented to a developer, e.g. as a tarball. Without looking at `results.tap` (or its XML/JSON equivalents) it is not possible for the developer to tell which tests actually failed by merely looking at the `test-results` directory.

**Describe the solution you'd like**
It would be great if the directory hierarchy could be enhanced by using by a `status` directory containing further subdirectories mapping to the  `FAIL`, `PASS`, `ERROR`,etc. statuses. These status directories would then contain symlinks to the actual test results data. See the example below:

```
$ ls -lR avocado/job-results/latest/
...

avocado/job-results/latest/by-status:            
total 4                                                             
drwxr-xr-x. 2 test test  4096 Dec 17 17:25 FAIL                      
                                                                        
avocado/job-results/latest/by-status/FAIL:   
total 8                                                                       
lrwxrwxrwx. 1 test test 72 Dec 17 17:25 1-_bin_false -> avocado/job-results/latest/test-results/1-_bin_false
lrwxrwxrwx. 1 test test 72 Dec 17 17:25 3-_bin_false -> avocado/job-results/latest/test-results/3-_bin_false

...

avocado/job-results/latest/test-results:
total 12
drwxr-xr-x. 2 test test 4096 Dec 17 17:05 1-_bin_false
drwxr-xr-x. 2 test test  4096 Dec 17 17:05 2-_bin_true
drwxr-xr-x. 2 test test  4096 Dec 17 17:05 3-_bin_false

...
```

**Describe alternatives you've considered**
Given the scenario above, I have not found a suitable alternative.



Discussion:
@eskultety, thanks for reporting this. Just one consideration...

Although I like to browse the directories, over the last few months we've noticed that this model has some limitations and may not be as attractive for some users. For this reason, we create the `avocado show jobs` command.

So I would like to ask if you consider an extra option in this command as a valid option. Ex: `avocado jobs show latest --filter-by-status=FAIL` (or something like this).

I'm not excluding the initial ideal (or maybe both), but we need to consider if it's really necessary, as IMO accessing this through the interface can make it more attractive and also keeping two snippets of code is something we should consider.
> @eskultety, thanks for reporting this. Just one consideration...
> 
> Although I like to browse the directories, over the last few months we've noticed that this model has some limitations and may not be as attractive for some users. For this reason, we create the `avocado show jobs` command.
> 
> So I would like to ask if you consider an extra option in this command as a valid option. Ex: `avocado jobs show latest --filter-by-status=FAIL` (or something like this).
> 
> I'm not excluding the initial ideal (or maybe both), but we need to consider if it's really necessary, as IMO accessing this through the interface can make it more attractive and also keeping two snippets of code is something we should consider.

The problem is how such functionality would be consumed by non interactive systems.  Think of GitLab CI: you can not, post job execution, based on the output of a command or logic, tell it which results to keep as artifacts.  The location must be ready to be consumed based on a previous static job configuration (on the YAML file).
Thanks @clebergnu I got the user case, I missed that on the description. Makes sense. 
I should have been more exact in the description, @clebergnu  described exactly the use case I had in mind.


> I should have been more exact in the description, @clebergnu described exactly the use case I had in mind.

You were very precise. My fault for not having read it right. 

Resolving patch:
diff --git a/avocado/plugins/bystatus.py b/avocado/plugins/bystatus.py
new file mode 100644
--- /dev/null
+++ b/avocado/plugins/bystatus.py
@@ -0,0 +1,44 @@
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See LICENSE for more details.
+#
+# Copyright: Red Hat Inc. 2022
+# Authors: Beraldo Leal <bleal@redhat.com>
+
+import os
+
+from avocado.core.plugin_interfaces import ResultEvents
+
+
+class ByStatusLink(ResultEvents):
+    description = "Creates symlinks on file system grouped by status"
+
+    def __init__(self, config):
+        pass
+
+    def pre_tests(self, job):
+        pass
+
+    def post_tests(self, job):
+        pass
+
+    def start_test(self, result, state):
+        pass
+
+    def test_progress(self, progress=False):
+        pass
+
+    def end_test(self, result, state):
+        link = state.get('logdir')
+        where = os.path.join(os.path.dirname(link),
+                             'by-status',
+                             state.get('status'))
+        os.makedirs(where, exist_ok=True)
+        os.symlink(link, os.path.join(where, os.path.basename(link)))
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -382,6 +382,7 @@ def run(self):
                   'fetchasset = avocado.plugins.assets:FetchAssetJob',
                   'sysinfo = avocado.plugins.sysinfo:SysInfoJob',
                   'testlogging = avocado.plugins.testlogs:TestLogging',
+                  'bystatus = avocado.plugins.bystatus:ByStatusLink',
                   'beaker = avocado.plugins.beaker_result:BeakerResult',
                   ],
               'avocado.plugins.varianter': [


Test patch:
diff --git a/avocado/core/test.py b/avocado/core/test.py
--- a/avocado/core/test.py
+++ b/avocado/core/test.py
@@ -33,7 +33,7 @@
 import warnings
 from difflib import unified_diff
 
-from avocado.core import exceptions, output, parameters
+from avocado.core import exceptions, parameters
 from avocado.core.output import LOG_JOB
 from avocado.core.settings import settings
 from avocado.core.test_id import TestID
@@ -276,18 +276,6 @@ def record_and_warn(*args, **kwargs):
             self.__base_logdir_tmp = tempfile.TemporaryDirectory(prefix=prefix)
             self.__base_logdir = self.__base_logdir_tmp.name
 
-        if self._config.get("run.test_runner") == 'nrunner':
-            logdir = self.__base_logdir
-            self.__logdir = logdir
-        else:
-            self.__base_logdir = os.path.join(self.__base_logdir, 'test-results')
-            logdir = os.path.join(self.__base_logdir, self.name.str_filesystem)
-
-            if os.path.exists(logdir):
-                raise exceptions.TestSetupFail(f"Log dir already exists, "
-                                               f"this should never happen: "
-                                               f"{logdir}")
-            self.__logdir = utils_path.init_dir(logdir)
         self.__logfile = os.path.join(self.logdir, 'debug.log')
 
         self._stdout_file = os.path.join(self.logdir, 'stdout')
@@ -302,9 +290,6 @@ def record_and_warn(*args, **kwargs):
         self.__log_warn_used = False
         self.log.warn = self.log.warning = record_and_warn
 
-        # Initialized by _start_logging and terminated by _stop_logging
-        self._file_handler = None
-
         self.log.info('INIT %s', self.name)
 
         paths = ['/test/*']
@@ -382,7 +367,7 @@ def logdir(self):
         """
         Path to this test's logging dir
         """
-        return self.__logdir
+        return self.__base_logdir
 
     @property
     def logfile(self):
@@ -584,63 +569,6 @@ def _register_log_file_handler(self, logger, formatter, filename,
         logger.addHandler(file_handler)
         self._logging_handlers[logger.name] = file_handler
 
-    def _start_logging(self):
-        """
-        Simple helper for adding a file logger to the main logger.
-        """
-        self._file_handler = logging.FileHandler(filename=self.logfile)
-        self._file_handler.setLevel(logging.DEBUG)
-
-        fmt = '%(asctime)s %(levelname)-5.5s| %(message)s'
-        formatter = logging.Formatter(fmt=fmt, datefmt='%H:%M:%S')
-
-        self._file_handler.setFormatter(formatter)
-        self.log.addHandler(self._file_handler)
-        self.log.propagate = False
-
-        # Adding the test log FileHandler to the Avocado's main logger so
-        # that everything logged while the test is running, for every logger,
-        # also makes its way into the test log file
-        logger = logging.getLogger('avocado')
-        logger.addHandler(self._file_handler)
-
-        stream_fmt = '%(message)s'
-        stream_formatter = logging.Formatter(fmt=stream_fmt)
-
-        log_test_stdout = LOG_JOB.getChild("stdout")
-        log_test_stderr = LOG_JOB.getChild("stderr")
-        log_test_output = LOG_JOB.getChild("output")
-
-        self._register_log_file_handler(log_test_stdout,
-                                        stream_formatter,
-                                        self._stdout_file,
-                                        raw=True)
-        self._register_log_file_handler(log_test_stderr,
-                                        stream_formatter,
-                                        self._stderr_file,
-                                        raw=True)
-        self._register_log_file_handler(log_test_output,
-                                        stream_formatter,
-                                        self._output_file,
-                                        raw=True)
-
-        if isinstance(sys.stdout, output.LoggingFile):
-            sys.stdout.add_logger(log_test_stdout)
-        if isinstance(sys.stderr, output.LoggingFile):
-            sys.stderr.add_logger(log_test_stderr)
-
-    def _stop_logging(self):
-        """
-        Stop the logging activity of the test by cleaning the logger handlers.
-        """
-        self.log.removeHandler(self._file_handler)
-        if isinstance(sys.stderr, output.LoggingFile):
-            sys.stderr.rm_logger(LOG_JOB.getChild("stderr"))
-        if isinstance(sys.stdout, output.LoggingFile):
-            sys.stdout.rm_logger(LOG_JOB.getChild("stdout"))
-        for name, handler in self._logging_handlers.items():
-            logging.getLogger(name).removeHandler(handler)
-
     def _check_reference(self, produced_file_path, reference_file_name,
                          diff_file_name, child_log_name, name='Content'):
         '''
@@ -703,8 +631,6 @@ def _run_test(self):
         """
         self._tag_start()
         testMethod = getattr(self, self._testMethodName)
-        if self._config.get("run.test_runner") != 'nrunner':
-            self._start_logging()
         skip_test_condition = getattr(testMethod, '__skip_test_condition__', False)
         skip_test_condition_negate = getattr(testMethod, '__skip_test_condition_negate__', False)
         if skip_test_condition:
@@ -839,8 +765,6 @@ def run_avocado(self):
         self._tag_end()
         self._report()
         self.log.info("")
-        if self._config.get("run.test_runner") != 'nrunner':
-            self._stop_logging()
 
     def _report(self):
         """
diff --git a/selftests/functional/plugin/test_bystatus.py b/selftests/functional/plugin/test_bystatus.py
new file mode 100644
--- /dev/null
+++ b/selftests/functional/plugin/test_bystatus.py
@@ -0,0 +1,23 @@
+import json
+import os
+
+from avocado.utils import process
+from selftests.utils import AVOCADO, TestCaseTmpDir
+
+
+class ByStatusTest(TestCaseTmpDir):
+
+    def test_symlinks(self):
+        result = process.run(f"{AVOCADO} run "
+                             f"--job-results-dir {self.tmpdir.name} "
+                             f"--disable-sysinfo --json - "
+                             f"/bin/true /bin/false", ignore_status=True)
+        res = json.loads(result.stdout_text)
+
+        for test in res['tests']:
+            logdir = test['logdir']
+            status = test['status']
+            where = os.path.dirname(logdir)
+            basename = os.path.basename(logdir)
+            link = os.path.join(where, 'by-status', status, basename)
+            self.assertTrue(os.path.exists(os.readlink(link)))
diff --git a/selftests/functional/test_export_variables.py b/selftests/functional/test_export_variables.py
--- a/selftests/functional/test_export_variables.py
+++ b/selftests/functional/test_export_variables.py
@@ -8,17 +8,11 @@
 
 SCRIPT_CONTENT = (f"""#!/bin/sh
 echo "Avocado Version: $AVOCADO_VERSION"
-echo "Avocado Test basedir: $AVOCADO_TEST_BASEDIR"
 echo "Avocado Test workdir: $AVOCADO_TEST_WORKDIR"
-echo "Avocado Test logdir: $AVOCADO_TEST_LOGDIR"
-echo "Avocado Test logfile: $AVOCADO_TEST_LOGFILE"
 echo "Avocado Test outputdir: $AVOCADO_TEST_OUTPUTDIR"
 
 test "$AVOCADO_VERSION" = "{VERSION}" -a \
-     -d "$AVOCADO_TEST_BASEDIR" -a \
      -d "$AVOCADO_TEST_WORKDIR" -a \
-     -d "$AVOCADO_TEST_LOGDIR" -a \
-     -f "$AVOCADO_TEST_LOGFILE" -a \
      -d "$AVOCADO_TEST_OUTPUTDIR"
 """)
 
@@ -36,7 +30,7 @@ def setUp(self):
     def test_environment_vars(self):
         os.chdir(BASEDIR)
         cmd_line = (f'{AVOCADO} run --job-results-dir {self.tmpdir.name} '
-                    f'{self.script.path} --test-runner=runner')
+                    f'{self.script.path}')
         result = process.run(cmd_line, ignore_status=True)
         expected_rc = exit_codes.AVOCADO_ALL_OK
         self.assertEqual(result.exit_status, expected_rc,
diff --git a/selftests/functional/test_job_timeout.py b/selftests/functional/test_job_timeout.py
--- a/selftests/functional/test_job_timeout.py
+++ b/selftests/functional/test_job_timeout.py
@@ -106,18 +106,20 @@ def test_sleep_longer_timeout(self):
                     f'examples/tests/passtest.py')
         self.run_and_check(cmd_line, 0, 2, 0, 0, 0)
 
+    @unittest.skip("Job timeout is failing with nrunner, until we fix: #5295")
     def test_sleep_short_timeout(self):
         cmd_line = (f'{AVOCADO} run --job-results-dir {self.tmpdir.name} '
-                    f'--disable-sysinfo --test-runner=runner --xunit - '
+                    f'--disable-sysinfo --xunit - '
                     f'--job-timeout=1 {self.script.path} '
                     f'examples/tests/passtest.py')
         self.run_and_check(cmd_line, exit_codes.AVOCADO_JOB_INTERRUPTED,
                            2, 1, 0, 1)
         self._check_timeout_msg(1)
 
+    @unittest.skip("Job timeout is failing with nrunner, until we fix: #5295")
     def test_sleep_short_timeout_with_test_methods(self):
         cmd_line = (f'{AVOCADO} run --job-results-dir {self.tmpdir.name} '
-                    f'--disable-sysinfo --test-runner=runner --xunit - '
+                    f'--disable-sysinfo --xunit - '
                     f'--job-timeout=1 {self.py.path}')
         self.run_and_check(cmd_line, exit_codes.AVOCADO_JOB_INTERRUPTED,
                            3, 1, 0, 2)