Repository: avocado-framework/avocado

Issue:
[JobAPI] testlog plugin is not triggered by the Job API
With the following test code:

```python
from avocado import Test


class PassTest(Test):

    def test(self):
        """
        A test simply doesn't have to fail in order to pass
        """
        self.fail()
```

and the following config:

```
[job.output.testlogs]
statuses = ["FAIL", "ERROR"]
```

The testlog plugin works fine:

```
$ avocado --config=testlog.conf run passtest.py 
JOB ID     : 338f3c5b52e7905e3cabf0557f14e34b4859628a
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-12-03T18.25-338f3c5/job.log
 (1/1) passtest.py:PassTest.test: FAIL: None (0.04 s)
RESULTS    : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/wrampazz/avocado/job-results/job-2020-12-03T18.25-338f3c5/results.html
JOB TIME   : 0.53 s
Log content for test "1-passtest.py:PassTest.test" (FAIL)
18:25:23 ERROR| 
18:25:23 ERROR| Reproduced traceback from: /home/wrampazz/src/avocado/avocado.quick-dev/avocado/core/test.py:767
18:25:23 ERROR| Traceback (most recent call last):
18:25:23 ERROR|   File "/home/wrampazz/src/avocado/custom/passtest.py", line 10, in test
18:25:23 ERROR|     self.fail()
18:25:23 ERROR|   File "/home/wrampazz/src/avocado/avocado.quick-dev/avocado/core/test.py", line 953, in fail
18:25:23 ERROR|     raise exceptions.TestFail(message)
18:25:23 ERROR| avocado.core.exceptions.TestFail: None
18:25:23 ERROR| 
18:25:23 DEBUG| Local variables:
18:25:23 DEBUG|  -> self <class 'passtest.PassTest'>: 1-passtest.py:PassTest.test
18:25:23 DEBUG| DATA (filename=output.expected) => NOT FOUND (data sources: variant, test, file)
18:25:23 DEBUG| DATA (filename=stdout.expected) => NOT FOUND (data sources: variant, test, file)
18:25:23 DEBUG| DATA (filename=stderr.expected) => NOT FOUND (data sources: variant, test, file)
18:25:23 ERROR| FAIL 1-passtest.py:PassTest.test -> TestFail: None
18:25:23 INFO | 
```

But when using the Job API, the test log does not get triggered. Test code:

```python
#!/usr/bin/env python3
import sys

from avocado.core.job import Job

job_config = {'run.references': ['/bin/false'],
              'job.output.testlogs.statuses': ['FAIL', 'ERROR']}

with Job.from_config(job_config=job_config) as job:
    sys.exit(job.run())
```

Output:

```
$ python3 passjob.py 
JOB ID     : 290cb185b882a90bdbbe3cf54134e85bd29674ca
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-12-03T18.35-290cb18/job.log
 (1-1/1) /bin/false: FAIL: Exited with status: '1', stdout: '' stderr: '' (0.05 s)
RESULTS    : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/wrampazz/avocado/job-results/job-2020-12-03T18.35-290cb18/results.html
```

I expected the same behavior as the command-line.


Discussion:
@willianrampazzo not sure if I agree or understand. Maybe I'm missing something.

But in your first case, the test is your class. That is calling `self.fail()` with an exception expected. In your second case, your test is `/bin/false` that has no output, just an exit code != 0.

What I'm missing? What would be your expected output?

Here I did the tests, and the outputs are "similar/consistent".

With your first example:

```bash
(avocado-new) [local@laptop avocado]$ avocado --config /tmp/avocado.cfg run /tmp/first.py 
JOB ID     : 358ff57819aaeea6ee00c0ceaa14f1e0d136201a
JOB LOG    : /home/local/avocado/job-results/job-2020-12-04T09.49-358ff57/job.log
 (1/1) /tmp/first.py:PassTest.test: FAIL: None (0.03 s)
RESULTS    : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 0.32 s
```

And with your second example:

```bash
(avocado-new) [local@laptop avocado]$ python3 /tmp/second.py 
JOB ID     : 5cf3bb1ea771e4fab1879b4fa48f5fc0bed89003
JOB LOG    : /home/local/avocado/job-results/job-2020-12-04T09.49-5cf3bb1/job.log
 (1-1/1) /bin/false: FAIL: Exited with status: '1', stdout: '' stderr: '' (0.05 s)
RESULTS    : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0

``` 


Okay, mixing different tests does not help, sorry.

Here is a consistent example:

```python
#!/usr/bin/env python3
import sys

from avocado import Test
from avocado.core.job import Job

class PassTest(Test):

    def test(self):
        """
        A test simply doesn't have to fail in order to pass
        """
        self.fail()

def main():
    job_config = {'run.references': ['%s:PassTest' % __file__],
                  'job.output.testlogs.statuses': ['FAIL', 'ERROR']}

    with Job.from_config(job_config=job_config) as job:
        return job.run()

if __name__ == '__main__':
    sys.exit(main())
```

Still, I have the same result:

```
$ python3 passjob.py 
JOB ID     : 481bd544b5c90d31ea1cf3ad8e9928d3c3f56620
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-12-04T10.00-481bd54/job.log
 (1-1/1) /home/wrampazz/src/avocado/custom/passjob.py:PassTest.test: FAIL: None (0.04 s)
RESULTS    : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/wrampazz/avocado/job-results/job-2020-12-04T10.00-481bd54/results.html
```
Just to make sure, I did another test running `/bin/false` with the testlog enabled and it shows the log in the output:

```
$ avocado --config=testlog.conf run /bin/false
JOB ID     : e90d7cc35dc82e3739165fe6dfb4fc94b8552f3e
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-12-04T10.10-e90d7cc/job.log
 (1/1) /bin/false: FAIL: Exited with status: '1', stdout: '' stderr: '' (0.05 s)
RESULTS    : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/wrampazz/avocado/job-results/job-2020-12-04T10.10-e90d7cc/results.html
JOB TIME   : 0.60 s
Log content for test "1-/bin/false" (FAIL)
10:10:31 INFO | Running '/bin/false'
10:10:31 INFO | Command '/bin/false' finished with 1 after 0.0008797645568847656s
10:10:31 INFO | Exit status: 1
10:10:31 INFO | Duration: 0.0008797645568847656
10:10:31 ERROR| 
10:10:31 ERROR| Reproduced traceback from: /home/wrampazz/src/avocado/avocado.quick-dev/avocado/core/test.py:767
10:10:31 ERROR| Traceback (most recent call last):
10:10:31 ERROR|   File "/home/wrampazz/src/avocado/avocado.quick-dev/avocado/core/test.py", line 1109, in _execute_cmd
10:10:31 ERROR|     result = process.run(self._command,
10:10:31 ERROR|   File "/home/wrampazz/src/avocado/avocado.quick-dev/avocado/utils/process.py", line 1037, in run
10:10:31 ERROR|     raise CmdError(cmd, sp.result)
10:10:31 ERROR| avocado.utils.process.CmdError: Command '/bin/false' failed.
10:10:31 ERROR| stdout: b''
10:10:31 ERROR| stderr: b''
10:10:31 ERROR| additional_info: None
10:10:31 ERROR| 
10:10:31 ERROR| During handling of the above exception, another exception occurred:
10:10:31 ERROR| 
10:10:31 ERROR| Traceback (most recent call last):
10:10:31 ERROR|   File "/home/wrampazz/src/avocado/avocado.quick-dev/avocado/core/test.py", line 1156, in test
10:10:31 ERROR|     self._execute_cmd()
10:10:31 ERROR|   File "/home/wrampazz/src/avocado/avocado.quick-dev/avocado/core/test.py", line 1118, in _execute_cmd
10:10:31 ERROR|     raise exceptions.TestFail(test_failure)
10:10:31 ERROR| avocado.core.exceptions.TestFail: Exited with status: '1', stdout: '' stderr: ''
10:10:31 ERROR| 
10:10:31 DEBUG| Local variables:
10:10:31 DEBUG|  -> self <class 'avocado.core.test.SimpleTest'>: 1-/bin/false
10:10:31 DEBUG| DATA (filename=output.expected) => NOT FOUND (data sources: variant, file)
10:10:31 DEBUG| DATA (filename=stdout.expected) => NOT FOUND (data sources: variant, file)
10:10:31 DEBUG| DATA (filename=stderr.expected) => NOT FOUND (data sources: variant, file)
10:10:31 ERROR| FAIL 1-/bin/false -> TestFail: Exited with status: '1', stdout: '' stderr: ''
10:10:31 INFO | 
```
I got you.

I guess the job is being halted before "post" be executed. Take a look at the "JOB TIME" line. It is missing. 
> I got you.
> 
> I guess the job is being halted before "post" be executed. Take a look at the "JOB TIME" line. It is missing.

Good catch; I did not notice it.

Resolving patch:
diff --git a/avocado/core/job.py b/avocado/core/job.py
--- a/avocado/core/job.py
+++ b/avocado/core/job.py
@@ -549,6 +549,9 @@ def run(self):
             self.time_start = time.time()
         try:
             self.result.tests_total = self.size
+            pre_post_dispatcher = dispatcher.JobPrePostDispatcher()
+            output.log_plugin_failures(pre_post_dispatcher.load_failures)
+            pre_post_dispatcher.map_method('pre', self)
             self.pre_tests()
             return self.run_tests()
         except exceptions.JobBaseException as details:
@@ -582,6 +585,7 @@ def run(self):
                 self.time_end = time.time()
                 self.time_elapsed = self.time_end - self.time_start
             self.render_results()
+            pre_post_dispatcher.map_method('post', self)
 
     def run_tests(self):
         """
diff --git a/avocado/plugins/run.py b/avocado/plugins/run.py
--- a/avocado/plugins/run.py
+++ b/avocado/plugins/run.py
@@ -19,8 +19,7 @@
 import argparse
 import sys
 
-from avocado.core import exit_codes, job, loader, output, parser_common_args
-from avocado.core.dispatcher import JobPrePostDispatcher
+from avocado.core import exit_codes, job, loader, parser_common_args
 from avocado.core.output import LOG_UI
 from avocado.core.plugin_interfaces import CLICmd, Init
 from avocado.core.settings import settings
@@ -302,15 +301,4 @@ def run(self, config):
             LOG_UI.error(err)
             sys.exit(exit_codes.AVOCADO_JOB_FAIL)
         with job.Job(config, [suite]) as job_instance:
-            pre_post_dispatcher = JobPrePostDispatcher()
-            try:
-                # Run JobPre plugins
-                output.log_plugin_failures(pre_post_dispatcher.load_failures)
-                pre_post_dispatcher.map_method('pre', job_instance)
-
-                job_run = job_instance.run()
-            finally:
-                # Run JobPost plugins
-                pre_post_dispatcher.map_method('post', job_instance)
-
-        return job_run
+            return job_instance.run()


Test patch: