Repository: avocado-framework/avocado

Issue:
Job API run.external_runner not working
with command line no problem :

```
$ avocado run 2 --external-runner /bin/sleep
JOB ID     : b9232acb0f8847c470f520067b88c3ad403908a9
JOB LOG    : /home/jarichte/avocado/job-results/job-2020-10-21T12.11-b9232ac/job.log
 (1/1) 2: PASS (2.02 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/jarichte/avocado/job-results/job-2020-10-21T12.11-b9232ac/results.html
JOB TIME   : 3.30 s
```
Job API not working:

```
from avocado.core.job import Job                                                
from avocado.core.suite import TestSuite                                        
                                                                                 
config = {                                                                      
     'run.external_runner': '/bin/sleep',                                        
     'run.references': ['2']                                                     
}                                                                               
                                                                                 
suite = TestSuite.from_config(config)                                           
                                                                                 
with Job(config, [suite]) as j:                                                 
     sys.exit(j.run())

```

```
$ python test.py 
Traceback (most recent call last):
  File "/home/jarichte/Avocado/avocado/avocado/core/suite.py", line 99, in _from_config_with_loader
    tests = loader.discover(references,
  File "/home/jarichte/Avocado/avocado/avocado/core/loader.py", line 256, in discover
    raise LoaderUnhandledReferenceError(unhandled_references,
avocado.core.loader.LoaderUnhandledReferenceError: Unable to resolve reference(s) '3' with plugins(s) 'file', 'tap', 'external', try running 'avocado -V list 3' to see the details.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 9, in <module>
    suite = TestSuite.from_config(config)
  File "/home/jarichte/Avocado/avocado/avocado/core/suite.py", line 299, in from_config
    suite = cls._from_config_with_loader(config, name)
  File "/home/jarichte/Avocado/avocado/avocado/core/suite.py", line 109, in _from_config_with_loader
    raise TestSuiteError(details)
avocado.core.suite.TestSuiteError: Unable to resolve reference(s) '3' with plugins(s) 'file', 'tap', 'external', try running 'avocado -V list 3' to see the details.
```




Discussion:
Regarding the use of the `external-runner`. Using a reference as an argument to the external runner while trying to execute `avocado run` works in the command-line, but I feel it is counterintuitive. IMHO, a reference should be something tangible and not an argument for another test executor. I would like to hear other opinions here.

Another situation I found is related to the reference resolution. I created a bash script with the following content:

```bash
#!/bin/bash

echo "This is a test!"
```

Then I tried to run it with `/bin/bash` as the external runner and it failed just like the example here:

```python
import sys

from avocado.core.job import Job
from avocado.core.suite import TestSuite

config = {
    'run.external_runner': '/bin/bash',
    'run.references': ['/tmp/test.sh']
}

suite = TestSuite.from_config(config)

with Job(config, [suite]) as j:
    sys.exit(j.run())
```

```
[wrampazz@wrampazz ~]$ python3 /tmp/test.py 
Traceback (most recent call last):
  File "/home/wrampazz/src/avocado/avocado.dev/avocado/core/suite.py", line 99, in _from_config_with_loader
    tests = loader.discover(references,
  File "/home/wrampazz/src/avocado/avocado.dev/avocado/core/loader.py", line 256, in discover
    raise LoaderUnhandledReferenceError(unhandled_references,
avocado.core.loader.LoaderUnhandledReferenceError: Unable to resolve reference(s) '/tmp/test.sh' with plugins(s) 'file', 'tap', 'external', try running 'avocado -V list /tmp/test.sh' to see the details.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/test.py", line 11, in <module>
    suite = TestSuite.from_config(config)
  File "/home/wrampazz/src/avocado/avocado.dev/avocado/core/suite.py", line 299, in from_config
    suite = cls._from_config_with_loader(config, name)
  File "/home/wrampazz/src/avocado/avocado.dev/avocado/core/suite.py", line 109, in _from_config_with_loader
    raise TestSuiteError(details)
avocado.core.suite.TestSuiteError: Unable to resolve reference(s) '/tmp/test.sh' with plugins(s) 'file', 'tap', 'external', try running 'avocado -V list /tmp/test.sh' to see the details.
```

As stated by the error, the reference was not resolved:

```
[wrampazz@wrampazz ~]$ avocado -V list /tmp/test.sh
Type       Test                                                                                                                     Tag(s)
NOT_A_TEST /tmp/test.sh: Not an INSTRUMENTED (avocado.Test based), PyUNITTEST (unittest.TestCase based) or SIMPLE (executable) test
!GOLANG    /tmp/test.sh: No test matching this reference.
!ROBOT     /tmp/test.sh: Unsupported file format 'sh'.
NOT_A_TEST /tmp/test.sh: Not a supported test

TEST TYPES SUMMARY
==================
!golang: 1
!robot: 1
not_a_test: 2
```

But, if I make `test.sh` executable, Avocado can resolve it as a `Simple` test.

```
[wrampazz@wrampazz ~]$ chmod u+x /tmp/test.sh

[wrampazz@wrampazz ~]$ avocado -V list /tmp/test.sh
Type    Test                                           Tag(s)
SIMPLE  /tmp/test.sh
!GOLANG /tmp/test.sh: No test matching this reference.
!ROBOT  /tmp/test.sh: Unsupported file format 'sh'.
TAP     /tmp/test.sh

TEST TYPES SUMMARY
==================
!golang: 1
!robot: 1
simple: 1
tap: 1
```

And, the test using the job API works because Avocado was able to resolve the reference:

```
[wrampazz@wrampazz ~]$ python3 /tmp/test.py
JOB ID     : 9897e94c1a41c5e4219d2a404e712d603d8fcac9
JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-10-21T15.13-9897e94/job.log
 (f7f6a7b7-e8d0-4fd7-ace9-59f94b84154d-1/1) /tmp/test.sh: PASS (0.01 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
```

So, I think the main issue here is how and where the reference resolution happens using the command-line and the Job API and less related to the `external.runner` namespace.

@willianrampazzo Thanks for the investigation, I just create an issue and didn't investigate yet.

> Regarding the use of the `external-runner`. Using a reference as an argument to the external runner while trying to execute `avocado run` works in the command-line, but I feel it is counterintuitive. IMHO, a reference should be something tangible and not an argument for another test executor. I would like to hear other opinions here.

In this, I don't see a problem. I understand the reference as an argument passed to the `runner` and it doesn't matter if it's a path or number or string, if the runner understands it. And if change this behavior, we have to solve how to pass parameters to `external runners` if it's needed.

> 
> Another situation I found is related to the reference resolution. I created a bash script with the following content:
> 
> ```shell
> #!/bin/bash
> 
> echo "This is a test!"
> ```
> 
> Then I tried to run it with `/bin/bash` as the external runner and it failed just like the example here:
> 
> ```python
> import sys
> 
> from avocado.core.job import Job
> from avocado.core.suite import TestSuite
> 
> config = {
>     'run.external_runner': '/bin/bash',
>     'run.references': ['/tmp/test.sh']
> }
> 
> suite = TestSuite.from_config(config)
> 
> with Job(config, [suite]) as j:
>     sys.exit(j.run())
> ```
> 
> ```
> [wrampazz@wrampazz ~]$ python3 /tmp/test.py 
> Traceback (most recent call last):
>   File "/home/wrampazz/src/avocado/avocado.dev/avocado/core/suite.py", line 99, in _from_config_with_loader
>     tests = loader.discover(references,
>   File "/home/wrampazz/src/avocado/avocado.dev/avocado/core/loader.py", line 256, in discover
>     raise LoaderUnhandledReferenceError(unhandled_references,
> avocado.core.loader.LoaderUnhandledReferenceError: Unable to resolve reference(s) '/tmp/test.sh' with plugins(s) 'file', 'tap', 'external', try running 'avocado -V list /tmp/test.sh' to see the details.
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>   File "/tmp/test.py", line 11, in <module>
>     suite = TestSuite.from_config(config)
>   File "/home/wrampazz/src/avocado/avocado.dev/avocado/core/suite.py", line 299, in from_config
>     suite = cls._from_config_with_loader(config, name)
>   File "/home/wrampazz/src/avocado/avocado.dev/avocado/core/suite.py", line 109, in _from_config_with_loader
>     raise TestSuiteError(details)
> avocado.core.suite.TestSuiteError: Unable to resolve reference(s) '/tmp/test.sh' with plugins(s) 'file', 'tap', 'external', try running 'avocado -V list /tmp/test.sh' to see the details.
> ```
> 
> As stated by the error, the reference was not resolved:
> 
> ```
> [wrampazz@wrampazz ~]$ avocado -V list /tmp/test.sh
> Type       Test                                                                                                                     Tag(s)
> NOT_A_TEST /tmp/test.sh: Not an INSTRUMENTED (avocado.Test based), PyUNITTEST (unittest.TestCase based) or SIMPLE (executable) test
> !GOLANG    /tmp/test.sh: No test matching this reference.
> !ROBOT     /tmp/test.sh: Unsupported file format 'sh'.
> NOT_A_TEST /tmp/test.sh: Not a supported test
> 
> TEST TYPES SUMMARY
> ==================
> !golang: 1
> !robot: 1
> not_a_test: 2
> ```
> 
> But, if I make `test.sh` executable, Avocado can resolve it as a `Simple` test.
> 
> ```
> [wrampazz@wrampazz ~]$ chmod u+x /tmp/test.sh
> 
> [wrampazz@wrampazz ~]$ avocado -V list /tmp/test.sh
> Type    Test                                           Tag(s)
> SIMPLE  /tmp/test.sh
> !GOLANG /tmp/test.sh: No test matching this reference.
> !ROBOT  /tmp/test.sh: Unsupported file format 'sh'.
> TAP     /tmp/test.sh
> 
> TEST TYPES SUMMARY
> ==================
> !golang: 1
> !robot: 1
> simple: 1
> tap: 1
> ```
> 
> And, the test using the job API works because Avocado was able to resolve the reference:
> 
> ```
> [wrampazz@wrampazz ~]$ python3 /tmp/test.py
> JOB ID     : 9897e94c1a41c5e4219d2a404e712d603d8fcac9
> JOB LOG    : /home/wrampazz/avocado/job-results/job-2020-10-21T15.13-9897e94/job.log
>  (f7f6a7b7-e8d0-4fd7-ace9-59f94b84154d-1/1) /tmp/test.sh: PASS (0.01 s)
> RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> ```
> 
> So, I think the main issue here is how and where the reference resolution happens using the command-line and the Job API and less related to the `external.runner` namespace.


> @willianrampazzo Thanks for the investigation, I just create an issue and didn't investigate yet.

I thought this was the case. I did an initial investigation because I was curious about the behavior for different kinds of references.

> 
> > Regarding the use of the `external-runner`. Using a reference as an argument to the external runner while trying to execute `avocado run` works in the command-line, but I feel it is counterintuitive. IMHO, a reference should be something tangible and not an argument for another test executor. I would like to hear other opinions here.
> 
> In this, I don't see a problem. I understand the reference as an argument passed to the `runner` and it doesn't matter if it's a path or number or string, if the runner understands it. And if change this behavior, we have to solve how to pass parameters to `external runners` if it's needed.
> 

Okay, I think `avocado-vt` also has a use case for references different than real files. Now that I thought more about it, I got the point, that's fine.


Resolving patch:
diff --git a/avocado/core/loader.py b/avocado/core/loader.py
--- a/avocado/core/loader.py
+++ b/avocado/core/loader.py
@@ -117,14 +117,17 @@ def _str_loaders():
                                        ", ".join(_good_test_types(plugin)))
             return out.rstrip('\n')
 
-        subcommand = config.get('subcommand')
+        # When running from the JobAPI there is no subcommand
+        subcommand = config.get('subcommand') or 'run'
         self.register_plugin(TapLoader)
         # Register external runner when --external-runner is used
         external_runner = config.get("{}.external_runner".format(subcommand))
         if external_runner:
             self.register_plugin(ExternalLoader)
             key = "{}.loaders".format(subcommand)
-            if set(config[key]) != {'file', '@DEFAULT'}:
+            default_loaders = {'file', '@DEFAULT'}
+            loaders = config.get(key) or default_loaders
+            if set(loaders) != default_loaders:
                 warnings.warn("The loaders and external-runner are incompatible."
                               "The values in loaders will be ignored.",
                               RuntimeWarning)


Test patch: