Repository: avocado-framework/avocado

Issue:
Allow users to set a custom identifier format per runner
**Is your feature request related to a problem? Please describe.**

With the new runner architecture, currently we don't have an "easy" way to change the test identifier without using some "advanced" features like "yaml2mux", for instance.

Users that have a wrapper as an entry point for the tests will always have the same wrapper as test id. For instance, imagine a Makefile with some targets ('foo', 'bar') and each target is one test. Having a single test suite with those tests will make avocado print something like this:

```
JOB ID     : b6e5bdf2c891382bbde7f24e906a168af351154a
JOB LOG    : /home/local/avocado/job-results/job-2021-09-24T17.39-b6e5bdf/job.log
 (kubernetes-CRI_CONTAINERD_K8S-1/2) make: STARTED
 (kubernetes-CRI_CONTAINERD_K8S-1/2) make: PASS (2.72 s)
 (kubernetes-CRI_CONTAINERD_K8S-2/2) make: STARTED
 (kubernetes-CRI_CONTAINERD_K8S-2/2) make: PASS (0.02 s)
RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/local/avocado/job-results/job-2021-09-24T17.39-b6e5bdf/results.html
JOB TIME   : 5.49 s
```

This is happening because avocado is using the 'uri' as identifier in our current Runners. 

(Yes, we could use a multiplexer plugin for that, but some users are not so familiar with those plugins.)

**Describe the solution you'd like**

Would be nice to have a way to configure how tests should be identifier so results could be easly distinguishable. For instance:

```
JOB ID     : b6e5bdf2c891382bbde7f24e906a168af351154a
JOB LOG    : /home/local/avocado/job-results/job-2021-09-24T17.39-b6e5bdf/job.log
 (kubernetes-CRI_CONTAINERD_K8S-1/2) /usr/bin/make-foo: STARTED
 (kubernetes-CRI_CONTAINERD_K8S-1/2) /usr/bin/make-foo: PASS (2.72 s)
 (kubernetes-CRI_CONTAINERD_K8S-2/2) /usr/bin/make-bar: STARTED
 (kubernetes-CRI_CONTAINERD_K8S-2/2) /usr/bin/make-bar: PASS (0.02 s)
RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/local/avocado/job-results/job-2021-09-24T17.39-b6e5bdf/results.html
JOB TIME   : 5.49 s
```

**Describe alternatives you've considered**

After discussing this with @clebergnu, one possible solution, would be to allow users to configure an "identifier format" per test runner. Something like:

```python
config = {'runner.exectest.identifier_format = "%u-%a"'}
```

This would allow the exectest runner use "UID-ARGS" as identifier. In this example, this would be translated to `/usr/bin/make-foo`, `/usr/bin/make-bar`.
 
**Additional information**

Most of the runners could have "%u" as default identifier.



Discussion:
@beraldoleal first of all, thanks for working on this requirement!

Context: I'm doing some experiments to see if Avocado can solve some problems on Kata Containers project.

The tests are executed by a kind of job script and you can think of evocations to `make <target>` are conceptually equivalent to run  individual tests. The evocation of those make commands aren't unfortunately so simple: there are something like `/usr/bin/sudo -E PATH=$PATH CRI_RUNTIME=containerd bash -c 'make <target>'`. Therefore, using the `"%u-%a"` formatter doesn't help either as the result will be a bit "ugly" (assuming that avocado will concatenate the arguments somehow). Yes I can try to simplify those evocations, but still ugly.

Another issue is to distinguish two tests which have the same command and arguments but different environment variables (changing the behavior of the tests).

IMHO it needs more thoughts. :)
@wainersm, Thanks for the reply. I missed some information in the description.

%u and %a are just examples, for sure we need a bigger table of "special variables". ENV variables could be supported as well. 

One think that we discussed it was adding support to something like this "%a[6-7]". Where, in your example would result in something like "make target".
> @wainersm, Thanks for the reply. I missed some information in the description.
> 
> %u and %a are just examples, for sure we need a bigger table of "special variables". ENV variables could be supported as well.
> 
> One think that we discussed it was adding support to something like this "%a[6-7]". Where, in your example would result in something like "make target".

hello @beraldoleal , got it, having the flexibility to choose so specifics arguments should work out.
Thanks!

Resolving patch:
diff --git a/avocado/core/__init__.py b/avocado/core/__init__.py
--- a/avocado/core/__init__.py
+++ b/avocado/core/__init__.py
@@ -188,6 +188,14 @@ def register_core_options():
                          key_type=list,
                          help_msg=help_msg)
 
+    help_msg = ('By default Avocado runners will use the {uri} of a test as '
+                'its identifier. Use a custom f-string identifier in order to '
+                'change it.')
+    stgs.register_option(section='runner',
+                         key='identifier_format',
+                         default="{uri}",
+                         help_msg=help_msg)
+
     help_msg = 'List of test references (aliases or paths)'
     stgs.register_option(section='resolver',
                          key='references',
diff --git a/avocado/core/nrunner.py b/avocado/core/nrunner.py
--- a/avocado/core/nrunner.py
+++ b/avocado/core/nrunner.py
@@ -131,6 +131,57 @@ def __repr__(self):
                           self.kwargs, self.tags, self.requirements,
                           self.variant)
 
+    @property
+    def identifier(self):
+        """Runnable identifier respecting user's format string.
+
+        This is still experimental and we have room for improvements.
+
+        This property it will return an unique identifier for this runnable.
+        Please use this property in order to respect user's customization.
+
+        By default runnables has its '{uri}' as identifier.
+
+        Custom formatter can be configured and currently we accept the
+        following values as normal f-strings replacements: {uri}, {args},
+        and {kwargs}. "args" and "kwargs" are special cases.
+
+        For args, since it is a list, you can use in two different ways:
+        "{args}" for the entire list, or "{args[n]}" for a specific element
+        inside this list.  The same is valid when using "{kwargs}". With
+        kwargs, since it is a dictionary, you have to specify a key as index
+        and then the values are used. For instance if you have a kwargs value
+        named 'DEBUG', a valid usage could be: "{kwargs[DEBUG]}" and this will
+        print the current value to this variable (i.e: True or False).
+
+        Since this is formatter, combined values can be used. Example:
+        "{uri}-{args}".
+        """
+        fmt = self.config.get("runner.identifier_format")
+
+        # For the cases where there is no config (when calling the Runnable
+        # directly
+        if not fmt:
+            return self.uri
+
+        # For args we can use the entire list of arguments or with a specific
+        # index.
+        args = '-'.join(self.args)
+        if 'args' in fmt and '[' in fmt:
+            args = self.args
+
+        # For kwargs we can use the entire list of values or with a specific
+        # index.
+        kwargs = '-'.join(self.kwargs.values())
+        if 'kwargs' in fmt and '[' in fmt:
+            kwargs = self.kwargs
+
+        options = {'uri': self.uri,
+                   'args': args,
+                   'kwargs': kwargs}
+
+        return fmt.format(**options)
+
     @classmethod
     def from_args(cls, args):
         """Returns a runnable from arguments"""
diff --git a/avocado/plugins/runner_nrunner.py b/avocado/plugins/runner_nrunner.py
--- a/avocado/plugins/runner_nrunner.py
+++ b/avocado/plugins/runner_nrunner.py
@@ -218,7 +218,7 @@ def _create_runtime_tasks_for_test(self, test_suite, runnable, no_digits,
         else:
             prefix = index
         test_id = TestID(prefix,
-                         runnable.uri,
+                         runnable.identifier,
                          variant,
                          no_digits)
         # inject variant on runnable


Test patch:
diff --git a/selftests/unit/test_nrunner.py b/selftests/unit/test_nrunner.py
--- a/selftests/unit/test_nrunner.py
+++ b/selftests/unit/test_nrunner.py
@@ -70,6 +70,12 @@ def test_recipe_exec(self):
         self.assertEqual(runnable.args, ("/etc/profile",))
         self.assertEqual(runnable.kwargs, {"TERM": "vt3270"})
 
+    def test_identifier_args(self):
+        config = {'runner.identifier_format': '{uri}-{args[0]}'}
+        runnable = nrunner.Runnable('exec-text', 'uri', 'arg1', 'arg2',
+                                    config=config)
+        self.assertEqual(runnable.identifier, 'uri-arg1')
+
     def test_runnable_command_args(self):
         runnable = nrunner.Runnable('noop', 'uri', 'arg1', 'arg2')
         actual_args = runnable.get_command_args()