Repository: cocotb/cocotb

Issue:
Customization of results.xml with "factory.add_option(...)"
Hello! I'd like to suggest a feature to the results.xml file that we get at the end of a simulation.

I'm running some tests permutations with a code that looks like this:
```python
for test in [test1, test2]:
        factory = TestFactory(test)
        factory.add_option("config1",            [True, False])
        factory.add_option("config2",            ["opt-a", "opt-b", "opt-c"])
        factory.add_option("config3",            [True, False])
        factory.generate_tests()
```

Then in my results.xml, I got the list of all the test permutations with this setting.
Right now it is only possible to get the name of the test, if it has passed/failed, its simulation time, its real time, and its ratio.

Could you guys add a way to add to each test in the results.xml the value of the custom options?



Discussion:
Doesn't the name of the test in the xml file contain the option information?
When the tests are generated, they get a number after their names, and that's that.
In the example I gave, my results.xml looks like this:

```xml
<testsuites name="results">
  <testsuite name="all" package="all">
    <property name="random_seed" value="1621359523" />
    <testcase classname="module_test" file="C:\Users\henri\Anaconda3\lib\site-packages\cocotb\regression.py" lineno="604" name="test_001" ratio_time="3610.792267049259" sim_time_ns="18.001" time="0.004985332489013672" />
    <testcase classname="module_test" file="C:\Users\henri\Anaconda3\lib\site-packages\cocotb\regression.py" lineno="604" name="test_002" ratio_time="3007.7948491753646" sim_time_ns="18.001" time="0.005984783172607422" />
    <testcase classname="module_test" file="C:\Users\henri\Anaconda3\lib\site-packages\cocotb\regression.py" lineno="604" name="test_003" ratio_time="2256.070827227634" sim_time_ns="18.000999999999998" time="0.00797891616821289" />
    <testcase classname="module_test" file="C:\Users\henri\Anaconda3\lib\site-packages\cocotb\regression.py" lineno="604" name="test_004" ratio_time="4512.141654455269" sim_time_ns="18.001000000000005" time="0.003989458084106445" />
    <testcase classname="module_test" file="C:\Users\henri\Anaconda3\lib\site-packages\cocotb\regression.py" lineno="604" name="test_005" ratio_time="3591.5548617638647" sim_time_ns="18.00099999999999" time="0.005012035369873047" />
    <testcase classname="module_test" file="C:\Users\henri\Anaconda3\lib\site-packages\cocotb\regression.py" lineno="604" name="test_006" ratio_time="3610.446934965571" sim_time_ns="18.001000000000005" time="0.004985809326171875" />
    <testcase classname="module_test" file="C:\Users\henri\Anaconda3\lib\site-packages\cocotb\regression.py" lineno="604" name="test_007" ratio_time="4511.602408365702" sim_time_ns="18.001000000000005" time="0.0039899349212646484" />
    <testcase classname="module_test" file="C:\Users\henri\Anaconda3\lib\site-packages\cocotb\regression.py" lineno="604" name="test_008" ratio_time="4482.939455171596" sim_time_ns="18.001000000000005" time="0.004015445709228516" />
    <testcase classname="module_test" file="C:\Users\henri\Anaconda3\lib\site-packages\cocotb\regression.py" lineno="604" name="test_009" ratio_time="3008.3940831175" sim_time_ns="18.000999999999976" time="0.005983591079711914" />
    <testcase classname="module_test" file="C:\Users\henri\Anaconda3\lib\site-packages\cocotb\regression.py" lineno="604" name="test_010" ratio_time="3610.2742936929194" sim_time_ns="18.001000000000005" time="0.0049860477447509766" />
    <testcase classname="module_test" file="C:\Users\henri\Anaconda3\lib\site-packages\cocotb\regression.py" lineno="604" name="test_011" ratio_time="4512.141654455269" sim_time_ns="18.001000000000005" time="0.003989458084106445" />
    <testcase classname="module_test" file="C:\Users\henri\Anaconda3\lib\site-packages\cocotb\regression.py" lineno="604" name="test_012" ratio_time="3608.721264888635" sim_time_ns="18.001000000000005" time="0.004988193511962891" />
  </testsuite>
</testsuites>
``` 

After looking in line 481 of cocotb/regression.py I found the function `_log_test_summary()` that generates this file. It looks to me that the options are ignored because the generation of the XML is static.
What do pytest tests marked with `pytest.mark.parameterize` output in XML? Is there a some standard output for test parameterization in xUnit?
I'm not sure, but pytest generates much better test names of the form `test_foo[bar-baz]`. Perhaps just doing that is enough to solve the problem, by changing this last line to stringify its arguments:

https://github.com/cocotb/cocotb/blob/b862e096249f1043a1be1958bb56ce8de0fc5634/cocotb/regression.py#L706-L711
Changing this would change the test name and break people using `TestFactory` and `TESTCASE` together. Still I see no reason to not change it.
Using pytest style test names would also not work easily because test arguments are passed by kwargs and not by position. We could add code to figure out argument position, or we could just log the name using the repr of the kwargs dict: `test_name{"arg1": 4, "arg2": "sample text"}` or maybe we can make it look like the function call: `test_name(arg1=4, arg2="sample_text")`. This may be useful for supporting command-line parameterized tests in the future. I'm also wondering how this would work with `TESTCASE`. I think you would need to change `TESTCASE` to support wildcards or a regex.
Also, if we change how tests are named, the `prefix` and `postfix` arguments to `generate_tests` can be deprecated as they will serve no purpose where test names will no longer be numbered. So repeated usage of the same test with different `TestFactory`s would never have collisions. If there were cases of collisions (due to same test and same arguments) we could issue a warning and not duplicate the test.

Resolving patch:
diff --git a/cocotb/regression.py b/cocotb/regression.py
--- a/cocotb/regression.py
+++ b/cocotb/regression.py
@@ -637,6 +637,9 @@ class TestFactory:
 
     .. versionchanged:: 1.5
         Groups of options are now supported
+
+    .. versionchanged:: 2.0
+        Options are listed in test names instead of a counter
     """
 
     # Prevent warnings from collection of TestFactories by unit testing frameworks.
@@ -708,9 +711,6 @@ def generate_tests(self, prefix="", postfix=""):
                 product(*d.values())
         ):
 
-            name = "%s%s%s_%03d" % (prefix, self.name, postfix, index + 1)
-            doc = "Automatically generated test\n\n"
-
             # preprocess testoptions to split tuples
             testoptions_split = {}
             for optname, optvalue in testoptions.items():
@@ -722,6 +722,8 @@ def generate_tests(self, prefix="", postfix=""):
                     for n, v in zip(optname, optvalue):
                         testoptions_split[n] = v
 
+            doc = "Automatically generated test\n\n"
+
             for optname, optvalue in testoptions_split.items():
                 if callable(optvalue):
                     if not optvalue.__doc__:
@@ -732,15 +734,19 @@ def generate_tests(self, prefix="", postfix=""):
                 else:
                     doc += "\t{}: {}\n".format(optname, repr(optvalue))
 
-            self.log.debug("Adding generated test \"%s\" to module \"%s\"" %
-                           (name, mod.__name__))
             kwargs = {}
             kwargs.update(self.kwargs_constant)
             kwargs.update(testoptions_split)
+
+            options_str = ",".join(f"{optname}={optvalue!r}" for optname, optvalue in kwargs.items())
+            name = f"{prefix}{self.name}{postfix}({options_str})"
+
             if hasattr(mod, name):
                 self.log.error("Overwriting %s in module %s. "
                                "This causes a previously defined testcase "
                                "not to be run. Consider setting/changing "
-                               "name_postfix" % (name, mod))
+                               "the postfix option" % (name, mod))
+
+            self.log.debug(f"Adding generated test {name!r} to module {mod.__name__!r}")
             setattr(mod, name, _create_test(self.test_function, name, doc, mod,
                                             *self.args, **kwargs))


Test patch: