Repository: cocotb/cocotb

Issue:
parameterizing tests and passing arguments via the command line
There was some discussion during the hackathon regarding the need to parameterize a test and feed argument from the command line. @alexforencich brought this up, and I have needed a feature like this in the past as well. 

### Requirements
* Must support multiple tests per simulator run, like we do currently
* Should be able to pass arguments to particular tests
* Should be able to parameterize multiple tests with the same parameter
* Should have support as a make variable in the current flow

### Plusargs
Currently we have plusargs, which are also fed to the simulator. There is a limit to what can be passed in a plusargs, so this might not be the best answer. And officially they are only supported by SystemVerilog compatible simulators. GHDL, for instance, consumes them as simulator arguments. But they are supported in the make flow currently.

### Environment Variables
There is also ad-hoc argument passing via environment variables. This does work, but could be argued that it's tedium on the user that could be handled automatically. It's up to the user to decide how to handle each variable, and environment variables can be set in the current makefile flow.

### Pytest
OOTB pytest does not support this easily. According to [this](https://stackoverflow.com/questions/40880259/how-to-pass-arguments-in-pytest-by-command-line) answer it requires adding argument parser in conftest.py and then more code in the test to handle those arguments. This isn't really possible without extra work, due to inversion of control. And it requires the user to do most of the work because of the need to modify pytest's argument parser. The arguments would also need to be fed to the call to the main pytest invocation by way of environment variable. This ends up not being a very clean solution.



Discussion:
If we were to use pytest as the regression manager, we could provide a general command line option that accepts some syntax for parameterizing tests, and apply them as appropriate. Similar to how the single `--tx` option in `pytest-xdist` takes multiple forms.
>If we were to use pytest as the regression manager, we could provide a general command line option that accepts some syntax for parameterizing tests, and apply them as appropriate. Similar to how the single --tx option in pytest-xdist takes multiple forms.

Example pytest plugin I did: https://github.com/themperek/cocotb-test/blob/master/cocotb_test/plugin.py 
One can add extra options quite easily.
One more comment. 
With pytest what you can do is:

```python3
@pytest.mark.parametrize("seed", range(6))
def test_param(seed):
    pass
```

One can run such a test from the command line:
```
pytest -k test_param[1]
```
But if you would like to run with `seed==10` it is not straight forward and needs some extra code.

More: https://docs.pytest.org/en/stable/example/parametrize.html





`@cocotb.parametrize` could be made to generate tests with more interesting names than `my_test_001`.

Resolving patch:
diff --git a/src/cocotb/decorators.py b/src/cocotb/decorators.py
--- a/src/cocotb/decorators.py
+++ b/src/cocotb/decorators.py
@@ -27,21 +27,28 @@
 
 import functools
 import sys
+from enum import Enum
+from itertools import product
 from typing import (
     Any,
     Callable,
     Coroutine,
+    Dict,
+    Generic,
+    Iterable,
+    List,
     Optional,
     Sequence,
     Tuple,
     Type,
     TypeVar,
     Union,
+    cast,
     overload,
 )
 
 import cocotb
-from cocotb.regression import Test, TestFactory
+from cocotb.regression import Test
 
 Result = TypeVar("Result")
 
@@ -88,8 +95,127 @@ def wrapper(*args, **kwargs):
 F = TypeVar("F", bound=Callable[..., Coroutine[Any, Any, None]])
 
 
+class _Parameterized(Generic[F]):
+    def __init__(
+        self,
+        test_function: F,
+        options: List[
+            Union[
+                Tuple[str, Sequence[Any]], Tuple[Sequence[str], Sequence[Sequence[Any]]]
+            ]
+        ],
+    ) -> None:
+        self.test_function = test_function
+        self.options = options
+        # we are assuming the input checking is done in parameterize()
+
+        self._option_reprs: Dict[str, List[str]] = {}
+
+        for name, values in options:
+            if isinstance(name, str):
+                self._option_reprs[name] = _reprs(values)
+            else:
+                # transform to Dict[name, values]
+                transformed: Dict[str, List[Optional[str]]] = {}
+                for nam_idx, nam in enumerate(name):
+                    transformed[nam] = []
+                    for value_array in cast(Sequence[Sequence[Any]], values):
+                        value = value_array[nam_idx]
+                        transformed[nam].append(value)
+                for n, vs in transformed.items():
+                    self._option_reprs[n] = _reprs(vs)
+
+    def generate_tests(
+        self,
+        *,
+        name: Optional[str] = None,
+        timeout_time: Optional[float] = None,
+        timeout_unit: str = "step",
+        expect_fail: bool = False,
+        expect_error: Union[Type[Exception], Sequence[Type[Exception]]] = (),
+        skip: bool = False,
+        stage: int = 0,
+    ) -> Iterable[Test]:
+        test_func_name = self.test_function.__qualname__ if name is None else name
+
+        # this value is a list of ranges of the same length as each set of values in self.options for passing to itertools.product
+        option_indexes = [range(len(option[1])) for option in self.options]
+
+        # go through the cartesian product of all values of all options
+        for selected_options in product(*option_indexes):
+            test_kwargs: Dict[str, Sequence[Any]] = {}
+            test_name_pieces: List[str] = [test_func_name]
+            for option_idx, select_idx in enumerate(selected_options):
+                option_name, option_values = self.options[option_idx]
+                selected_value = option_values[select_idx]
+
+                if isinstance(option_name, str):
+                    # single params per option
+                    selected_value = cast(Sequence[Any], selected_value)
+                    test_kwargs[option_name] = selected_value
+                    test_name_pieces.append(
+                        f"/{option_name}={self._option_reprs[option_name][select_idx]}"
+                    )
+                else:
+                    # multiple params per option
+                    selected_value = cast(Sequence[Any], selected_value)
+                    for n, v in zip(option_name, selected_value):
+                        test_kwargs[n] = v
+                        test_name_pieces.append(
+                            f"/{n}={self._option_reprs[n][select_idx]}"
+                        )
+
+            parameterized_test_name = "".join(test_name_pieces)
+
+            # create wrapper function to bind kwargs
+            @functools.wraps(self.test_function)
+            async def _my_test(dut, kwargs: Dict[str, Any] = test_kwargs) -> None:
+                await self.test_function(dut, **kwargs)
+
+            yield Test(
+                func=_my_test,
+                name=parameterized_test_name,
+                timeout_time=timeout_time,
+                timeout_unit=timeout_unit,
+                expect_fail=expect_fail,
+                expect_error=expect_error,
+                skip=skip,
+                stage=stage,
+            )
+
+
+def _reprs(values: Sequence[Any]) -> List[str]:
+    result: List[str] = []
+    for value in values:
+        value_repr = _repr(value)
+        if value_repr is None:
+            # non-representable value in option, so default to index strings and give up
+            return [str(i) for i in range(len(values))]
+        else:
+            result.append(value_repr)
+    return result
+
+
+def _repr(v: Any) -> Optional[str]:
+    if isinstance(v, str):
+        if len(v) <= 10 and v.isidentifier():
+            return v
+        else:
+            return None
+    elif isinstance(v, (int, float, bool, type(None))):
+        return repr(v)
+    elif isinstance(v, Enum):
+        return v.name
+    elif isinstance(v, type):
+        return v.__qualname__
+    elif callable(v) and hasattr(v, "__qualname__"):
+        return v.__qualname__
+    else:
+        return None
+
+
 @overload
-def test(_func: Union[F, TestFactory[F]]) -> F:
+def test(_func: Union[F, _Parameterized[F]]) -> F:
     ...
 
 
@@ -103,12 +229,12 @@ def test(
     skip: bool = False,
     stage: int = 0,
     name: Optional[str] = None,
-) -> Callable[[Union[F, TestFactory[F]]], F]:
+) -> Callable[[Union[F, _Parameterized[F]]], F]:
     ...
 
 
 def test(
-    _func: Optional[Union[F, TestFactory[F]]] = None,
+    _func: Optional[Union[F, _Parameterized[F]]] = None,
     *,
     timeout_time: Optional[float] = None,
     timeout_unit: str = "step",
@@ -117,7 +243,7 @@ def test(
     skip: bool = False,
     stage: int = 0,
     name: Optional[str] = None,
-) -> Callable[[Union[F, TestFactory[F]]], F]:
+) -> Callable[[Union[F, _Parameterized[F]]], F]:
     """
     Decorator to register a Callable which returns a Coroutine as a test.
 
@@ -216,20 +342,20 @@ def _add_tests(module_name: str, *tests: Test) -> None:
         mod.__cocotb_tests__.extend(tests)
 
     if _func is not None:
-        if isinstance(_func, TestFactory):
+        if isinstance(_func, _Parameterized):
             test_func = _func.test_function
-            _add_tests(test_func.__module__, *_func._generate_tests())
+            _add_tests(test_func.__module__, *_func.generate_tests())
             return test_func
         else:
             _add_tests(_func.__module__, Test(func=_func))
             return _func
 
-    def wrapper(f: Union[F, TestFactory[F]]) -> F:
-        if isinstance(f, TestFactory):
+    def wrapper(f: Union[F, _Parameterized[F]]) -> F:
+        if isinstance(f, _Parameterized):
             test_func = f.test_function
             _add_tests(
                 test_func.__module__,
-                *f._generate_tests(
+                *f.generate_tests(
                     name=name,
                     timeout_time=timeout_time,
                     timeout_unit=timeout_unit,
@@ -260,11 +386,11 @@ def wrapper(f: Union[F, TestFactory[F]]) -> F:
 
 
 def parameterize(
-    *args: Union[
+    *options_by_tuple: Union[
         Tuple[str, Sequence[Any]], Tuple[Sequence[str], Sequence[Sequence[Any]]]
     ],
-    **kwargs: Sequence[Any],
-) -> Callable[[Callable[..., Coroutine[Any, Any, None]]], TestFactory]:
+    **options_by_name: Sequence[Any],
+) -> Callable[[F], _Parameterized[F]]:
     """Decorator to generate parameterized tests from a single test function.
 
     Decorates a test function with named test parameters.
@@ -319,7 +445,7 @@ async def my_test_1_b() -> None:
         )
 
     Args:
-        args:
+        options_by_tuple:
             Tuple of parameter name to sequence of values for that parameter,
             or tuple of sequence of parameter names to sequence of sequences of values for that pack of parameters.
 
@@ -329,12 +455,29 @@ async def my_test_1_b() -> None:
     .. versionadded:: 2.0
     """
 
-    def wrapper(f):
-        tf = TestFactory(f)
-        for option_tuple in args:
-            tf.add_option(*option_tuple)
-        for key, lis in kwargs.items():
-            tf.add_option(key, lis)
-        return tf
+    # check good inputs
+    for i, option_by_tuple in enumerate(options_by_tuple):
+        if len(option_by_tuple) != 2:
+            raise ValueError(
+                f"Invalid option tuple {i}, expected exactly two fields `(name, values)`"
+            )
+        name, values = option_by_tuple
+        if not isinstance(name, str):
+            for n in name:
+                if not n.isidentifier():
+                    raise ValueError("Option names must be valid Python identifiers")
+            values = cast(Sequence[Sequence[Any]], values)
+            for value in values:
+                if len(name) != len(value):
+                    raise ValueError(
+                        f"Invalid option tuple {i}, mismatching number of parameters ({name}) and values ({value})"
+                    )
+        elif not name.isidentifier():
+            raise ValueError("Option names must be valid Python identifiers")
+
+    options = [*options_by_tuple, *options_by_name.items()]
+
+    def wrapper(f: F) -> _Parameterized[F]:
+        return _Parameterized(f, options)
 
     return wrapper
diff --git a/src/cocotb/regression.py b/src/cocotb/regression.py
--- a/src/cocotb/regression.py
+++ b/src/cocotb/regression.py
@@ -46,7 +46,6 @@
     Coroutine,
     Dict,
     Generic,
-    Iterable,
     List,
     Optional,
     Sequence,
@@ -963,6 +962,11 @@ class TestFactory(Generic[F]):
     """
 
     def __init__(self, test_function: F, *args: Any, **kwargs: Any) -> None:
+        warnings.warn(
+            "TestFactory is deprecated, use `@cocotb.parameterize` instead",
+            DeprecationWarning,
+            stacklevel=2,
+        )
         self.test_function = test_function
         self.args = args
         self.kwargs_constant = kwargs
@@ -1078,54 +1082,7 @@ def generate_tests(
 
                 .. versionadded:: 2.0
         """
-        warnings.warn(
-            "TestFactory is deprecated, use `@cocotb.parameterize` instead",
-            DeprecationWarning,
-            stacklevel=2,
-        )
 
-        glbs = inspect.currentframe().f_back.f_globals
-
-        if "__cocotb_tests__" not in glbs:
-            glbs["__cocotb_tests__"] = []
-
-        for test in self._generate_tests(
-            prefix=prefix,
-            postfix=postfix,
-            name=name,
-            module=glbs["__name__"],
-            timeout_time=timeout_time,
-            timeout_unit=timeout_unit,
-            expect_fail=expect_fail,
-            expect_error=expect_error,
-            skip=skip,
-            stage=stage,
-        ):
-            if test.name in glbs:
-                _logger.error(
-                    "Overwriting %s in module %s. "
-                    "This causes a previously defined testcase not to be run. "
-                    "Consider using the `name`, `prefix`, or `postfix` arguments to augment the name.",
-                    name,
-                    glbs["__name__"],
-                )
-            glbs["__cocotb_tests__"].append(test)
-            glbs[test.name] = test
-
-    def _generate_tests(
-        self,
-        *,
-        prefix: Optional[str] = None,
-        postfix: Optional[str] = None,
-        name: Optional[str] = None,
-        module: Optional[str] = None,
-        timeout_time: Optional[float] = None,
-        timeout_unit: str = "steps",
-        expect_fail: bool = False,
-        expect_error: Union[Type[Exception], Sequence[Type[Exception]]] = (),
-        skip: bool = False,
-        stage: int = 0,
-    ) -> Iterable[Test]:
         if prefix is not None:
             warnings.warn(
                 "``prefix`` argument is deprecated. Use the more flexible ``name`` field instead.",
@@ -1142,6 +1099,11 @@ def _generate_tests(
         else:
             postfix = ""
 
+        glbs = inspect.currentframe().f_back.f_globals
+
+        if "__cocotb_tests__" not in glbs:
+            glbs["__cocotb_tests__"] = []
+
         test_func_name = self.test_function.__qualname__ if name is None else name
 
         for index, testoptions in enumerate(
@@ -1190,10 +1152,19 @@ async def _my_test(dut, kwargs: Dict[str, Any] = kwargs) -> None:
             _my_test.__name__ = name
             _my_test.__qualname__ = name
 
-            yield Test(
+            if name in glbs:
+                _logger.error(
+                    "Overwriting %s in module %s. "
+                    "This causes a previously defined testcase not to be run. "
+                    "Consider using the `name`, `prefix`, or `postfix` arguments to augment the name.",
+                    name,
+                    glbs["__name__"],
+                )
+
+            test = Test(
                 func=_my_test,
                 name=name,
-                module=module,
+                module=glbs["__name__"],
                 timeout_time=timeout_time,
                 timeout_unit=timeout_unit,
                 expect_fail=expect_fail,
@@ -1201,3 +1172,6 @@ async def _my_test(dut, kwargs: Dict[str, Any] = kwargs) -> None:
                 skip=skip,
                 stage=stage,
             )
+
+            glbs["__cocotb_tests__"].append(test)
+            glbs[test.name] = test


Test patch:
diff --git a/tests/pytest/test_parameterize.py b/tests/pytest/test_parameterize.py
new file mode 100644
--- /dev/null
+++ b/tests/pytest/test_parameterize.py
@@ -0,0 +1,52 @@
+# Copyright cocotb contributors
+# Licensed under the Revised BSD License, see LICENSE for details.
+# SPDX-License-Identifier: BSD-3-Clause
+
+from enum import Enum
+
+import cocotb
+import pytest
+from cocotb.decorators import _repr, _reprs
+
+
+class MyEnum(Enum):
+    ENUM_VALUE = 1
+
+
+class A:
+    ...
+
+
+def b():
+    ...
+
+
+def test_parameterize_repr():
+    assert _repr(1) == "1"
+    assert _repr(False) == "False"
+    assert _repr(0.14) == "0.14"
+    assert _repr(None) == "None"
+    assert _repr("wow") == "wow"
+    assert _repr("has space") is None
+    assert _repr("😊") is None
+    assert _repr("wowthisisareallylongnamethatidontwantonmyterminal") is None
+    assert _repr(A) == "A"
+    assert _repr(b) == "b"
+    assert _repr(MyEnum.ENUM_VALUE) == "ENUM_VALUE"
+    assert _repr(object()) is None
+
+
+def test_parameterize_reprs():
+    assert _reprs([1, 0.5, False]) == ["1", "0.5", "False"]
+    assert _reprs([9, 9, object()]) == ["0", "1", "2"]
+
+
+def test_parameterize_bad_args():
+    with pytest.raises(ValueError):
+        cocotb.parameterize(("not valid", [1, 2, 3], "extra arg whoops"))
+    with pytest.raises(ValueError):
+        cocotb.parameterize(("not valid", [1, 2, 3]))
+    with pytest.raises(ValueError):
+        cocotb.parameterize((("not valid", "valid"), [(1, 2), (3, 4)]))
+    with pytest.raises(ValueError):
+        cocotb.parameterize((("a", "b"), [(1, 2, "too", "many", "args"), (3, 4)]))
diff --git a/tests/test_cases/test_cocotb/test_deprecated.py b/tests/test_cases/test_cocotb/test_deprecated.py
--- a/tests/test_cases/test_cocotb/test_deprecated.py
+++ b/tests/test_cases/test_cocotb/test_deprecated.py
@@ -22,11 +22,12 @@ async def test_testfactory_deprecated_test(dut, a):
     test_testfactory_deprecated_values.append(a)
 
 
-tf = TestFactory(test_testfactory_deprecated_test)
-tf.add_option("a", [1, 2])
 with warnings.catch_warnings(record=True) as tf_warns:
     warnings.simplefilter("default", category=DeprecationWarning)
-    tf.generate_tests()
+    tf = TestFactory(test_testfactory_deprecated_test)
+
+tf.add_option("a", [1, 2])
+tf.generate_tests()
 
 
 @cocotb.test
diff --git a/tests/test_cases/test_cocotb/test_testfactory.py b/tests/test_cases/test_cocotb/test_testfactory.py
--- a/tests/test_cases/test_cocotb/test_testfactory.py
+++ b/tests/test_cases/test_cocotb/test_testfactory.py
@@ -29,8 +29,15 @@ async def test_testfactory_verify_args(dut):
         ("a1v1", "a2v2", "a3v2"),
         ("a1v2", "a2v2", "a3v2"),
     }
+
+
+@cocotb.test
+async def test_testfactory_verify_names(dut):
     assert testfactory_test_names == {
-        f"run_testfactory_test_{i:03}" for i in range(1, 5)
+        "run_testfactory_test/arg1=a1v1/arg2=a2v1/arg3=a3v1",
+        "run_testfactory_test/arg1=a1v1/arg2=a2v2/arg3=a3v2",
+        "run_testfactory_test/arg1=a1v2/arg2=a2v1/arg3=a3v1",
+        "run_testfactory_test/arg1=a1v2/arg2=a2v2/arg3=a3v2",
     }
 
 
@@ -77,10 +84,13 @@ async def test_params_verify_args(dut):
     }
 
 
-@cocotb.test()
+@cocotb.test
 async def test_params_verify_names(dut):
     assert p_testfactory_test_names == {
-        f"p_run_testfactory_test_{i:03}" for i in range(1, 5)
+        "p_run_testfactory_test/arg1=a1v1/arg2=a2v1",
+        "p_run_testfactory_test/arg1=a1v1/arg2=a2v2",
+        "p_run_testfactory_test/arg1=a1v2/arg2=a2v1",
+        "p_run_testfactory_test/arg1=a1v2/arg2=a2v2",
     }