Repository: cocotb/cocotb

Issue:
Add some basic assert methods ?
```
assertEqual(a, b) 
assertNotEqual(a, b) 
assertTrue(x) 
assertFalse(x)
```

It would potentially make logging easier/better.

Based on https://docs.python.org/3/library/unittest.html#unittest.TestCase.debug


Discussion:
I had actually started writing a check library based on https://vunit.github.io/check/user_guide.html which looks well thought out, but haven't gotten very far yet.
I like this solution better than using `assert`, so I think it is worthwhile.

Assert functions would allow for
1. delayed error message evaluation
2. throwing the proper exception: `TestFailure`

Right now we have support for `assert` statements that requires us to add `AssertionError` as another exception wherever `TestFailure` is expected. I think that was worked around by making `TestFailure` inherit from `AssertionError` as well as `TestComplete`, so you would catch `AssertionError` instead. However, you still have to catch `AssertionError` and `TestComplete` if you want to accept any test ending exception. We can't work around that by making `TestComplete` inherit from `AssertionError`, that's not doing the right thing. The whole thing is just kind of messy.
Some questions here:

* How hard would it be to just use the `unittest` ones directly, rather than writing our own? Can we support `@test` decorators on `unittest.TestCase` objects?
* Is the eventual goal to be able to use pytest natively within the tests? (xref #pytest). If so, it might be better not to build these only to have them become defunct.


> However, you still have to catch AssertionError and TestComplete if you want to accept any test ending exception.

Are you describing `except` statements in cocotb internals, or in actual tests?

My gut feeling is that in the long run, having "test ending exception" objects is not a particularly good design. A pattern I've seen in pytest tests before is:
```python
def test_something():
    assert isinstance(a.foo(), Bar)
    try:
        assert a.foo().b == "b"
        assert a.foo().c == "c"
    except AssertionError as e:
        raise AssertionError("a.foo() was unexpected") from e
```
This approach doesn't work if `AssertionError` blows up the universe.

`AssertionError` only currently blows up the universe if it bubbles up to the root of a forked coroutine. This was done because the legacy semantics of `fork` are that. This could probably change, we discussed this in #922, but nothing came of it.
I've had a lot of luck with [PyHamcrest](https://pyhamcrest.readthedocs.io/en/release-1.8/tutorial/) for adding richer asserts to Python packages (except when using pytest, which seems to use dark magic to get fancy looking asserts).

I will grant that it's a little verbose and I could see how some people might not like that.

```python
from hamcrest import *


@cocotb.test()
def the_test(dut):
    yield Timer(1)
    assert_that(2+2, equal_to(7))
```

```
Send raised exception:
Traceback (most recent call last):
  File "/nix/store/srfxgwcc5x53c08m3gfwxkxjmg9pwyz1-python3.7-cocotb-1.2.0/lib/python3.7/site-packages/cocotb/decorators.py", line 259, in _advance
    return outcome.send(self._coro)
  File "/nix/store/srfxgwcc5x53c08m3gfwxkxjmg9pwyz1-python3.7-cocotb-1.2.0/lib/python3.7/site-packages/cocotb/outcomes.py", line 37, in send
    return gen.send(self.value)
  File "/home/jwilcox/repo/sim/test.py", line 10, in the_test
    assert_that(2+2, equal_to(7))
  File "/nix/store/fplm7gph3m4ws74zvqby6gqbjl7czqy1-python3.7-PyHamcrest-1.9.0/lib/python3.7/site-packages/hamcrest/core/assert_that.py", line 43, in assert_that
    _assert_match(actual=arg1, matcher=arg2, reason=arg3)
  File "/nix/store/fplm7gph3m4ws74zvqby6gqbjl7czqy1-python3.7-PyHamcrest-1.9.0/lib/python3.7/site-packages/hamcrest/core/assert_that.py", line 57, in _assert_match
    raise AssertionError(description)
AssertionError:
Expected: <7>
     but: was <4>
```
@Nitori- Nice find. If this work maybe we should document this and do not reinvent anything new for now.

Resolving patch:
diff --git a/cocotb/__init__.py b/cocotb/__init__.py
--- a/cocotb/__init__.py
+++ b/cocotb/__init__.py
@@ -240,6 +240,23 @@ def _initialise_testbench(argv_):
 
     dut = cocotb.handle.SimHandle(handle)
 
+    try:
+        import pytest
+    except ImportError:
+        log.warning("Pytest not found, assertion rewriting will not occur")
+    else:
+        try:
+            # Install the assertion rewriting hook, which must be done before we
+            # import the test modules.
+            from _pytest.config import Config
+            from _pytest.assertion import install_importhook
+            pytest_conf = Config.fromdictargs([], {})
+            install_importhook(pytest_conf)
+        except Exception:
+            log.exception(
+                "Configuring the assertion rewrite hook using pytest {} failed. "
+                "Please file a bug report!".format(pytest.__version__))
+
     # start Regression Manager
     global regression_manager
     regression_manager = RegressionManager.from_discovery(dut)


Test patch:
diff --git a/tests/test_cases/test_cocotb/Makefile b/tests/test_cases/test_cocotb/Makefile
--- a/tests/test_cases/test_cocotb/Makefile
+++ b/tests/test_cases/test_cocotb/Makefile
@@ -18,6 +18,7 @@ MODULE := "\
 	test_async_coroutines,\
 	test_handle,\
 	test_logging,\
+	test_pytest,\
 	"
 
 ifeq ($(shell python -c "import sys; print(sys.version_info >= (3, 6))"), "True")
diff --git a/tests/test_cases/test_cocotb/test_pytest.py b/tests/test_cases/test_cocotb/test_pytest.py
new file mode 100644
--- /dev/null
+++ b/tests/test_cases/test_cocotb/test_pytest.py
@@ -0,0 +1,22 @@
+# Copyright cocotb contributors
+# Licensed under the Revised BSD License, see LICENSE for details.
+# SPDX-License-Identifier: BSD-3-Clause
+""" Tests relating to pytest integration """
+
+import cocotb
+
+# pytest is an optional dependency
+try:
+    import pytest
+except ImportError:
+    pytest = None
+
+
+@cocotb.test(skip=pytest is None)
+async def test_assertion_rewriting(dut):
+    """ Test that assertion rewriting hooks take effect in cocotb tests """
+    try:
+        assert 1 != 42
+    except AssertionError as e:
+        assert "42" in str(e), (
+            "Assertion rewriting seems not to work, message was {}".format(e))