Repository: cocotb/cocotb

Issue:
Regression manager does not handle BaseException raised in task
<!--
In case you are reporting a problem with running cocotb,
please remember to add relevant information about your environment, such as
* the cocotb version used,
* the operating system and version (32/64 bit),
* the simulator and version (32/64 bit),
* the Python version, and where it's coming from (e.g. system, Anaconda, self-installed, ...),
* (part of) the log file (cleaned of proprietary information).
If you are unsure about any of the above items, open the issue anyway and we will figure it out together.
Thanks for helping improve cocotb!
-->
cococtb: master (4aee0a0895a36606f9263df9370405663f496d22)
Ubuntu 20.04.2 LTS on WSL2
simulator: Verilator 5.007 devel rev v5.006-7-g4a8cfe367

---

When `pytest.raises()` fails, it raises a `Failed` exception.

This exception derives from `BaseException`, so the check in `RegressionManager._score_test()` fails:
https://github.com/cocotb/cocotb/blob/4aee0a0895a36606f9263df9370405663f496d22/cocotb/regression.py#L395-L400

 The raised `Failed` exception propagates up through `Scheduler._test_completed()` and into the simulator extension module.

Here is an example traceback:
```
567237.42ns INFO     cocotb.regression                  running test_integer_overflow (158/183)
                                                          Test integer overflow.
ERROR: called callback function threw exception
Traceback (most recent call last):
  File "/home/marlon/git/cocotb/.nox/dev_test_sim-sim-verilator-toplevel_lang-verilog-gpi_interface-vpi/lib/python3.8/site-packages/cocotb/scheduler.py", line 349, in _test_completed
    self._handle_result(test)
  File "/home/marlon/git/cocotb/.nox/dev_test_sim-sim-verilator-toplevel_lang-verilog-gpi_interface-vpi/lib/python3.8/site-packages/cocotb/regression.py", line 332, in _handle_result
    self._record_result(
  File "/home/marlon/git/cocotb/.nox/dev_test_sim-sim-verilator-toplevel_lang-verilog-gpi_interface-vpi/lib/python3.8/site-packages/cocotb/regression.py", line 506, in _record_result
    test_pass, sim_failed = self._score_test(test, outcome)
  File "/home/marlon/git/cocotb/.nox/dev_test_sim-sim-verilator-toplevel_lang-verilog-gpi_interface-vpi/lib/python3.8/site-packages/cocotb/regression.py", line 396, in _score_test
    outcome.get()
  File "/home/marlon/git/cocotb/.nox/dev_test_sim-sim-verilator-toplevel_lang-verilog-gpi_interface-vpi/lib/python3.8/site-packages/cocotb/outcomes.py", line 53, in get
    raise self.error
  File "/home/marlon/git/cocotb/tests/test_cases/test_cocotb/test_handle.py", line 312, in test_integer_overflow
    await int_overflow_test(dut.stream_in_int, 32, "ovfl", limits)
  File "/home/marlon/git/cocotb/tests/test_cases/test_cocotb/test_handle.py", line 148, in int_overflow_test
    signal.value = value
  File "/home/marlon/git/cocotb/.nox/dev_test_sim-sim-verilator-toplevel_lang-verilog-gpi_interface-vpi/lib/python3.8/site-packages/_pytest/python_api.py", line 984, in __exit__
    fail(self.message)
  File "/home/marlon/git/cocotb/.nox/dev_test_sim-sim-verilator-toplevel_lang-verilog-gpi_interface-vpi/lib/python3.8/site-packages/_pytest/outcomes.py", line 194, in fail
    raise Failed(msg=reason, pytrace=pytrace)
Failed: DID NOT RAISE <class 'OverflowError'>
- :0: Verilog $finish
- /home/marlon/git/cocotb/.nox/dev_test_sim-sim-verilator-toplevel_lang-verilog-gpi_interface-vpi/lib/python3.8/site-packages/cocotb/share/lib/verilator/verilator.cpp:118: Verilog $finish
- /home/marlon/git/cocotb/.nox/dev_test_sim-sim-verilator-toplevel_lang-verilog-gpi_interface-vpi/lib/python3.8/site-packages/cocotb/share/lib/verilator/verilator.cpp:118: Second verilog $finish, exiting
ERROR: results.xml was not written by the simulation!
```


Discussion:
>This exception derives from BaseException

Yikers... Do we treat `Failure` as a test failure or a test exception? We might need to special case it here.
I think it should probably be handled the same as `AssertionError`.

If the failure is expected, it would match `expect_fail` and `expect_error` depending on how the `@cocotb.test()` decorator is called.
`Failed` is only available as a private implementation detail (`_pytest.outcomes.Failed`). I'm not sure how often the internals change, but the whole thing is starting to stink. I have a clever way of getting around it though, but I don't like it.

Resolving patch:
diff --git a/cocotb/decorators.py b/cocotb/decorators.py
--- a/cocotb/decorators.py
+++ b/cocotb/decorators.py
@@ -503,7 +503,9 @@ class test(coroutine, metaclass=_decorator_helper):
                 Using ``None`` as the *timeout_unit* argument is deprecated, use ``'step'`` instead.
 
         expect_fail (bool, optional):
-            Don't mark the result as a failure if the test fails.
+            If ``True`` and the test fails a functional check via an ``assert`` statement, :class:`pytest.raises`,
+            :class:`pytest.warns`, or :class:`pytest.deprecated_call` the test is considered to have passed.
+            If ``True`` and the test passes successfully, the test is considered to have failed.
 
         expect_error (exception type or tuple of exception types, optional):
             Mark the result as a pass only if one of the exception types is raised in the test.
@@ -589,7 +591,7 @@ async def f(*args, **kwargs):
                 stacklevel=2,
             )
         if expect_error is True:
-            expect_error = (Exception,)
+            expect_error = (BaseException,)
         elif expect_error is False:
             expect_error = ()
         self.expect_error = expect_error
diff --git a/cocotb/regression.py b/cocotb/regression.py
--- a/cocotb/regression.py
+++ b/cocotb/regression.py
@@ -37,7 +37,7 @@
 import time
 import traceback
 from itertools import product
-from typing import Any, Iterable, Optional, Tuple
+from typing import Any, Iterable, Optional, Tuple, Type
 
 import cocotb
 import cocotb.ANSI as ANSI
@@ -78,6 +78,20 @@ def _my_import(name: str) -> Any:
 
 _logger = SimLog(__name__)
 
+_Failed: Type[BaseException]
+try:
+    import pytest
+except ModuleNotFoundError:
+    _Failed = AssertionError
+else:
+    try:
+        with pytest.raises(Exception):
+            pass
+    except BaseException as _raises_e:
+        _Failed = type(_raises_e)
+    else:
+        assert "pytest.raises doesn't raise an exception when it fails"
+
 
 class RegressionManager:
     """Encapsulates all regression capability into a single place"""
@@ -422,6 +436,9 @@ def _score_test(self, test: Test, outcome: Outcome) -> Tuple[bool, bool]:
             # whether we expected it or not, the simulation has failed unrecoverably
             sim_failed = True
 
+        elif isinstance(result, (AssertionError, _Failed)) and test.expect_fail:
+            self._log_test_passed(test, result, "failed as expected")
+
         elif test.expect_error:
             if isinstance(result, test.expect_error):
                 self._log_test_passed(test, result, "errored as expected")
@@ -429,9 +446,6 @@ def _score_test(self, test: Test, outcome: Outcome) -> Tuple[bool, bool]:
                 self._log_test_failed(test, result, "errored with unexpected type ")
                 result_pass = False
 
-        elif test.expect_fail:
-            self._log_test_passed(test, result, "failed as expected")
-
         else:
             self._log_test_failed(test, result, None)
             result_pass = False


Test patch:
diff --git a/tests/Makefile b/tests/Makefile
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -27,7 +27,8 @@
 
 # Do not fail on DeprecationWarning caused by virtualenv, which might come from
 # the site module.
-export PYTHONWARNINGS = error,ignore::DeprecationWarning:site,always::FutureWarning:cocotb.scheduler
+# Do not fail on DeprecationWarning caused by attrs dropping 3.6 support
+export PYTHONWARNINGS = error,ignore::DeprecationWarning:site,always::FutureWarning:cocotb.scheduler,ignore::DeprecationWarning:attr
 
 REGRESSIONS :=  $(shell ls test_cases/)
 
diff --git a/tests/test_cases/test_cocotb/test_tests.py b/tests/test_cases/test_cocotb/test_tests.py
--- a/tests/test_cases/test_cocotb/test_tests.py
+++ b/tests/test_cases/test_cocotb/test_tests.py
@@ -11,7 +11,6 @@
 from collections.abc import Coroutine
 
 import pytest
-from _pytest.outcomes import Failed
 from common import MyBaseException, MyException
 
 import cocotb
@@ -135,13 +134,13 @@ async def test_empty_docstring(dut) -> None:
     """"""
 
 
-@cocotb.test(expect_error=Failed)
+@cocotb.test(expect_fail=True)
 async def test_pytest_raises_fail(dut):
     with pytest.raises(AssertionError):
         assert True
 
 
-@cocotb.test(expect_error=Failed)
+@cocotb.test(expect_fail=True)
 async def test_pytest_warns_fail(dut):
     def test_func():
         pass
@@ -150,7 +149,7 @@ def test_func():
         test_func()
 
 
-@cocotb.test(expect_error=Failed)
+@cocotb.test(expect_fail=True)
 async def test_pytest_deprecated_call_fail(dut):
     def test_func():
         pass
@@ -159,7 +158,7 @@ def test_func():
         test_func()
 
 
-@cocotb.test(expect_error=Failed)
+@cocotb.test(expect_fail=True)
 async def test_pytest_raises_fail_in_task(dut):
     async def test_func():
         with pytest.raises(AssertionError):
@@ -169,7 +168,7 @@ async def test_func():
     await NullTrigger()
 
 
-@cocotb.test(expect_error=Failed)
+@cocotb.test(expect_fail=True)
 async def test_pytest_warns_fail_in_task(dut):
     def inner_func():
         pass
@@ -182,7 +181,7 @@ async def test_func():
     await NullTrigger()
 
 
-@cocotb.test(expect_error=Failed)
+@cocotb.test(expect_fail=True)
 async def test_pytest_deprecated_call_fail_in_task(dut):
     def inner_func():
         pass
@@ -195,35 +194,15 @@ async def test_func():
     await NullTrigger()
 
 
-@cocotb.test(expect_fail=True)
-async def test_pytest_raises_fail_expect_fail(dut):
-    with pytest.raises(AssertionError):
-        assert True
-
-
-@cocotb.test(expect_fail=True)
+@cocotb.test(expect_error=MyBaseException)
 async def test_base_exception_expect_fail(dut):
     raise MyBaseException
 
 
-@cocotb.test(expect_fail=True)
+@cocotb.test(expect_error=MyBaseException)
 async def test_base_exception_in_task_expect_fail(dut):
     async def test_func():
         raise MyBaseException
 
     cocotb.start_soon(test_func())
     await NullTrigger()
-
-
-@cocotb.test(expect_fail=True)
-async def test_exception_expect_fail(dut):
-    raise MyException()
-
-
-@cocotb.test(expect_fail=True)
-async def test_exception_in_task_expect_fail(dut):
-    async def test_func():
-        raise MyException
-
-    cocotb.start_soon(test_func())
-    await NullTrigger()
diff --git a/tests/test_cases/test_failure/Makefile b/tests/test_cases/test_failure/Makefile
--- a/tests/test_cases/test_failure/Makefile
+++ b/tests/test_cases/test_failure/Makefile
@@ -9,6 +9,6 @@ MODULE := test_failure
 override_for_this_test:
 	-$(MAKE) all
 	$(call check_for_results_file)
-	test $$(../../../bin/combine_results.py | grep "Failure in testsuite" | wc -l) -eq 4 && rm -f results.xml
+	test $$(../../../bin/combine_results.py | grep "Failure in testsuite" | wc -l) -eq 5 && rm -f results.xml
 
 include ../../designs/sample_module/Makefile
diff --git a/tests/test_cases/test_failure/test_failure.py b/tests/test_cases/test_failure/test_failure.py
--- a/tests/test_cases/test_failure/test_failure.py
+++ b/tests/test_cases/test_failure/test_failure.py
@@ -28,3 +28,8 @@ async def test_pass_expect_error(_):
 @cocotb.test(expect_error=ValueError)
 async def test_wrong_error(_):
     raise TypeError
+
+
+@cocotb.test(expect_fail=True)
+async def test_expect_fail_but_errors(_):
+    raise Exception()