Repository: cocotb/cocotb

Issue:
Regression manager does not handle BaseException raised in task
<!--
In case you are reporting a problem with running cocotb,
please remember to add relevant information about your environment, such as
* the cocotb version used,
* the operating system and version (32/64 bit),
* the simulator and version (32/64 bit),
* the Python version, and where it's coming from (e.g. system, Anaconda, self-installed, ...),
* (part of) the log file (cleaned of proprietary information).
If you are unsure about any of the above items, open the issue anyway and we will figure it out together.
Thanks for helping improve cocotb!
-->
cococtb: master (4aee0a0895a36606f9263df9370405663f496d22)
Ubuntu 20.04.2 LTS on WSL2
simulator: Verilator 5.007 devel rev v5.006-7-g4a8cfe367

---

When `pytest.raises()` fails, it raises a `Failed` exception.

This exception derives from `BaseException`, so the check in `RegressionManager._score_test()` fails:
https://github.com/cocotb/cocotb/blob/4aee0a0895a36606f9263df9370405663f496d22/cocotb/regression.py#L395-L400

 The raised `Failed` exception propagates up through `Scheduler._test_completed()` and into the simulator extension module.

Here is an example traceback:
```
567237.42ns INFO     cocotb.regression                  running test_integer_overflow (158/183)
                                                          Test integer overflow.
ERROR: called callback function threw exception
Traceback (most recent call last):
  File "/home/marlon/git/cocotb/.nox/dev_test_sim-sim-verilator-toplevel_lang-verilog-gpi_interface-vpi/lib/python3.8/site-packages/cocotb/scheduler.py", line 349, in _test_completed
    self._handle_result(test)
  File "/home/marlon/git/cocotb/.nox/dev_test_sim-sim-verilator-toplevel_lang-verilog-gpi_interface-vpi/lib/python3.8/site-packages/cocotb/regression.py", line 332, in _handle_result
    self._record_result(
  File "/home/marlon/git/cocotb/.nox/dev_test_sim-sim-verilator-toplevel_lang-verilog-gpi_interface-vpi/lib/python3.8/site-packages/cocotb/regression.py", line 506, in _record_result
    test_pass, sim_failed = self._score_test(test, outcome)
  File "/home/marlon/git/cocotb/.nox/dev_test_sim-sim-verilator-toplevel_lang-verilog-gpi_interface-vpi/lib/python3.8/site-packages/cocotb/regression.py", line 396, in _score_test
    outcome.get()
  File "/home/marlon/git/cocotb/.nox/dev_test_sim-sim-verilator-toplevel_lang-verilog-gpi_interface-vpi/lib/python3.8/site-packages/cocotb/outcomes.py", line 53, in get
    raise self.error
  File "/home/marlon/git/cocotb/tests/test_cases/test_cocotb/test_handle.py", line 312, in test_integer_overflow
    await int_overflow_test(dut.stream_in_int, 32, "ovfl", limits)
  File "/home/marlon/git/cocotb/tests/test_cases/test_cocotb/test_handle.py", line 148, in int_overflow_test
    signal.value = value
  File "/home/marlon/git/cocotb/.nox/dev_test_sim-sim-verilator-toplevel_lang-verilog-gpi_interface-vpi/lib/python3.8/site-packages/_pytest/python_api.py", line 984, in __exit__
    fail(self.message)
  File "/home/marlon/git/cocotb/.nox/dev_test_sim-sim-verilator-toplevel_lang-verilog-gpi_interface-vpi/lib/python3.8/site-packages/_pytest/outcomes.py", line 194, in fail
    raise Failed(msg=reason, pytrace=pytrace)
Failed: DID NOT RAISE <class 'OverflowError'>
- :0: Verilog $finish
- /home/marlon/git/cocotb/.nox/dev_test_sim-sim-verilator-toplevel_lang-verilog-gpi_interface-vpi/lib/python3.8/site-packages/cocotb/share/lib/verilator/verilator.cpp:118: Verilog $finish
- /home/marlon/git/cocotb/.nox/dev_test_sim-sim-verilator-toplevel_lang-verilog-gpi_interface-vpi/lib/python3.8/site-packages/cocotb/share/lib/verilator/verilator.cpp:118: Second verilog $finish, exiting
ERROR: results.xml was not written by the simulation!
```


Discussion:
>This exception derives from BaseException

Yikers... Do we treat `Failure` as a test failure or a test exception? We might need to special case it here.
I think it should probably be handled the same as `AssertionError`.

If the failure is expected, it would match `expect_fail` and `expect_error` depending on how the `@cocotb.test()` decorator is called.
`Failed` is only available as a private implementation detail (`_pytest.outcomes.Failed`). I'm not sure how often the internals change, but the whole thing is starting to stink. I have a clever way of getting around it though, but I don't like it.

Resolving patch:
diff --git a/cocotb/regression.py b/cocotb/regression.py
--- a/cocotb/regression.py
+++ b/cocotb/regression.py
@@ -391,7 +391,9 @@ def _score_test(self, test: Test, outcome: Outcome) -> Tuple[bool, bool]:
 
         try:
             outcome.get()
-        except Exception as e:
+        except (KeyboardInterrupt, SystemExit):
+            raise
+        except BaseException as e:
             result = remove_traceback_frames(e, ["_score_test", "get"])
         else:
             result = TestSuccess()
@@ -403,9 +405,6 @@ def _score_test(self, test: Test, outcome: Outcome) -> Tuple[bool, bool]:
         ):
             self._log_test_passed(test, None, None)
 
-        elif isinstance(result, AssertionError) and test.expect_fail:
-            self._log_test_passed(test, result, "failed as expected")
-
         elif isinstance(result, TestSuccess) and test.expect_error:
             self._log_test_failed(test, None, "passed but we expected an error")
             result_pass = False
@@ -430,6 +429,9 @@ def _score_test(self, test: Test, outcome: Outcome) -> Tuple[bool, bool]:
                 self._log_test_failed(test, result, "errored with unexpected type ")
                 result_pass = False
 
+        elif test.expect_fail:
+            self._log_test_passed(test, result, "failed as expected")
+
         else:
             self._log_test_failed(test, result, None)
             result_pass = False
diff --git a/cocotb/scheduler.py b/cocotb/scheduler.py
--- a/cocotb/scheduler.py
+++ b/cocotb/scheduler.py
@@ -581,7 +581,7 @@ def _unschedule(self, coro):
                 coro.log.info("Test stopped by this forked coroutine")
                 e = remove_traceback_frames(e, ["_unschedule", "get"])
                 self._abort_test(e)
-            except Exception as e:
+            except BaseException as e:
                 coro.log.error("Exception raised by this forked coroutine")
                 e = remove_traceback_frames(e, ["_unschedule", "get"])
                 warnings.warn(


Test patch:
diff --git a/tests/Makefile b/tests/Makefile
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -27,7 +27,7 @@
 
 # Do not fail on DeprecationWarning caused by virtualenv, which might come from
 # the site module.
-export PYTHONWARNINGS = error,ignore::DeprecationWarning:site
+export PYTHONWARNINGS = error,ignore::DeprecationWarning:site,always::FutureWarning:cocotb.scheduler
 
 REGRESSIONS :=  $(shell ls test_cases/)
 
diff --git a/tests/test_cases/test_cocotb/common.py b/tests/test_cases/test_cocotb/common.py
--- a/tests/test_cases/test_cocotb/common.py
+++ b/tests/test_cases/test_cocotb/common.py
@@ -26,3 +26,7 @@ async def _check_traceback(running_coro, exc_type, pattern, *match_args):
 
 class MyException(Exception):
     ...
+
+
+class MyBaseException(BaseException):
+    ...
diff --git a/tests/test_cases/test_cocotb/test_tests.py b/tests/test_cases/test_cocotb/test_tests.py
--- a/tests/test_cases/test_cocotb/test_tests.py
+++ b/tests/test_cases/test_cocotb/test_tests.py
@@ -10,10 +10,12 @@
 """
 from collections.abc import Coroutine
 
-from common import MyException
+import pytest
+from _pytest.outcomes import Failed
+from common import MyBaseException, MyException
 
 import cocotb
-from cocotb.triggers import Timer
+from cocotb.triggers import NullTrigger, Timer
 
 
 @cocotb.test(expect_error=NameError)
@@ -131,3 +133,97 @@ def __await__(self):
 @cocotb.test()
 async def test_empty_docstring(dut) -> None:
     """"""
+
+
+@cocotb.test(expect_error=Failed)
+async def test_pytest_raises_fail(dut):
+    with pytest.raises(AssertionError):
+        assert True
+
+
+@cocotb.test(expect_error=Failed)
+async def test_pytest_warns_fail(dut):
+    def test_func():
+        pass
+
+    with pytest.warns(RuntimeWarning):
+        test_func()
+
+
+@cocotb.test(expect_error=Failed)
+async def test_pytest_deprecated_call_fail(dut):
+    def test_func():
+        pass
+
+    with pytest.deprecated_call():
+        test_func()
+
+
+@cocotb.test(expect_error=Failed)
+async def test_pytest_raises_fail_in_task(dut):
+    async def test_func():
+        with pytest.raises(AssertionError):
+            assert True
+
+    cocotb.start_soon(test_func())
+    await NullTrigger()
+
+
+@cocotb.test(expect_error=Failed)
+async def test_pytest_warns_fail_in_task(dut):
+    def inner_func():
+        pass
+
+    async def test_func():
+        with pytest.warns(RuntimeWarning):
+            inner_func()
+
+    cocotb.start_soon(test_func())
+    await NullTrigger()
+
+
+@cocotb.test(expect_error=Failed)
+async def test_pytest_deprecated_call_fail_in_task(dut):
+    def inner_func():
+        pass
+
+    async def test_func():
+        with pytest.deprecated_call():
+            inner_func()
+
+    cocotb.start_soon(test_func())
+    await NullTrigger()
+
+
+@cocotb.test(expect_fail=True)
+async def test_pytest_raises_fail_expect_fail(dut):
+    with pytest.raises(AssertionError):
+        assert True
+
+
+@cocotb.test(expect_fail=True)
+async def test_base_exception_expect_fail(dut):
+    raise MyBaseException
+
+
+@cocotb.test(expect_fail=True)
+async def test_base_exception_in_task_expect_fail(dut):
+    async def test_func():
+        raise MyBaseException
+
+    cocotb.start_soon(test_func())
+    await NullTrigger()
+
+
+@cocotb.test(expect_fail=True)
+async def test_exception_expect_fail(dut):
+    raise MyException()
+
+
+@cocotb.test(expect_fail=True)
+async def test_exception_in_task_expect_fail(dut):
+    async def test_func():
+        raise MyException
+
+    cocotb.start_soon(test_func())
+    await NullTrigger()
diff --git a/tests/test_cases/test_failure/Makefile b/tests/test_cases/test_failure/Makefile
--- a/tests/test_cases/test_failure/Makefile
+++ b/tests/test_cases/test_failure/Makefile
@@ -9,6 +9,6 @@ MODULE := test_failure
 override_for_this_test:
 	-$(MAKE) all
 	$(call check_for_results_file)
-	test $$(../../../bin/combine_results.py | grep "Failure in testsuite" | wc -l) -eq 1 && rm -f results.xml
+	test $$(../../../bin/combine_results.py | grep "Failure in testsuite" | wc -l) -eq 4 && rm -f results.xml
 
 include ../../designs/sample_module/Makefile
diff --git a/tests/test_cases/test_failure/test_failure.py b/tests/test_cases/test_failure/test_failure.py
--- a/tests/test_cases/test_failure/test_failure.py
+++ b/tests/test_cases/test_failure/test_failure.py
@@ -13,3 +13,18 @@
 @cocotb.test()
 async def test_fail(_):
     assert False
+
+
+@cocotb.test(expect_fail=True)
+async def test_pass_expect_fail(_):
+    assert True
+
+
+@cocotb.test(expect_error=Exception)
+async def test_pass_expect_error(_):
+    assert True
+
+
+@cocotb.test(expect_error=ValueError)
+async def test_wrong_error(_):
+    raise TypeError