Repository: cocotb/cocotb

Issue:
TestError and expect_error
The `cocotb.test` decorator can take an `expect_error` argument. Currently this is a boolean deciding that the given test passed if it got a `TestError`, and failing otherwise. There are many reasons why a test would fail, and usually only one that is actually expected. However, `TestError` muddles all errors, making `expect_error` work if *any* error is caught. Most good regression systems allow you to expect an error and specify what type of exception is anticipated, failing otherwise.

I'd like to see this
```python
class MyException(Exception): pass

@cocotb.coroutine
def throws_my_exception():
    raise MyException

@cocotb.coroutine
def throws_other_exception():
    raise NotImplementedError

@cocotb.test(expect_error=MyException)
def passing_test(dut):
    yield throws_my_exception()

@cocotb.test(expect_error=MyException)
def failing_test(dut):
    yield throws_other_exception()
```

This would require re-writing a good bit of the scheduler and regression manager to not use `TestError` and properly propagate exceptions. I can't see this being too big of a break, user code doesn't/shouldn't be catching or throwing `TestError`.


Discussion:
I don't think it's safe to assume that non-core code wont use `TestError`.
But you could keep backwards compatibility if `expect_error=True` expects a `TestError` object.
@garmin-mjames 
You are correct about that. TestError can be used to short-circuit test failure in forked coroutines, a behavior not available in any other way. TestError isn't bad, but how it is being used in the regression manager/scheduler is an issue.
> This would require re-writing a good bit of the scheduler and regression manager to not use TestError and properly propagate exceptions. 

I think this might be done? Your proposed change looks very straightforward to me now.

Resolving patch:
diff --git a/cocotb/decorators.py b/cocotb/decorators.py
--- a/cocotb/decorators.py
+++ b/cocotb/decorators.py
@@ -1,7 +1,7 @@
 # Copyright (c) 2013 Potential Ventures Ltd
 # Copyright (c) 2013 SolarFlare Communications Inc
 # All rights reserved.
-# 
+#
 # Redistribution and use in source and binary forms, with or without
 # modification, are permitted provided that the following conditions are met:
 #     * Redistributions of source code must retain the above copyright
@@ -13,7 +13,7 @@
 #       SolarFlare Communications Inc nor the
 #       names of its contributors may be used to endorse or promote products
 #       derived from this software without specific prior written permission.
-# 
+#
 # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
 # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 # WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
@@ -437,10 +437,24 @@ class test(_py_compat.with_metaclass(_decorator_helper, coroutine)):
             value representing simulation timeout (not implemented).
         expect_fail (bool, optional):
             Don't mark the result as a failure if the test fails.
-        expect_error (bool, optional):
-            Don't mark the result as an error if an error is raised.
-            This is for cocotb internal regression use 
-            when a simulator error is expected.
+        expect_error (bool or exception type or tuple of exception types, optional):
+            If ``True``, consider this test passing if it raises *any* :class:`Exception`, and failing if it does not.
+            If given an exception type or tuple of exception types, catching *only* a listed exception type is considered passing.
+            This is primarily for cocotb internal regression use for when a simulator error is expected.
+
+            Users are encouraged to use the following idiom instead::
+
+                @cocotb.test()
+                def my_test(dut):
+                    try:
+                        yield thing_that_should_fail()
+                    except ExceptionIExpect:
+                        pass
+                    else:
+                        assert False, "Exception did not occur"
+
+            .. versionchanged:: 1.3
+                Specific exception types can be expected
         skip (bool, optional):
             Don't execute this test as part of the regression.
         stage (int, optional)
@@ -452,6 +466,10 @@ def __init__(self, f, timeout=None, expect_fail=False, expect_error=False,
 
         self.timeout = timeout
         self.expect_fail = expect_fail
+        if expect_error is True:
+            expect_error = (Exception,)
+        elif expect_error is False:
+            expect_error = ()
         self.expect_error = expect_error
         self.skip = skip
         self.stage = stage
diff --git a/cocotb/regression.py b/cocotb/regression.py
--- a/cocotb/regression.py
+++ b/cocotb/regression.py
@@ -305,7 +305,7 @@ def _result_was():
             result_pass = False
 
         elif isinstance(result, SimFailure):
-            if test.expect_error:
+            if isinstance(result, test.expect_error):
                 self.log.info("Test errored as expected: " + _result_was())
             else:
                 self.log.error("Test error has lead to simulator shutting us "
@@ -316,7 +316,12 @@ def _result_was():
                 return
 
         elif test.expect_error:
-            self.log.info("Test errored as expected: " + _result_was())
+            if isinstance(result, test.expect_error):
+                self.log.info("Test errored as expected: " + _result_was())
+            else:
+                self.log.info("Test errored with unexpected type: " + _result_was())
+                self._add_failure(result)
+                result_pass = False
 
         else:
             self.log.error("Test Failed: " + _result_was(), exc_info=exc_info)


Test patch:
diff --git a/tests/test_cases/test_cocotb/test_cocotb.py b/tests/test_cases/test_cocotb/test_cocotb.py
--- a/tests/test_cases/test_cocotb/test_cocotb.py
+++ b/tests/test_cases/test_cocotb/test_cocotb.py
@@ -1251,5 +1251,20 @@ def test_assertion_is_failure(dut):
     assert False
 
 
+class MyException(Exception):
+    pass
+
+@cocotb.test(expect_error=MyException)
+def test_expect_particular_exception(dut):
+    yield Timer(1)
+    raise MyException()
+
+
+@cocotb.test(expect_error=(MyException, ValueError))
+def test_expect_exception_list(dut):
+    yield Timer(1)
+    raise MyException()
+
+
 if sys.version_info[:2] >= (3, 5):
     from test_cocotb_35 import *