Repository: cocotb/cocotb

Issue:
RecursionError in _resume_coro_upon when error happen
<!--
In case you are reporting a problem with running cocotb,
please remember to add relevant information about your environment, such as
* the cocotb version used,
* the operating system and version (32/64 bit),
* the simulator and version (32/64 bit),
* the Python version, and where it's coming from (e.g. system, Anaconda, self-installed, ...),
* (part of) the log file (cleaned of proprietary information).
If you are unsure about any of the above items, open the issue anyway and we will figure it out together.
Thanks for helping improve cocotb!
-->

Using :
 - Cocotb 1.7.2
 - Ubuntu 22.04 (In WSL 2.0 for Windows 10) 
 - Verilator 5.004

When an exception occurs in the scheduler `_react` function on:
```python
        if self._pending_triggers:
            raise InternalError(
                "Expected all triggers to be handled but found {}".format(
                    self._pending_triggers
                )
            )
```
With remaining triggers, during the call to the `trigger.prime` function of the `_resume_coro_upon()` :
```python
if not trigger.primed:
    # ... 
    try:
        trigger.prime(self._react)
    except Exception as e:
        # discard the trigger we associated, it will never fire
        self._trigger2coros.pop(trigger)
        # replace it with a new trigger that throws back the exception
        self._resume_coro_upon(
            coro,
            NullTrigger(
                name="Trigger.prime() Error", outcome=outcomes.Error(e)
            ),
        )
```

The call of `self._resume_coro_upon` with the `NullTrigger` will end up throwing the same `InternalError`, hence recursively throwing until Python itself explodes.

I managed to avoid the recursion error by changing the code around `InternalError`. Adding a `self._cleanup()` force the scheduler to cleanup the remaining triggers and fail in another way, after properly reporting the `InternalError`. 

The resulting code for `_react` is:
```python
def _react(self, trigger):
    """
    Called when a trigger fires.
    We ensure that we only start the event loop once, rather than
    letting it recurse.
    """
    # ... 
    if self._pending_triggers:
        self._cleanup() # <--- Force the cleanup of remaining _pending_triggers
        raise InternalError(
            "Expected all triggers to be handled but found {}".format(
                self._pending_triggers
            )
        )
```



Discussion:
I think instead `InternalError` should either specifically handled in the wrapper around `trigger.prime` or it should become a `BaseException`.
IMHO, the `Internal Error`  is fine in itself. The actual issue is that the error management replicate the problem that raised the error in the first place.

Maybe you could have a custom error (For example `RemainingTriggersError`) that you handle specifically if you wish to keep the current error management. 
Then you could have `except RemainingTriggersError`  and `except Exception` side by side or test the type of the exception and manage them separately as you see fit. 
I tend to agree that it should be a `BaseException` and caught in `_react()`, rather than passing it to the `Task`. We should clean up and fail current test and remaining tests if there in an internal scheduler error, the same as if the simulator fails and stops early.
@suzizecat Can you provide a test for this?

Resolving patch:
diff --git a/cocotb/scheduler.py b/cocotb/scheduler.py
--- a/cocotb/scheduler.py
+++ b/cocotb/scheduler.py
@@ -76,7 +76,7 @@
 _debug = "COCOTB_SCHEDULER_DEBUG" in os.environ
 
 
-class InternalError(RuntimeError):
+class InternalError(BaseException):
     """An error internal to scheduler. If you see this, report a bug!"""
 
     pass
@@ -623,7 +623,6 @@ def _resume_coro_upon(self, coro, trigger):
             trigger_coros.append(coro)
 
         if not trigger.primed:
-
             if trigger_coros != [coro]:
                 # should never happen
                 raise InternalError(
@@ -981,7 +980,6 @@ def _schedule(self, coroutine, trigger=None):
             # where a sim might change what this thread is.
 
             if self._main_thread is threading.current_thread():
-
                 for ext in self._pending_threads:
                     ext.thread_start()
                     if _debug:
@@ -1064,7 +1062,7 @@ def _cleanup(self):
         Unprime all pending triggers and kill off any coroutines, stop all externals.
         """
         # copy since we modify this in kill
-        items = list(self._trigger2coros.items())
+        items = list((k, list(v)) for k, v in self._trigger2coros.items())
 
         # reversing seems to fix gh-928, although the order is still somewhat
         # arbitrary.
@@ -1073,16 +1071,27 @@ def _cleanup(self):
                 if _debug:
                     self.log.debug("Killing %s" % str(coro))
                 coro.kill()
+        assert not self._trigger2coros
 
         # if there are coroutines being scheduled when the test ends, kill them (gh-1347)
         for coro in self._scheduling:
             if _debug:
                 self.log.debug("Killing %s" % str(coro))
             coro.kill()
+        self._scheduling = []
+
+        # cancel outstanding triggers *before* queued coroutines (gh-3270)
+        while self._pending_triggers:
+            trigger = self._pending_triggers.pop(0)
+            if _debug:
+                self.log.debug("Unpriming %r", trigger)
+            trigger.unprime()
+        assert not self._pending_triggers
 
         # kill any queued coroutines
         for task in self._pending_coros:
             task.kill()
+        assert not self._pending_coros
 
         if self._main_thread is not threading.current_thread():
             raise Exception("Cleanup() called outside of the main thread")


Test patch:
diff --git a/tests/test_cases/test_3270/Makefile b/tests/test_cases/test_3270/Makefile
new file mode 100644
--- /dev/null
+++ b/tests/test_cases/test_3270/Makefile
@@ -0,0 +1,20 @@
+# Copyright cocotb contributors
+# Licensed under the Revised BSD License, see LICENSE for details.
+# SPDX-License-Identifier: BSD-3-Clause
+
+TOPLEVEL_LANG ?= verilog
+
+ifneq ($(TOPLEVEL_LANG),verilog)
+
+sim:
+	@echo Only Verilog is supported for this test, skipping...
+
+else
+
+MODULE := test_3270
+VERILOG_SOURCES := test_3270.v
+TOPLEVEL := trigger_counter
+
+include $(shell cocotb-config --makefiles)/Makefile.sim
+
+endif
diff --git a/tests/test_cases/test_3270/test_3270.py b/tests/test_cases/test_3270/test_3270.py
new file mode 100644
--- /dev/null
+++ b/tests/test_cases/test_3270/test_3270.py
@@ -0,0 +1,108 @@
+# Copyright cocotb contributors
+# Licensed under the Revised BSD License, see LICENSE for details.
+# SPDX-License-Identifier: BSD-3-Clause
+import cocotb
+from cocotb.clock import Clock
+from cocotb.triggers import First, RisingEdge, Timer
+
+
+class MonitorChange:
+    def __init__(self, signal, valid, expect_change=True) -> None:
+        self.monitor_process = None
+        self.signal = signal
+        self.valid = valid
+        self.expect_change = expect_change
+
+    def __enter__(self):
+        self.start_monitor()
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        if exc_type is None:
+            self.assert_change()
+            return True
+        else:
+            self.stop_monitor()
+            return False
+
+    async def _monitor_req_design_update(self):
+        update = False
+
+        while not update:
+            await First(RisingEdge(self.signal), RisingEdge(self.valid))
+            update = self.signal.value == 1 and self.valid.value == 1
+
+        assert self.expect_change, "Unexpected change happened"
+
+    def stop_monitor(self):
+        if self.monitor_process is not None:
+            self.monitor_process.kill()
+        self.monitor_process = None
+
+    def start_monitor(self):
+        self.stop_monitor()
+        self.monitor_process = cocotb.start_soon(self._monitor_req_design_update())
+
+    def assert_change(self):
+        is_ok = True
+        if not (self.monitor_process is None or self.monitor_process.done()):
+            is_ok = not self.expect_change
+        self.stop_monitor()
+        assert is_ok, "Change SHOULD have happened"
+
+
+async def init_dut(dut):
+    dut.i_rst_n.value = 1
+    dut.i_clk.value = 0
+    dut.i_trg.value = 0
+    dut.i_valid.value = 0
+    dut.i_cnt.value = 0
+    await Timer(1)
+    dut.i_rst_n.value = 0
+    await Timer(10, "ns")
+    dut.i_rst_n.value = 1
+    return cocotb.start_soon(Clock(dut.i_clk, 1, "ns").start())
+
+
+@cocotb.test()
+async def buggy(dut):
+    clk = await init_dut(dut)
+
+    async def stuff(dut):
+        with MonitorChange(dut.o_pulse, dut.o_valid, expect_change=False):
+            with MonitorChange(dut.i_trg, dut.o_valid, expect_change=False):
+                await Timer(10, "us")
+
+    coro = cocotb.start_soon(stuff(dut))
+
+    await Timer(5, "ns")
+    await RisingEdge(dut.i_clk)
+    dut.i_cnt.value = 5
+    dut.i_trg.value = 1
+    dut.i_valid.value = 0
+    await RisingEdge(dut.i_clk)
+    dut.i_trg.value = 0
+
+    await RisingEdge(dut.o_pulse)
+    clk.kill()
+    coro.kill()
+    await Timer(1, "us")
+
+
+@cocotb.test()
+async def basic_ok(dut):
+    clk = await init_dut(dut)
+
+    with MonitorChange(dut.o_pulse, dut.o_valid, expect_change=True):
+        await Timer(5, "ns")
+        await RisingEdge(dut.i_clk)
+        dut.i_cnt.value = 5
+        dut.i_trg.value = 1
+        dut.i_valid.value = 1
+        await RisingEdge(dut.i_clk)
+        dut.i_trg.value = 0
+
+        await RisingEdge(dut.o_pulse)
+        await Timer(10, "ns")
+
+    await Timer(10, "ns")
+    clk.kill()
diff --git a/tests/test_cases/test_3270/test_3270.v b/tests/test_cases/test_3270/test_3270.v
new file mode 100644
--- /dev/null
+++ b/tests/test_cases/test_3270/test_3270.v
@@ -0,0 +1,35 @@
+// Copyright cocotb contributors
+// Licensed under the Revised BSD License, see LICENSE for details.
+// SPDX-License-Identifier: BSD-3-Clause
+
+module trigger_counter (
+    input logic i_clk,
+    input logic i_rst_n,
+    input logic i_trg,
+    input logic i_valid,
+    input logic [7:0] i_cnt,
+    output logic o_pulse,
+    output logic o_valid
+);
+
+
+  logic [7:0] cnt;
+
+  always @(posedge i_clk or negedge i_rst_n) begin : p_seq_
+    if (~i_rst_n) begin
+      cnt <= 0;
+      o_pulse <= 0;
+      o_valid <= 0;
+    end else begin
+      o_pulse <= 0;
+      o_valid <= i_valid;
+      if (i_trg) begin
+        cnt <= i_cnt;
+      end else if (cnt != 0) begin
+        cnt <= cnt - 1;
+        o_pulse <= (cnt == 1) ? 1'b1 : 1'b0;
+      end
+    end
+  end
+
+endmodule