Repository: cocotb/cocotb

Issue:
Stack Trace shows scheduler code and not user code under some circumstances
cocotb version: 1.8.2
operating system: Ubuntu 22.04 64 bit
Simulator: Riviera Pro 2022.10 64 bit
Python Version: 3.11.1 self install

Description: Under certain conditions an assertion failure results in a stack trace from scheduler code and not the user code that is typical. The exact conditions is two or more start_soon's in a row

I've attached an example using the new python runner. To run an example where the assertion failure output is correct run:
`TESTCASE=dummy_test_good SIM=riviera HDL_TOPLEVEL_LANG=vhdl pytest test_dummy.py`

and to run an example where the output is incorrect run:
`TESTCASE=dummy_test_bad SIM=riviera HDL_TOPLEVEL_LANG=vhdl pytest test_dummy.py`

Below is an example of the bad output:
```
  exec(co, module.__dict__)
     0.00ns INFO     cocotb.regression                  Found test test_dummy.dummy_test_bad
     0.00ns INFO     cocotb.regression                  Found test test_dummy.dummy_test_good
     0.00ns INFO     cocotb.regression                  running dummy_test_bad (1/2)
Traceback (most recent call last):
  File "/home/jprice/aps_env/lib/python3.11/site-packages/cocotb/scheduler.py", line 633, in _resume_coro_upon
    trigger.prime(self._react)
  File "/home/jprice/aps_env/lib/python3.11/site-packages/cocotb/triggers.py", line 685, in prime
    callback(self)
  File "/home/jprice/aps_env/lib/python3.11/site-packages/cocotb/scheduler.py", line 384, in _react
    self._event_loop(trigger)
  File "/home/jprice/aps_env/lib/python3.11/site-packages/cocotb/scheduler.py", line 487, in _event_loop
    self._schedule(coro, trigger=trigger)
  File "/home/jprice/aps_env/lib/python3.11/site-packages/cocotb/scheduler.py", line 957, in _schedule
    self._unschedule(coroutine)
  File "/home/jprice/aps_env/lib/python3.11/site-packages/cocotb/scheduler.py", line 571, in _unschedule
    self._cleanup()
  File "/home/jprice/aps_env/lib/python3.11/site-packages/cocotb/scheduler.py", line 1094, in _cleanup
    assert not self._pending_coros
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jprice/aps_env/lib/python3.11/site-packages/cocotb/__init__.py", line 233, in _initialise_testbench
    _initialise_testbench_(argv_)
  File "/home/jprice/aps_env/lib/python3.11/site-packages/cocotb/__init__.py", line 322, in _initialise_testbench_
    regression_manager._execute()
  File "/home/jprice/aps_env/lib/python3.11/site-packages/cocotb/regression.py", line 554, in _execute
    return self._start_test()
           ^^^^^^^^^^^^^^^^^^
  File "/home/jprice/aps_env/lib/python3.11/site-packages/cocotb/regression.py", line 576, in _start_test
    cocotb.scheduler._add_test(self._test_task)
  File "/home/jprice/aps_env/lib/python3.11/site-packages/cocotb/scheduler.py", line 842, in _add_test
    self._resume_coro_upon(
  File "/home/jprice/aps_env/lib/python3.11/site-packages/cocotb/scheduler.py", line 636, in _resume_coro_upon
    self._trigger2coros.pop(trigger)
KeyError: <NullTrigger for Start <Test dummy_test_bad> at 0x7f8ea30afb50>
     0.00ns ERROR    cocotb.scheduler                   Failing test at simulator request before test run completion: Simulator shut down prematurely
sys:1: RuntimeWarning: coroutine 'dummy_coroutine' was never awaited
KERNEL: Simulation has finished. There are no more test vectors to simulate.
exit
VSIM: Simulation has finished.
```



Discussion:
@jwprice100 Did you forget to attach the example?
Whoops sorry here you go!

[example.zip](https://github.com/cocotb/cocotb/files/11936473/example.zip)

The cause is this for loop:
https://github.com/cocotb/cocotb/blob/08ef2f60f2816eda822ca953bc148fca40d2f506/cocotb/scheduler.py#L1091-L1094

`task.kill()` calls `Scheduler._unschedule()`, which removes the task from `Scheduler._pending_coros` while we are iterating over it.

The assert was added in 1.8, exposing the problem.
I'm guessing we just need to change that to `while self._pending_coros`, then we can remove the assert?
That could pretty easily go into a 1.8.1.
I would appreciate it if the fix could be incorporated. I love the Riviera fix in 1.8 but now whenever I start up a new testbench I get a million hard to trace exceptions. I'm happy to test it.

Resolving patch:
diff --git a/cocotb/scheduler.py b/cocotb/scheduler.py
--- a/cocotb/scheduler.py
+++ b/cocotb/scheduler.py
@@ -1088,8 +1088,11 @@ def _cleanup(self):
             trigger.unprime()
         assert not self._pending_triggers
 
-        # kill any queued coroutines
-        for task in self._pending_coros:
+        # Kill any queued coroutines.
+        # We use a while loop because task.kill() calls _unschedule(), which will remove the task from _pending_coros.
+        # If that happens a for loop will stop early and then the assert will fail.
+        while self._pending_coros:
+            task = self._pending_coros.pop(0)
             task.kill()
         assert not self._pending_coros
 


Test patch:
diff --git a/tests/test_cases/test_cocotb/test_scheduler.py b/tests/test_cases/test_cocotb/test_scheduler.py
--- a/tests/test_cases/test_cocotb/test_scheduler.py
+++ b/tests/test_cases/test_cocotb/test_scheduler.py
@@ -610,6 +610,18 @@ async def test_previous_start_soon_not_scheduled(_):
     assert start_soon_started is False
 
 
+@cocotb.test()
+async def test_test_end_with_multiple_pending_tasks(_):
+    """Test ending a test with multiple tasks queued by start_soon."""
+
+    async def coro():
+        return 0
+
+    # gh-3354
+    cocotb.start_soon(coro())
+    cocotb.start_soon(coro())
+
+
 @cocotb.test()
 async def test_start(_):
     async def coro():