Repository: cocotb/cocotb

Issue:
Two coroutines that fail on the same trigger are not handled gracefully by the scheduler.
```python
@cocotb.coroutine
async def coroutine1(timer):
    await timer
    assert False

@cocotb.coroutine
async def coroutine2(timer):
    await timer
    assert False

@cocotb.test()
async def test_top(dut):
    timer = triggers.Timer(100, units='ns')                                                                                            
    cocotb.fork(coroutine1(timer))                                                                                                     
    cocotb.fork(coroutine2(timer))
    await timer
```
This will fail when `_force_outcome` is called by the scheduler for the second time.  The scheduler needs to keep track of whether any coroutines that have been scheduled by this trigger have already failed.


Discussion:
This first forked coro failing should immediately stop the test and kill any outstanding forked coros.

The problem logic is the last section in `Scheduler.unschedule`. It should set `self._terminate` and call `cleanup()` (like above for the test) so the scheduler ends after handling the first failing forked coro.
Is there a similar problem if the two coroutines return different values instead of `raise`ing?
Only Exceptions cause `scheduler._abort_test()` to be called, so just returning a value shouldn't cause the same problem.
Sounds like a scheduler architecture issue. Once the test is unscheduled no more tasks should run which may also cause the test to fail.
Is it good to move this to a warning (or not to do this check), which will end the simulation clean
https://github.com/cocotb/cocotb/blob/997ee6dd906ea19954cb37f75a1406874b518db7/cocotb/scheduler.py#L1024 
```
Traceback (most recent call last):
  File "C:\Apps\WPy64-3940\python-3.9.4.amd64\lib\site-packages\cocotb\scheduler.py", line 357, in _react
    self._event_loop(trigger)
  File "C:\Apps\WPy64-3940\python-3.9.4.amd64\lib\site-packages\cocotb\scheduler.py", line 451, in _event_loop
    self._schedule(coro, trigger=trigger)
  File "C:\Apps\WPy64-3940\python-3.9.4.amd64\lib\site-packages\cocotb\scheduler.py", line 897, in _schedule
    self._unschedule(coroutine)
  File "C:\Apps\WPy64-3940\python-3.9.4.amd64\lib\site-packages\cocotb\scheduler.py", line 537, in _unschedule
    self._abort_test(e)
  File "C:\Apps\WPy64-3940\python-3.9.4.amd64\lib\site-packages\cocotb\scheduler.py", line 955, in _abort_test
    raise InternalError("Outcome already has a value, but is being set again.")
cocotb.scheduler.InternalError: Outcome already has a value, but is being set again.
```

This happens while running in middle of regression tests . and terminates the remaining tests (as in #3062). And some times questa sim will hang (but modelsim (10.4e) wont)  as well 
That's not a good solution. After a test fails, no more tasks should run. If no more tasks run, no InternalError can happen. InternalError implies there is a bug in the implementation, so that's where it should be fixed. 
#3062 shows that this affects `cocotb.start_soon`, so it is not necessarily due to the re-entrant behavior of `cocotb.fork`, so this is a legitimate scheduler bug that we might be able to fix.
Well, there are two things: a bug in the scheduler implementation, and the way the user is informed about it. If we cannot find a quick solution to the scheduler bug (and we're clearly not, this bug has been open for > 2 years), it would at least be nice to have a workaround that is least annoying to our users when they encounter the bug. A crashing regression is certainly the most extreme measure we can take. But are there less extreme measures, such as a warning? I haven't looked into the details, but it sounds like that's feasible. 
I discounted the issue because I thought the issue was due to `cocotb.fork`'s re-entrant behavior, making it unsolvable. It's the same issue, but it's kind of a different issue, so I wouldn't say it's 2 years old, there is new information since then. With `cocotb.start_soon`, this shouldn't be happening so I don't think we should be demoting it to a warning.
Thanks Guys!, its good to fix the issue without hiding it... 

Resolving patch:
diff --git a/cocotb/scheduler.py b/cocotb/scheduler.py
--- a/cocotb/scheduler.py
+++ b/cocotb/scheduler.py
@@ -265,6 +265,7 @@ def __init__(self, handle_result: Callable[[Task], None]) -> None:
         self._pending_triggers = []
         self._pending_threads = []
         self._pending_events = []  # Events we need to call set on once we've unwound
+        self._scheduling = []
 
         self._terminate = False
         self._test = None
@@ -437,7 +438,7 @@ def _event_loop(self, trigger):
                 # Scheduled coroutines may append to our waiting list so the first
                 # thing to do is pop all entries waiting on this trigger.
                 try:
-                    scheduling = self._trigger2coros.pop(trigger)
+                    self._scheduling = self._trigger2coros.pop(trigger)
                 except KeyError:
                     # GPI triggers should only be ever pending if there is an
                     # associated coroutine waiting on that trigger, otherwise it would
@@ -463,19 +464,19 @@ def _event_loop(self, trigger):
 
                 if _debug:
                     debugstr = "\n\t".join(
-                        [coro._coro.__qualname__ for coro in scheduling]
+                        [coro._coro.__qualname__ for coro in self._scheduling]
                     )
-                    if len(scheduling) > 0:
+                    if len(self._scheduling) > 0:
                         debugstr = "\n\t" + debugstr
                     self.log.debug(
                         "%d pending coroutines for event %s%s"
-                        % (len(scheduling), str(trigger), debugstr)
+                        % (len(self._scheduling), str(trigger), debugstr)
                     )
 
                 # This trigger isn't needed any more
                 trigger.unprime()
 
-                for coro in scheduling:
+                for coro in self._scheduling:
                     if coro._outcome is not None:
                         # coroutine was killed by another coroutine waiting on the same trigger
                         continue
@@ -494,6 +495,8 @@ def _event_loop(self, trigger):
                     # happened in gh-957)
                     del coro
 
+                self._scheduling = []
+
                 # Handle any newly queued coroutines that need to be scheduled
                 while self._pending_coros:
                     task = self._pending_coros.pop(0)
@@ -522,7 +525,6 @@ def _event_loop(self, trigger):
                 # to try and avoid them being destroyed at a weird time (as
                 # happened in gh-957)
                 del trigger
-                del scheduling
 
             # no more pending triggers
             self._check_termination()
@@ -1072,6 +1074,12 @@ def _cleanup(self):
                     self.log.debug("Killing %s" % str(coro))
                 coro.kill()
 
+        # if there are coroutines being scheduled when the test ends, kill them (gh-1347)
+        for coro in self._scheduling:
+            if _debug:
+                self.log.debug("Killing %s" % str(coro))
+            coro.kill()
+
         # kill any queued coroutines
         for task in self._pending_coros:
             task.kill()


Test patch:
diff --git a/tests/test_cases/test_cocotb/test_scheduler.py b/tests/test_cases/test_cocotb/test_scheduler.py
--- a/tests/test_cases/test_cocotb/test_scheduler.py
+++ b/tests/test_cases/test_cocotb/test_scheduler.py
@@ -12,7 +12,7 @@
 import re
 import warnings
 from asyncio import CancelledError, InvalidStateError
-from typing import Coroutine
+from typing import Any, Awaitable, Coroutine
 
 import pytest
 from common import MyException
@@ -794,3 +794,15 @@ async def coro():
         task.result()
     with pytest.raises(InvalidStateError):
         task.exception()
+
+
+@cocotb.test(expect_fail=True)
+async def test_multiple_concurrent_test_fails(_) -> None:
+    async def call_error(thing: Awaitable[Any]) -> None:
+        await thing
+        assert False
+
+    thing = Timer(1, "ns")
+    cocotb.start_soon(call_error(thing))
+    cocotb.start_soon(call_error(thing))
+    await Timer(10, "ns")