Repository: python-trio/trio

Issue:
Delay deciding which cancel scope a Cancelled exception belongs to
Right now, whenever we create a `Cancelled` exception, we immediately tag it with the cancel scope that's going to catch it.

There's another option that's briefly mentioned in #606: we could wait until `CancelScope.__exit__` to decide whether a given `Cancelled` exception "belongs" to this scope. The check would be "on the current task's stack, am I the outermost unshielded scope that's in the cancelled state?"

This can behave differently from what we have now in cases where there are multiple cancelled `CancelScope`s on the stack.

I'm not entirely sure if this would actually help with #606, because of complications noted there (hello `KeyboardInterrupt`). But it would probably simplify the cancel scope code in general, in particular `_cancel_no_notify`, and the issues described in this comment: https://github.com/python-trio/trio/pull/835#issuecomment-455107324


Discussion:
I just posted #900 which offers a less elegant, but also less invasive, solution to this problem. I prefer the solution in this issue on aesthetic grounds but I couldn't convince myself that it would be safe in the presence of a cancellation landscape that might be changing as the `Cancelled` exception propagates. Also, it makes the delivery of a cancellation be quadratic in the depth of the cancel stack, but probably that doesn't add up to much in practice?

OK, let's try doing this analysis semi-rigorously.

I'm interpreting "outermost unshielded scope that's in the cancelled state" here as "the thing that `pending_cancel_scope()` returns", with the properties:
* it's always a cancel scope with `cancel_called = True`, or `None`
* it's only `None` if the innermost cancelled scope encloses the innermost shielded scope

Looking from the perspective of a single `CancelScope.__exit__` with a `Cancelled` exception propagating:
* If `cancel_called` is false, we can't possibly be the target of this exception; let it continue.
* If this scope is `pending_cancel_scope()`, swallow the exception and set `cancelled_caught`, we're done. By premise, we did the right thing for the current state of the cancel stack.
* If not, then since we're at the top of the cancel stack, whatever `pending_cancel_scope()` returned must be outside us.
  * It can't be None, since you only get None if the innermost shield is further in than the innermost cancelled scope, and we're a cancelled scope and we're the furthest-in thing there is.
  * So it must be some cancel scope outside us (call it C), and we should let the exception keep propagating so they can catch it.
    * By the time the exception gets to C, the result of `pending_cancel_scope()` might well have changed. But for C to have been returned from `pending_cancel_scope()` in the first place, it must have had `cancel_called = True`, and `cancel_called` never goes from True to False. So if the exception gets to C, either `pending_cancel_scope()` will return C (i.e., nothing changed), or it will return some scope outside C, in which case it will be proper for C to let the exception continue propagating to that further-out scope. It can't return None for C for the same reasons it couldn't return None for us: C is the innermost CancelScope at that point and it's cancelled, no possibility for a shield further in.
    * Something something induction eventually we run out of cancel stack.

I think my conclusion here is that I was being overly conservative, this will work fine, and we can drop #900. But I'd like another set of eyes on it before being quite sure. :-)
> couldn't convince myself that it would be safe in the presence of a cancellation landscape that might be changing as the `Cancelled` exception propagates

Good question, yeah.

Right now, at the moment when we inject the `Cancelled` exception, we look at a single consistent snapshot of the whole stack, determine where the outermost-unshielded-cancelled-scope is, and then lock that in as the destination of the exception.

In this proposal, each time the exception passes through a cancel scope, we look at a single consistent snapshot of the whole stack, and determine whether the current scope is the outermost-unshielded-cancelled-scope. (So the naive implementation is O(stack depth * number of scopes in the cancel-requested state), which is probably much better than O(stack depth * stack depth), and I think this could be optimized further.)

Now, if we discover that this *is* the outermost-unshielded-cancelled-scope, and catch the exception, then that's definitely OK, because it was the right thing to do at the moment we did it, and then there's nothing afterwards that could go wrong.

The riskier case is when we decide *not* to catch the exception. We know that at the moment we made the decision, there was some outer scope that was a better candidate for catching it. But, what it won't actually be caught until later. Could something happen between now and then that would mess things up? In particular, the thing that would be bad is if we got into a state where there was a live `Cancelled` propagating, but there *was no* outer scope that was a candidate for catching it. So, could that outer scope somehow stop being a candidate between when we raise the exception and when the exception reaches it?

To be a candidate, a scope has to meet two criteria:

* It has to be in the cancel-requested state
* There can't be any other scope that's in the cancel-requested state and that falls between our candidate and the next outermost shield

Once a scope is in the cancel-requested state, it can never leave it. So if we identified a scope as being a candidate at time T1, then for all times T2 > T1 it is definitely still in the cancel-requested state. But what about the second criterion? It could certainly happen that our original candidate stops fulfilling that – while our `Cancelled` is propagating, an outer scope could enter the cancel-requested state, or a shield could be removed to reveal an outer scope that was already in the cancel-requested state. But, in all of these cases, it means that our candidate is *replaced by another, even-more-outer candidate*.

Therefore, if in a snapshot at time T1 we decide that our exception has an outer candidate, then we are guaranteed that it will eventually be caught at time T2, even if it's by a different scope than the one we originally thought.

Ah, and I see you've just posted osmething too! I will read it.
OK looks like we made pretty much the same argument :-)
Put another way, if a cancel scope does interpret a Cancelled exception as being "for us", we can assume that determination is correct (as a matter of definition - the mapping from cancel stacks to "which scope is cancelled here right now" defines the cancellation semantics, we're just following it). So the only real correctness concern is that everyone could conclude "not mine!" and let a `Cancelled` exception fly out of the entire program. But for a cancelled scope to say "not mine!", it has to find some specific other further-out cancelled scope that it believes will say "mine!", based on the current state of the cancel stack. Since scopes never become uncancelled, at minimum the outermost cancelled scope will be forced to conclude "mine!". So as long as we never throw `Cancelled` into a task without any cancelled scopes on its cancel stack, we'll be OK.
Great, I'm sufficiently convinced now and will close #900 and work on this instead. :-)

Resolving patch:
diff --git a/trio/_core/_exceptions.py b/trio/_core/_exceptions.py
--- a/trio/_core/_exceptions.py
+++ b/trio/_core/_exceptions.py
@@ -46,11 +46,9 @@ class Cancelled(BaseException):
 
     then this *won't* catch a :exc:`Cancelled` exception.
 
-    Attempting to raise :exc:`Cancelled` yourself will cause a
-    :exc:`RuntimeError`. It would not be associated with a cancel scope and thus
-    not be caught by Trio. Use
-    :meth:`cancel_scope.cancel() <trio.CancelScope.cancel>`
-    instead.
+    You cannot raise :exc:`Cancelled` yourself. Attempting to do so
+    will produce a :exc:`RuntimeError`. Use :meth:`cancel_scope.cancel()
+    <trio.CancelScope.cancel>` instead.
 
     .. note::
 
@@ -65,7 +63,6 @@ class Cancelled(BaseException):
        everywhere.
 
     """
-    _scope = None
     __marker = object()
 
     def __init__(self, _marker=None):
diff --git a/trio/_core/_run.py b/trio/_core/_run.py
--- a/trio/_core/_run.py
+++ b/trio/_core/_run.py
@@ -342,15 +342,6 @@ def shield(self, new_value):
             for task in self._tasks:
                 task._attempt_delivery_of_any_pending_cancel()
 
-    def _cancel_no_notify(self):
-        # returns the affected tasks
-        if not self.cancel_called:
-            with self._might_change_effective_deadline():
-                self.cancel_called = True
-            return self._tasks
-        else:
-            return set()
-
     @enable_ki_protection
     def cancel(self):
         """Cancels this scope immediately.
@@ -358,7 +349,11 @@ def cancel(self):
         This method is idempotent, i.e., if the scope was already
         cancelled then this method silently does nothing.
         """
-        for task in self._cancel_no_notify():
+        if self.cancel_called:
+            return
+        with self._might_change_effective_deadline():
+            self.cancel_called = True
+        for task in self._tasks:
             task._attempt_delivery_of_any_pending_cancel()
 
     def _add_task(self, task):
@@ -379,24 +374,22 @@ def _tasks_removed_by_adoption(self, tasks):
     def _tasks_added_by_adoption(self, tasks):
         self._tasks.update(tasks)
 
-    def _make_exc(self):
-        exc = Cancelled._init()
-        exc._scope = self
-        return exc
-
     def _exc_filter(self, exc):
-        if isinstance(exc, Cancelled) and exc._scope is self:
+        if (
+            isinstance(exc, Cancelled) and self.cancel_called
+            and self._scope_task._pending_cancel_scope() is self
+        ):
             self.cancelled_caught = True
             return None
         return exc
 
     def _close(self, exc):
+        if exc is not None:
+            exc = MultiError.filter(self._exc_filter, exc)
         with self._might_change_effective_deadline():
             self._remove_task(self._scope_task)
         self._scope_task = None
-        if exc is not None:
-            return MultiError.filter(self._exc_filter, exc)
-        return None
+        return exc
 
 
 @deprecated("0.10.0", issue=607, instead="trio.CancelScope")
@@ -432,9 +425,8 @@ def started(self, value=None):
 
         # If the old nursery is cancelled, then quietly quit now; the child
         # will eventually exit on its own, and we don't want to risk moving
-        # the children into a different scope while they might have
-        # propagating Cancelled exceptions that assume they're under the old
-        # scope.
+        # children that might have propagating Cancelled exceptions into
+        # a place with no cancelled cancel scopes to catch them.
         if _pending_cancel_scope(self._old_nursery._cancel_stack) is not None:
             return
 
@@ -758,13 +750,11 @@ def _attempt_abort(self, raise_cancel):
     def _attempt_delivery_of_any_pending_cancel(self):
         if self._abort_func is None:
             return
-        pending_scope = self._pending_cancel_scope()
-        if pending_scope is None:
+        if self._pending_cancel_scope() is None:
             return
-        exc = pending_scope._make_exc()
 
         def raise_cancel():
-            raise exc
+            raise Cancelled._init()
 
         self._attempt_abort(raise_cancel)
 
@@ -1523,21 +1513,16 @@ def run_impl(runner, async_fn, args):
         if runner.instruments:
             runner.instrument("after_io_wait", timeout)
 
+        # Process cancellations due to deadline expiry
         now = runner.clock.current_time()
-        # We process all timeouts in a batch and then notify tasks at the end
-        # to ensure that if multiple timeouts occur at once, then it's the
-        # outermost one that gets delivered.
-        cancelled_tasks = set()
         while runner.deadlines:
             (deadline, _), cancel_scope = runner.deadlines.peekitem(0)
             if deadline <= now:
                 # This removes the given scope from runner.deadlines:
-                cancelled_tasks.update(cancel_scope._cancel_no_notify())
+                cancel_scope.cancel()
                 idle_primed = False
             else:
                 break
-        for task in cancelled_tasks:
-            task._attempt_delivery_of_any_pending_cancel()
 
         if not runner.runq and idle_primed:
             while runner.waiting_for_idle:


Test patch:
diff --git a/trio/_core/tests/test_run.py b/trio/_core/tests/test_run.py
--- a/trio/_core/tests/test_run.py
+++ b/trio/_core/tests/test_run.py
@@ -637,21 +637,21 @@ async def crasher():
                     nursery.start_soon(crasher)  # t4
                 # and then our __aexit__ also receives an outer Cancelled
             except _core.MultiError as multi_exc:
-                # This is outside the nursery scope but inside the outer
-                # scope, so the nursery should have absorbed t1 and t2's
-                # exceptions but t3 and t4 should remain, plus the Cancelled
-                # from 'outer'
-                assert len(multi_exc.exceptions) == 3
+                # Since the outer scope became cancelled before the
+                # nursery block exited, all cancellations inside the
+                # nursery block continue propagating to reach the
+                # outer scope.
+                assert len(multi_exc.exceptions) == 5
                 summary = {}
                 for exc in multi_exc.exceptions:
                     summary.setdefault(type(exc), 0)
                     summary[type(exc)] += 1
-                assert summary == {_core.Cancelled: 2, KeyError: 1}
+                assert summary == {_core.Cancelled: 4, KeyError: 1}
                 raise
     except AssertionError:  # pragma: no cover
         raise
     except BaseException as exc:
-        # This is ouside the outer scope, so the two outer Cancelled
+        # This is ouside the outer scope, so all the Cancelled
         # exceptions should have been absorbed, leaving just a regular
         # KeyError from crasher()
         assert type(exc) is KeyError
@@ -1400,7 +1400,7 @@ async def main():
     assert len(record) == 2
     assert record[0] == "starting"
     assert record[1][0] == "run finished"
-    assert record[1][1] >= 20
+    assert record[1][1] >= 19
 
 
 def test_TrioToken_run_sync_soon_threaded_stress_test():
diff --git a/trio/tests/test_highlevel_open_tcp_stream.py b/trio/tests/test_highlevel_open_tcp_stream.py
--- a/trio/tests/test_highlevel_open_tcp_stream.py
+++ b/trio/tests/test_highlevel_open_tcp_stream.py
@@ -397,9 +397,11 @@ async def test_multi_success(autojump_clock):
         happy_eyeballs_delay=1,
     )
     assert not scenario.sockets["1.1.1.1"].succeeded
-    assert scenario.sockets["2.2.2.2"].succeeded
-    assert scenario.sockets["3.3.3.3"].succeeded
-    assert scenario.sockets["4.4.4.4"].succeeded
+    assert (
+        scenario.sockets["2.2.2.2"].succeeded
+        or scenario.sockets["3.3.3.3"].succeeded
+        or scenario.sockets["4.4.4.4"].succeeded
+    )
     assert not scenario.sockets["5.5.5.5"].succeeded
     assert sock.ip in ["2.2.2.2", "3.3.3.3", "4.4.4.4"]
     assert trio.current_time() == (0.5 + 10)