Repository: python-trio/trio

Issue:
Our code for recovering after a "foreign await" needs to be reworked
Just doing some testing to see if I can use `dask.distributed` with `trio` & `trio_asyncio` but have fallen at the first hurdle and hit a `TrioInternalError`:

```python
import asyncio
import trio
import trio_asyncio
from dask.distributed import Client


async def f():
    async with trio_asyncio.open_loop() as loop:
        client = await Client(asynchronous=True)
        future = client.submit(lambda x: x + 1, 10)
        return await future

trio.run(f)
```
```python-traceback
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
C:\Miniconda3\lib\site-packages\trio\_core\_run.py in run(async_fn, clock, instruments, restrict_keyboard_interrupt_to_checkpoints, *args)
   1228                     # is just to get rid of this extra indentation.
-> 1229                     result = run_impl(runner, async_fn, args)
   1230             except TrioInternalError:

C:\Miniconda3\lib\site-packages\trio\_core\_run.py in run_impl(runner, async_fn, args)
   1356                 # other exceptions as __context__ in unwanted ways.
-> 1357                 runner.task_exited(task, final_result)
   1358             else:

C:\Miniconda3\lib\site-packages\trio\_core\_run.py in task_exited(self, task, result)
    854             # the init task should be the last task to exit
--> 855             assert not self.tasks
    856         else:

AssertionError: 

The above exception was the direct cause of the following exception:

TrioInternalError                         Traceback (most recent call last)
<ipython-input-41-03c750b89f65> in <module>()
----> 1 trio.run(f)

C:\Miniconda3\lib\site-packages\trio\_core\_run.py in run(async_fn, clock, instruments, restrict_keyboard_interrupt_to_checkpoints, *args)
   1233                 raise TrioInternalError(
   1234                     "internal error in trio - please file a bug!"
-> 1235                 ) from exc
   1236             finally:
   1237                 GLOBAL_RUN_CONTEXT.__dict__.clear()

TrioInternalError: internal error in trio - please file a bug!
```

Happy to move this to the `trio_asyncio` repo if that's a more appropriate place...


Discussion:
For some context, [`dask.distributed`](https://github.com/dask/distributed) runs on top of tornado (>=5.0) which itself uses the `asyncio` event loop so I had hoped I could use it through `trio_asyncio`.

I'm fine if what I'm trying to do is too esoteric to support, I just thought it would be interesting/cool to be able to use `distributed` with `trio` rather than `asyncio` and had hoped it might Just Work. I was hoping to use `trio` as I think it will make my own code easier to understand/maintain and therefore reduce the chance I shoot myself in the foot (which I've done with `asyncio`!)

I'm mostly opening this issue in case it's of interest or an easy/obvious fix.



Yes, trio_asyncio is the right place for this bug.

No, you can't do that. You're still in a Trio context; the fact that you just started an asyncio loop doesn't change that.

This code should work (warning, literal translation but untested):
```python
from functools import partial

async def f():
    async with trio_asyncio.open_loop() as loop:
        client = await loop.run_asyncio(partial(Client, asynchronous=True))
        future = client.submit(lambda x: x + 1, 10)
        return await loop.run_future(future)
```
though you should be able to combine the last two lines to
```python
        return await loop.run_asyncio(client.submit, lambda x: x + 1, 10)
```
… and yes I need to improve error handling for this kind of mistake.
https://github.com/python-trio/trio-asyncio/issues/21
That error message from trio's side is pretty bad too... in general we should probably tweak that `TrioInternalError` so it says that `hazmat` misuse can also cause this error. In this specific case I'm also confused about how you could have gone down that particular path... trio tries to provide good error messages if you `await` a foreign function in trio context. Why didn't it fire this time?

(Re-opening so we don't forget these questions, though @smurfix may already know the answer.)
The link from #882 just reminded me to look at this again, and I figured it out. Here's a minimal reproducer:

```python
import trio
import asyncio

async def f():
    async with trio.open_nursery() as nursery:
        nursery.start_soon(trio.sleep, float("inf"))
        future = asyncio.Future()
        await future

trio.run(f)
```

* The main task creates a nursery, and starts a background task running.
* Then it does `await future`
* Trio detects that it's awaited a future, and tries to cope with this by killing the task. There's a long comment about this:

  https://github.com/python-trio/trio/blob/360f1fb45e4a3edc2fb4251dd969d0496a68ae60/trio/_core/_run.py#L1488-L1503

  As you can see, we actually zap the task (stop scheduling it, don't run any cleanup code, etc.) and hope for the best. This is very clever but...
* In our example program and in @dhirschfeld's program, this happens in the main task, so the main task gets zapped.
* Since the main task has exited, trio thinks that it's time for the whole program to exit... but we still have a task running (the `trio.sleep` in our example, trio-asyncio's background machinery in @dhirschfeld's example)
* `trio.run` explodes

So I think we need to give up on the clever task-zapping trick. Other options:

* Immediately blow up the whole program, instead of just the one task.
* Dump a warning to the console, and then `throw` in an exception and hope for the best.
thinking about this a bit more:

Blowing up the whole program is by far the easiest thing to do, but unfortunately it means users don't get any info at all about *where* they `await`ed the wrong thing, which is a pretty terrible UX :-(

We could like... recursively zap the whole task tree underneath the offending task... but this seems like a mess

So... I guess that means we need to immediately resume the task by `.throw`ing in an exception? I think that actually will work on both asyncio and curio. I've just been super-reluctant to go down that road because it complexifies the rest of trio's run loop (`.throw` is buggy, so we never use it otherwise → resuming via `.throw` complexifies the whole task state tracking machinery). But I don't see any alternative.

Resolving patch:
diff --git a/trio/_core/_run.py b/trio/_core/_run.py
--- a/trio/_core/_run.py
+++ b/trio/_core/_run.py
@@ -980,10 +980,17 @@ class Task:
     _counter = attr.ib(init=False, factory=itertools.count().__next__)
 
     # Invariant:
-    # - for unscheduled tasks, _next_send is None
-    # - for scheduled tasks, _next_send is an Outcome object,
-    #   and custom_sleep_data is None
+    # - for unscheduled tasks, _next_send_fn and _next_send are both None
+    # - for scheduled tasks, _next_send_fn(_next_send) resumes the task;
+    #   usually _next_send_fn is self.coro.send and _next_send is an
+    #   Outcome. When recovering from a foreign await, _next_send_fn is
+    #   self.coro.throw and _next_send is an exception. _next_send_fn
+    #   will effectively be at the top of every task's call stack, so
+    #   it should be written in C if you don't want to pollute Trio
+    #   tracebacks with extraneous frames.
+    # - for scheduled tasks, custom_sleep_data is None
     # Tasks start out unscheduled.
+    _next_send_fn = attr.ib(default=None)
     _next_send = attr.ib(default=None)
     _abort_func = attr.ib(default=None)
     custom_sleep_data = attr.ib(default=None)
@@ -1215,7 +1222,8 @@ def reschedule(self, task, next_send=_NO_SEND):
             next_send = Value(None)
 
         assert task._runner is self
-        assert task._next_send is None
+        assert task._next_send_fn is None
+        task._next_send_fn = task.coro.send
         task._next_send = next_send
         task._abort_func = None
         task.custom_sleep_data = None
@@ -1887,8 +1895,9 @@ def run_impl(runner, async_fn, args):
             if runner.instruments:
                 runner.instrument("before_task_step", task)
 
+            next_send_fn = task._next_send_fn
             next_send = task._next_send
-            task._next_send = None
+            task._next_send_fn = task._next_send = None
             final_outcome = None
             try:
                 # We used to unwrap the Outcome object here and send/throw its
@@ -1898,7 +1907,7 @@ def run_impl(runner, async_fn, args):
                 #   https://bugs.python.org/issue29590
                 # So now we send in the Outcome object and unwrap it on the
                 # other side.
-                msg = task.context.run(task.coro.send, next_send)
+                msg = task.context.run(next_send_fn, next_send)
             except StopIteration as stop_iteration:
                 final_outcome = Value(stop_iteration.value)
             except BaseException as task_exc:
@@ -1938,16 +1947,12 @@ def run_impl(runner, async_fn, args):
                         "other framework like asyncio? That won't work "
                         "without some kind of compatibility shim.".format(msg)
                     )
-                    # How can we resume this task? It's blocked in code we
-                    # don't control, waiting for some message that we know
-                    # nothing about. We *could* try using coro.throw(...) to
-                    # blast an exception in and hope that it propagates out,
-                    # but (a) that's complicated because we aren't set up to
-                    # resume a task via .throw(), and (b) even if we did,
-                    # there's no guarantee that the foreign code will respond
-                    # the way we're hoping. So instead we abandon this task
-                    # and propagate the exception into the task's spawner.
-                    runner.task_exited(task, Error(exc))
+                    # The foreign library probably doesn't adhere to our
+                    # protocol of unwrapping whatever outcome gets sent in.
+                    # Instead, we'll arrange to throw `exc` in directly,
+                    # which works for at least asyncio and curio.
+                    runner.reschedule(task, exc)
+                    task._next_send_fn = task.coro.throw
 
             if runner.instruments:
                 runner.instrument("after_task_step", task)


Test patch:
diff --git a/trio/_core/tests/test_run.py b/trio/_core/tests/test_run.py
--- a/trio/_core/tests/test_run.py
+++ b/trio/_core/tests/test_run.py
@@ -1782,14 +1782,31 @@ async def async_gen(arg):  # pragma: no cover
 
 
 def test_calling_asyncio_function_gives_nice_error():
-    async def misguided():
+    async def child_xyzzy():
         import asyncio
         await asyncio.Future()
 
+    async def misguided():
+        await child_xyzzy()
+
     with pytest.raises(TypeError) as excinfo:
         _core.run(misguided)
 
     assert "asyncio" in str(excinfo.value)
+    # The traceback should point to the location of the foreign await
+    assert any(  # pragma: no branch
+        entry.name == "child_xyzzy" for entry in excinfo.traceback
+    )
+
+
+async def test_asyncio_function_inside_nursery_does_not_explode():
+    # Regression test for https://github.com/python-trio/trio/issues/552
+    with pytest.raises(TypeError) as excinfo:
+        async with _core.open_nursery() as nursery:
+            import asyncio
+            nursery.start_soon(sleep_forever)
+            await asyncio.Future()
+    assert "asyncio" in str(excinfo.value)
 
 
 async def test_trivial_yields():