Repository: dask/distributed

Issue:
Accessing `ipython` on workers through `client.start_ipython_workers()`
I am trying to see if I can run ipython on workers interactively. The first example in the documentation (https://distributed.dask.org/en/latest/ipython.html) works, while the second one is not working for me. Here's the minimal code to reproduce:

```python
from dask.distributed import Client
client = Client()

info = client.start_ipython_workers(magic_names="w*")
%w0 1+1
# this gives TimeoutError: Timeout waiting for IPython output
```

This is more of a curiosity for me, just trying to explore what is possible. I posted it also as a question on StackOverflow: https://stackoverflow.com/questions/66167619/how-to-use-client-start-ipython-workers-in-dask-distributed

**Environment**:

- Dask version: 2021.02.0
- Python version: Python 3.7.9
- Operating System: MacOS Catalina
- Install method (conda, pip, source): conda



Discussion:
This used to work and be wonderful. Then I think when Tornado updated we
ended up xfailing the tests for this because it was hard to sustain.  I'm
not sure what the status is now.

@minrk do you happen to have anyone around who would be interested in
getting this back up to speed?

On Fri, Feb 19, 2021 at 11:52 AM Sultan Orazbayev <notifications@github.com>
wrote:

> I am trying to see if I can run ipython on workers interactively. The
> first example in the documentation (
> https://distributed.dask.org/en/latest/ipython.html) works, while the
> second one is not working for me. Here's the minimal code to reproduce:
>
> from dask.distributed import Clientclient = Client()
> info = client.start_ipython_workers(magic_names="w*")%w0 1+1# this gives TimeoutError: Timeout waiting for IPython output
>
> This is more of a curiosity for me, just trying to explore what is
> possible. I posted it also as a question on StackOverflow:
> https://stackoverflow.com/questions/66167619/how-to-use-client-start-ipython-workers-in-dask-distributed
>
> *Environment*:
>
>    - Dask version: 2021.02.0
>    - Python version: Python 3.7.9
>    - Operating System: MacOS Catalina
>    - Install method (conda, pip, source): conda
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/4522>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACKZTF7H3G7P7I6FNF33VLS72QMPANCNFSM4X4YZTZQ>
> .
>

Just to elaborate on one potential use-case. I wanted to run a given set of commands against multiple datasets, expecting that the commands will fail at different stages due to missing values. Some of the calculations are time-intensive and I would like to have the ability to get the state of calculations at the time of failure so I can iterate interactively. However, this was only for a very specific case, so probably not of broad interest.

Resolving patch:
diff --git a/distributed/_ipython_utils.py b/distributed/_ipython_utils.py
--- a/distributed/_ipython_utils.py
+++ b/distributed/_ipython_utils.py
@@ -199,16 +199,8 @@ def start_ipython(ip=None, ns=None, log=None):
     if get_ipython() is not None:
         raise RuntimeError("Cannot start IPython, it's already running.")
 
-    from zmq.eventloop.ioloop import ZMQIOLoop
     from ipykernel.kernelapp import IPKernelApp
 
-    # save the global IOLoop instance
-    # since IPython relies on it, but we are going to put it in a thread.
-    save_inst = IOLoop.instance()
-    IOLoop.clear_instance()
-    zmq_loop = ZMQIOLoop()
-    zmq_loop.install()
-
     # start IPython, disabling its signal handlers that won't work due to running in a thread:
     app = IPKernelApp.instance(log=log)
     # Don't connect to the history database
@@ -234,20 +226,17 @@ def _start():
         app.kernel.pre_handler_hook = noop
         app.kernel.post_handler_hook = noop
         app.kernel.start()
-        app.kernel.loop = IOLoop.instance()
         # save self in the IPython namespace as 'worker'
         # inject things into the IPython namespace
         if ns:
             app.kernel.shell.user_ns.update(ns)
         evt.set()
-        zmq_loop.start()
+        # start the app's IOLoop in its thread
+        IOLoop.current().start()
 
     zmq_loop_thread = Thread(target=_start)
     zmq_loop_thread.daemon = True
     zmq_loop_thread.start()
     assert evt.wait(timeout=5), "IPython didn't start in a reasonable amount of time."
 
-    # put the global IOLoop instance back:
-    IOLoop.clear_instance()
-    save_inst.install()
     return app


Test patch:
diff --git a/distributed/tests/test_ipython.py b/distributed/tests/test_ipython.py
--- a/distributed/tests/test_ipython.py
+++ b/distributed/tests/test_ipython.py
@@ -2,7 +2,6 @@
 
 import pytest
 from tlz import first
-import tornado
 
 from distributed import Client
 from distributed.utils_test import cluster, mock_ipython
@@ -15,11 +14,7 @@ def need_functional_ipython(func):
         import jupyter_client  # noqa: F401
     except ImportError:
         return pytest.mark.skip("need ipykernel and jupyter_client installed")(func)
-    if tornado.version_info >= (5,):
-        # https://github.com/ipython/ipykernel/issues/277
-        return pytest.mark.skip("IPython kernel broken with Tornado 5")(func)
-    else:
-        return func
+    return func
 
 
 @pytest.mark.ipython