Repository: ipython/ipyparallel

Issue:
client().shutdown() gives errors with KeyError: None and OSError: [WinError 10038] on the console
Dear @minrk and all, I've been pulling my hair for these past few days trying to understand what's wrong with my codes and how to fix it, so I hope you can help me!

Just so can you understand my "journey": I'm using ipyparallel together with bluepyopt to do a morphospace exploration, and the script calls for matlab to run a function with given values that fluctuate between a range. When I run my code with parallelization (which I do manually with ipcontroller and ipengine), all engines can finish except one, which stays "frozen" in matlab for hours (longest time I have left it running was around 8h, while all other engines finish in less than 1h). However, if I run the codes by calling the matlab function everytime with the same values, all engines complete in less than 1h, and in the end of the code I have a shutdown request by the client that gives an error.  So I decided to go back to basics and see if this was an overall problem with ipyparallel.
I just did a script that starts a controller with one engine, starts a client, prints the id, and then does a shutdown request.
**The code is as follow:**

`import os`
`import ipyparallel as ipp`
`from time import sleep`
`os.system('''START "controller" ipcontroller --log-to-file &''')`
`sleep(15)`
`tc = ipp.Client()`
`os.system('''START "engine 0" ipengine &''')`
`sleep(30)`
`tc = ipp.Client()`
`print(tc)`
`print(tc.ids)`
`sleep(10)`
`tc.shutdown(hub = True)`

**The console output:**

`<ipyparallel.client.client.Client object at 0x000001CE4D8714E0>`
`[0]`

**The controller window, right before shuting down, shows:**

`Unhandled message type: 'shutdown_notification'`
`ERROR:scheduler:Unhandled message type: 'shutdown_notification'`

**The error I see in logfile of the controller:**

`ERROR | Uncaught exception in <function Hub.dispatch_monitor_traffic at 0x000001F9A099C6A8>`
`Traceback (most recent call last):`
`  File "C:\Users\Natii\Anaconda3\lib\site-packages\ipyparallel\util.py", line 119, in log_errors`
`    return f(self, *args, **kwargs)`
`  File "C:\Users\Natii\Anaconda3\lib\site-packages\ipyparallel\controller\hub.py", line 520, in dispatch_monitor_traffic`
`    handler(idents, msg)`
`  File "C:\Users\Natii\Anaconda3\lib\site-packages\ipyparallel\controller\hub.py", line 846, in monitor_iopub_message`
`    uuid = self.engines[eid].uuid`
`KeyError: None`
`client::client b'\x00\x80\x00\x00-' requested 'shutdown_request'`
`hub::hub shutting down.`

**I also sometimes (yes, only sometimes, don't ask me why...) get this output on the console:**
(in the first exception, the value of the handle, **2204**, changes every time, and in the second exception, the number of the thread, Thread-**16**, also changes)

`Exception in callback BaseAsyncIOLoop._handle_events(2204, 1)`
`handle: <Handle BaseAsyncIOLoop._handle_events(2204, 1)>`
`Traceback (most recent call last):`
`   File "C:\Users\Natii\Anaconda3\lib\asyncio\events.py", line 88, in _run`
`    self._context.run(self._callback, *self._args)`
`  File "C:\Users\Natii\Anaconda3\lib\site-packages\tornado\platform\asyncio.py", line 122, in _handle_events`
`    handler_func(fileobj, events)`
`  File "C:\Users\Natii\Anaconda3\lib\site-packages\tornado\stack_context.py", line 300, in null_wrapper`
`    return fn(*args, **kwargs)`
`  File "C:\Users\Natii\Anaconda3\lib\site-packages\zmq\eventloop\zmqstream.py", line 465, in _handle_events`
`    self._rebuild_io_state()`
`  File "C:\Users\Natii\Anaconda3\lib\site-packages\zmq\eventloop\zmqstream.py", line 522, in _rebuild_io_state`
`    self._update_handler(state)`
`  File "C:\Users\Natii\Anaconda3\lib\site-packages\zmq\eventloop\zmqstream.py", line 539, in _update_handler`
`    if state & self.socket.events:`
`  File "C:\Users\Natii\Anaconda3\lib\site-packages\zmq\sugar\attrsettr.py", line 48, in __getattr__`
`    return self._get_attr_opt(upper_key, opt)`
`  File "C:\Users\Natii\Anaconda3\lib\site-packages\zmq\sugar\attrsettr.py", line 52, in _get_attr_opt`
`    return self.get(opt)`
`  File "zmq\backend\cython\socket.pyx", line 477, in zmq.backend.cython.socket.Socket.get`
`  File "zmq\backend\cython\socket.pyx", line 135, in zmq.backend.cython.socket._check_closed`
`zmq.error.ZMQError: Unknown error`
`Exception in thread Thread-16:`
`Traceback (most recent call last):`
`  File "C:\Users\Natii\Anaconda3\lib\threading.py", line 917, in _bootstrap_inner`
`    self.run()`
`  File "C:\Users\Natii\Anaconda3\lib\threading.py", line 865, in run`
`    self._target(*self._args, **self._kwargs)`
`  File "C:\Users\Natii\Anaconda3\lib\site-packages\ipyparallel\client\client.py", line 901, in _io_main`
`    self._io_loop.start()`
`  File "C:\Users\Natii\Anaconda3\lib\site-packages\tornado\platform\asyncio.py", line 132, in start`
`    self.asyncio_loop.run_forever()`
`  File "C:\Users\Natii\Anaconda3\lib\asyncio\base_events.py", line 528, in run_forever`
`    self._run_once()`
`  File "C:\Users\Natii\Anaconda3\lib\asyncio\base_events.py", line 1728, in _run_once`
`    event_list = self._selector.select(timeout)`
`  File "C:\Users\Natii\Anaconda3\lib\selectors.py", line 323, in select`
`    r, w, _ = self._select(self._readers, self._writers, [], timeout)`
`  File "C:\Users\Natii\Anaconda3\lib\selectors.py", line 314, in _select`
`    r, w, x = select.select(r, w, w, timeout)`
`OSError: [WinError 10038] An operation was attempted on something that is not a socket`

So long story short, there is some error associated with the `client().shutdown()`, that happens both for `hub = True` or `hub = False`. The error in the console, happens only when `hub = True`. I don't know if this is why my original code is not finishing...

I am using Windows 10 64-bit, pyhton version 3.7.1.final.0 on conda version 4.6.12.
I would really appreciate any help you could give me!!!

Comments:

Patch:
diff --git a/docs/source/api/ipyparallel.rst b/docs/source/api/ipyparallel.rst
index cbab7042..ee7ac089 100644
--- a/docs/source/api/ipyparallel.rst
+++ b/docs/source/api/ipyparallel.rst
@@ -12,16 +12,14 @@ Classes
 -------
 
 .. autoclass:: Client
-    :members:
 
 .. autoclass:: DirectView
-    :members:
 
 .. autoclass:: LoadBalancedView
-    :members:
+
+.. autoclass:: BroadcastView
 
 .. autoclass:: ViewExecutor
-    :members:
 
 Decorators
 ----------
diff --git a/docs/source/conf.py b/docs/source/conf.py
index 8571badb..689c240e 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -60,6 +60,15 @@ extensions = [
 # You can specify multiple suffix as a list of string:
 source_suffix = ['.rst', '.md']
 
+from traitlets.config import HasTraits
+
+# exclude members inherited from HasTraits by default
+autodoc_default_options = {
+    'members': None,
+    "inherited-members": None,
+    "exclude-members": ','.join(dir(HasTraits)),
+}
+
 # Add dev disclaimer.
 if iprelease['version_info'][-1] == 'dev':
     rst_prolog = """
diff --git a/ipyparallel/client/asyncresult.py b/ipyparallel/client/asyncresult.py
index 41ece2c5..cc19f4ff 100644
--- a/ipyparallel/client/asyncresult.py
+++ b/ipyparallel/client/asyncresult.py
@@ -269,6 +269,8 @@ class AsyncResult(Future):
         return self._ready
 
     def _resolve_result(self, f=None):
+        if self.done():
+            return
         try:
             if f:
                 results = f.result()
diff --git a/ipyparallel/client/client.py b/ipyparallel/client/client.py
index 5b688e16..01ec4501 100644
--- a/ipyparallel/client/client.py
+++ b/ipyparallel/client/client.py
@@ -1103,6 +1103,7 @@ class Client(HasTraits):
         track=False,
         header=None,
         metadata=None,
+        track_outstanding=False,
     ):
         """Send a message in the IO thread
 
@@ -1114,6 +1115,22 @@ class Client(HasTraits):
         )
         msg_id = msg['header']['msg_id']
 
+        if track_outstanding:
+            # add to outstanding, history
+            self.outstanding.add(msg_id)
+            self.history.append(msg_id)
+
+            if ident:
+                # possibly routed to a specific engine
+                ident_str = ident
+                if isinstance(ident_str, list):
+                    ident_str = ident_str[-1]
+                ident_str = ident_str.decode("utf-8")
+                if ident_str in self._engines.values():
+                    # save for later, in case of engine death
+                    self._outstanding_dict[ident_str].add(msg_id)
+            self.metadata['submitted'] = util.utcnow()
+
         futures = self.create_message_futures(
             msg_id,
             async_result=msg_type in {'execute_request', 'apply_request'},
@@ -1325,6 +1342,8 @@ class Client(HasTraits):
         if jobs is None:
             # get futures for results
             futures = [f for f in self._futures.values() if hasattr(f, 'output')]
+            if not futures:
+                return
             ar = AsyncResult(self, futures, owner=False)
         else:
             ar = self._asyncresult_from_jobs(jobs, owner=False)
@@ -1597,19 +1616,9 @@ class Client(HasTraits):
             ident=ident,
             metadata=metadata,
             track=track,
+            track_outstanding=True,
         )
-
         msg_id = future.msg_id
-        self.outstanding.add(msg_id)
-        if ident:
-            # possibly routed to a specific engine
-            if isinstance(ident, list):
-                ident = ident[-1]
-            ident = ident.decode("utf-8")
-            if ident in self._engines.values():
-                # save for later, in case of engine death
-                self._outstanding_dict[ident].add(msg_id)
-        self.history.append(msg_id)
 
         return future
 
@@ -1635,22 +1644,14 @@ class Client(HasTraits):
         content = dict(code=code, silent=bool(silent), user_expressions={})
 
         future = self._send(
-            socket, "execute_request", content=content, ident=ident, metadata=metadata
+            socket,
+            "execute_request",
+            content=content,
+            ident=ident,
+            metadata=metadata,
+            track_outstanding=True,
         )
 
-        msg_id = future.msg_id
-        self.outstanding.add(msg_id)
-        if ident:
-            # possibly routed to a specific engine
-            if isinstance(ident, list):
-                ident = ident[-1]
-            ident = ident.decode("utf-8")
-            if ident in self._engines.values():
-                # save for later, in case of engine death
-                self._outstanding_dict[ident].add(msg_id)
-        self.history.append(msg_id)
-        self.metadata[msg_id]['submitted'] = util.utcnow()
-
         return future
 
     # --------------------------------------------------------------------------
diff --git a/ipyparallel/controller/hub.py b/ipyparallel/controller/hub.py
index 526bdbd7..6b1146b1 100644
--- a/ipyparallel/controller/hub.py
+++ b/ipyparallel/controller/hub.py
@@ -277,14 +277,14 @@ class HubFactory(RegistrationFactory):
         'DictDB',
         config=True,
         help="""The class to use for the DB backend
-        
+
         Options include:
-        
+
         SQLiteDB: SQLite
         MongoDB : use MongoDB
         DictDB  : in-memory storage (fastest, but be mindful of memory growth of the Hub)
         NoDB    : disable database altogether (default)
-        
+
         """,
     )
 
@@ -1100,7 +1100,11 @@ class Hub(SessionFactory):
         msg_type = msg['header']['msg_type']
         if msg_type == 'shutdown_reply':
             session = msg['header']['session']
-            eid = self.by_ident.get(session, None)
+            uuid_bytes = session.encode("utf8", "replace")
+            eid = self.by_ident.get(uuid_bytes, None)
+            if eid is None:
+                self.log.error(f"Found no engine for {session}")
+                return
             uuid = self.engines[eid].uuid
             self.unregister_engine(
                 ident='shutdown_reply', msg=dict(content=dict(id=eid, queue=uuid))
diff --git a/ipyparallel/serialize/canning.py b/ipyparallel/serialize/canning.py
index f2ba4140..b83ef98e 100644
--- a/ipyparallel/serialize/canning.py
+++ b/ipyparallel/serialize/canning.py
@@ -193,6 +193,16 @@ class CannedFunction(CannedObject):
         else:
             self.defaults = None
 
+        if f.__kwdefaults__:
+            self.kwdefaults = can_dict(f.__kwdefaults__)
+        else:
+            self.kwdefaults = None
+
+        if f.__annotations__:
+            self.annotations = can_dict(f.__annotations__)
+        else:
+            self.annotations = None
+
         closure = py3compat.get_closure(f)
         if closure:
             self.closure = tuple(can(cell) for cell in closure)
@@ -218,11 +228,25 @@ class CannedFunction(CannedObject):
             defaults = tuple(uncan(cfd, g) for cfd in self.defaults)
         else:
             defaults = None
+
+        if self.kwdefaults:
+            kwdefaults = uncan_dict(self.kwdefaults)
+        else:
+            kwdefaults = None
+        if self.annotations:
+            annotations = uncan_dict(self.annotations)
+        else:
+            annotations = {}
+
         if self.closure:
             closure = tuple(uncan(cell, g) for cell in self.closure)
         else:
             closure = None
         newFunc = FunctionType(self.code, g, self.__name__, defaults, closure)
+        if kwdefaults:
+            newFunc.__kwdefaults__ = kwdefaults
+        if annotations:
+            newFunc.__annotations__ = annotations
         return newFunc
 
 


Test patch:
diff --git a/ipyparallel/tests/test_canning.py b/ipyparallel/tests/test_canning.py
index f462b343..090cb837 100644
--- a/ipyparallel/tests/test_canning.py
+++ b/ipyparallel/tests/test_canning.py
@@ -21,6 +21,10 @@ def loads(obj):
     return uncan(pickle.loads(obj))
 
 
+def roundtrip(obj):
+    return loads(dumps(obj))
+
+
 def test_no_closure():
     @interactive
     def foo():
@@ -109,3 +113,20 @@ def test_can_partial_buffers():
     loaded.buffers = buffers
     pfoo2 = uncan(loaded)
     assert pfoo2() == partial_foo()
+
+
+def test_keyword_only_arguments():
+    def compute(a, *, b=3):
+        return a * b
+
+    assert compute(2) == 6
+    compute_2 = roundtrip(compute)
+    assert compute_2(2) == 6
+
+
+def test_annotations():
+    def f(a: int):
+        return a
+
+    f2 = roundtrip(f)
+    assert f2.__annotations__ == f.__annotations__