Repository: pymeasure/pymeasure

Issue:
ZMQ error
Hi,

While running PyMeasure on a newer version of anaconda, I got an error which looks like it's coming from an update ZMQ had. In particular, the error occurs when a `worker` tries to use its `emit` method. The error is in the `self.publisher.send_serialized` function, which I'm guessing has changed in a newer version of ZMQ or something. A fix for now is to add a `TypeError` to the following `except` statement, but some functionality (which it seems I don't use) is probably being lost along the way. The full error message is at the bottom of this post, and the versions of everything that I'm using are below.

ZMQ version: 17.1.2
Python version: 3.6.6
Conda version: 4.5.11

```
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\Joseph\Anaconda3\lib\site-packages\zmq\sugar\socket.py", line 427, in send_multipart
    memoryview(msg)
TypeError: memoryview: a bytes-like object is required, not 'int'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Joseph\Anaconda3\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Users\Joseph\Anaconda3\lib\site-packages\pymeasure-0.5.1-py3.6.egg\pymeasure\experiment\workers.py", line 161, in run
    self.update_status(Procedure.RUNNING)
  File "C:\Users\Joseph\Anaconda3\lib\site-packages\pymeasure-0.5.1-py3.6.egg\pymeasure\experiment\workers.py", line 115, in update_status
    self.emit('status', status)
  File "C:\Users\Joseph\Anaconda3\lib\site-packages\pymeasure-0.5.1-py3.6.egg\pymeasure\experiment\workers.py", line 95, in emit
    self.publisher.send_serialized((topic, record), serialize=cloudpickle.dumps)
  File "C:\Users\Joseph\Anaconda3\lib\site-packages\zmq\sugar\socket.py", line 512, in send_serialized
    return self.send_multipart(frames, flags=flags, copy=copy, **kwargs)
  File "C:\Users\Joseph\Anaconda3\lib\site-packages\zmq\sugar\socket.py", line 434, in send_multipart
    i, rmsg,
TypeError: Frame 0 (128) does not support the buffer interface.
```


Discussion:
Same problem here

zmq: 18.0.1
zeromq: 4.3.1
python: 3.7.3
pymeasure: 0.6.1

I've got the same problem here as wel, but i figured out that the problem only arises when you have cloudpickle installed (for reference: in my case this is cloudpickle 1.0.0)

To solve this issue (for the time being) you could either remove the cloudpickle package from your setup or prevent importing the cloudpickle module (though might not be a very neat solution), e.g. using:
```
import sys
sys.modules['cloudpickle'] = None
```

Both solutions seem to be working for me, but are of course not real long-term solutions.
@jmittelstaedt and @lawa42, thanks for reporting this issue. It sounds like there is an incompatibility with `cloudpickle` as @CasperSchippers brings up. I will take a look when I get the opportunity.
This could also be an issue with Python 3.7, since we don't have any test visibility yet.

@jmittelstaedt, can you put together a minimal example that displays the behavior, and we can add it to the unit tests.
I've looked into this issue and it seems that for some reason the send_serialized function in zmq wants the serialization function result not as a single frame, but as a list (or other iterable) of those (perhaps it has changed some time if it worked earlier?), which cloudpickle.dumps itself doesn't provide. An obvious solution would be to change line 95 in workers.py from

`
	self.publisher.send_serialized((topic, record), serialize=cloudpickle.dumps)
`
to
`
	self.publisher.send_serialized((topic, record), serialize=lambda x : [cloudpickle.dumps(x)])
`

Corresponding adjustments are then also needed for any receiving party. Currently I think the only places within the package where recv_serialized is used is in the Listener/QListener classes. So using this as an example, in /experiment/listeners.py line 81, instead of

`
	topic, record = self.subscriber.recv_serialized(deserialize=cloudpickle.loads, flags=flags)
`
one should do
`
	topic, record = self.subscriber.recv_serialized(deserialize=lambda x : cloudpickle.loads(x[0]), flags=flags)
`

(and there is a similar definition of QListener class in /display/listeners.py). 

Also, if I understand correctly then any topic filtering can only be done after deserialization, i.e. the first call to setsockopt (zmq.SUBSCRIBE,'') has to be with an empty string.

There may of course be some side effects I'm not aware of, but generally this way the zmq pub/sub seems to be working.
> I've got the same problem here as wel, but i figured out that the problem only arises when you have cloudpickle installed (for reference: in my case this is cloudpickle 1.0.0)
> 
> To solve this issue (for the time being) you could either remove the cloudpickle package from your setup or prevent importing the cloudpickle module (though might not be a very neat solution), e.g. using:
> 
> ```
> import sys
> sys.modules['cloudpickle'] = None
> ```
> 
> Both solutions seem to be working for me, but are of course not real long-term solutions.

works for me, thanks
@CasperSchippers can you check if you still get this error? There have been a number of cloudpickle releases in the meantime, maybe this got fixed and we just need to bump the minimum required version?
I just checked, but I do indeed get the error still with cloudpickle version 1.6.0 (i.e. latest version).
We probably shouldn't expect it to be fixed as this isn't a bug but a documented feature and indeed that of [zmq's send_serialized](https://pyzmq.readthedocs.io/en/latest/api/zmq.html#zmq.Socket.send_serialized) (and correspondingly recv_serialized):

serialize (callable) – The serialization function to use. serialize(msg) should return **an iterable** of sendable message frames (e.g. bytes objects), which will be passed to send_multipart.

As it stands, serialize (i.e. cloudpickle.dumps) only returns a frame, so it needs to be converted to an iterable, e.g. list. I actually use zmq communication and for me the solution I proposed earlier, works (also considering the remark about topic filtering). In fact I've been having to fix it every time I install or upgrade pymeasure somewhere. If anyone has a better idea, be my guest.
OK then; is anybody up to implement a test showing this behaviour and proposing a fix?
Some datapoints from #353:
* Can be triggered by running e.g. `gui.py`, `image_gui.py`
* It looks like when cloudpickle and pyzmq are both installed this error is thrown (both are brought in by Spyder).
* If just one or the other is installed the code runs error free.
@Kassikakk could you elaborate on this point?
> Also, if I understand correctly then any topic filtering can only be done after deserialization, i.e. the first call to setsockopt (zmq.SUBSCRIBE,'') has to be with an empty string.

What goes into the topic arg of the constructor is most probably a string anyway, so why does it have to be empty first, and what does that have to do with deserialisation? Curiously, I could not find a spot where [Q]Listener are used, but I am not familiar with this part of pymeasure.
OK, I started a fix in #354, but I am getting weird issues in the test suite. Maybe someone else can help, I am not really familiar with that part of the codebase. @CasperSchippers ?
In order for the built-in topic filtering to work, the topic as specified in `socket.setsockopt(zmq.SUBSCRIBE, topic)` must come at the beginning of the incoming dataframe, which however is at that point serialized, i.e. in the current implementation the result of `cloudpickle.dumps((topic,record))`. Now if you try to substitute a string for topic (and anything for record) and view the returned result of this function, you see that the topic string is there somewhere, but not at the beginning, meaning that it will not pass the topic filter. Hence one way to get the data is to pass an empty string to the filter (thus letting through all the messages) and later when you have the deserialized result (topic, record), check if the topic is of interest and correspondingly do something with (or skip) the record. A benefit of this method is that you can actually monitor multiple topics with one listener.

A possibly cleaner method that might work (haven't tested it myself) is suggested in this [SO article](https://stackoverflow.com/questions/24048319/setting-the-topic-when-using-the-pyobj-subfunctions-in-zeromq-python) (a bit of discussion is added [here](https://stackoverflow.com/questions/33159395/how-to-add-topic-filters-when-calling-recv-pyobj-in-zeromq)). Apparently, if a multipart message is received and the first part has (or perhaps begins with) the filter string, the message will pass through. So one can set the first part to be the topic and add the serialized record in a second part. If one still wants to use send_serialized/recv_serialized, it would be something like

```
#publisher (Worker) part
self.publisher.send_serialized(record, serialize= lambda x: [topic, cloudpickle.dumps(x)])

#subscriber (Listener) part
topic, record = self.subscriber.recv_serialized(deserialize=lambda x : [x[0], cloudpickle.loads(x[1])])

```
but as these functions call the send/recv_multipart anyway, it may make more sense to call them directly, as in the referred example (replace 'pickle' with 'cloudpickle' there).
> OK, I started a fix in #354, but I am getting weird issues in the test suite. Maybe someone else can help, I am not really familiar with that part of the codebase. @CasperSchippers ?

I've had a look. On my pc, the new `test_zmq_serialisation` seems to run fine itself, but for some reason the test then hangs at another test. It is however not consistent at which test it precisely hangs, and all of the tests that I've seen it hanging at also work perfectly fine if the `test_zmq_serialisation` is removed. (it always seems to be one of the tests in the `test_instrument.py` file, but that seems to me more related to the fact that this is the file that is ran directly after the `test_zmq_serialisation` test.)
At this moment, I couldn't find a way to solve this problem, but admittedly I'm also not very familiar with this part of the code.
Yeah, by now I suspect that we aren't cleaning up ZMQ resources correctly at the end, which is why the test hangs. It also happens on CI, the builds time out. I'll have to investigate that.

Resolving patch:
diff --git a/pymeasure/display/listeners.py b/pymeasure/display/listeners.py
--- a/pymeasure/display/listeners.py
+++ b/pymeasure/display/listeners.py
@@ -71,7 +71,10 @@ def __init__(self, port, topic='', timeout=0.01):
         self.timeout = timeout
 
     def receive(self, flags=0):
-        topic, record = self.subscriber.recv_serialized(deserialize=cloudpickle.loads, flags=flags)
+        topic, record = self.subscriber.recv_serialized(
+            deserialize=lambda msg: (msg[0].decode(), cloudpickle.loads(msg[1])),
+            flags=flags
+        )
         return topic, record
 
     def message_waiting(self):
diff --git a/pymeasure/experiment/listeners.py b/pymeasure/experiment/listeners.py
--- a/pymeasure/experiment/listeners.py
+++ b/pymeasure/experiment/listeners.py
@@ -68,8 +68,8 @@ def __init__(self, port, topic='', timeout=0.01):
         self.context = zmq.Context()
         log.debug("%s has ZMQ Context: %r" % (self.__class__.__name__, self.context))
         self.subscriber = self.context.socket(zmq.SUB)
-        self.subscriber.connect('tcp://localhost:%d' % port)
         self.subscriber.setsockopt(zmq.SUBSCRIBE, topic.encode())
+        self.subscriber.connect('tcp://localhost:%d' % port)
         log.info("%s connected to '%s' topic on tcp://localhost:%d" % (
             self.__class__.__name__, topic, port))
 
@@ -78,11 +78,15 @@ def __init__(self, port, topic='', timeout=0.01):
         self.timeout = timeout
 
     def receive(self, flags=0):
-        topic, record = self.subscriber.recv_serialized(deserialize=cloudpickle.loads, flags=flags)
+        topic, record = self.subscriber.recv_serialized(
+            deserialize=lambda msg: (msg[0].decode(), cloudpickle.loads(msg[1])),
+            flags=flags
+        )
         return topic, record
 
     def message_waiting(self):
-        return self.poller.poll(self.timeout)
+        """Check if we have a message, wait at most until timeout."""
+        return self.poller.poll(self.timeout * 1000)  # poll timeout is in ms
 
     def __repr__(self):
         return "<%s(port=%s,topic=%s,should_stop=%s)>" % (
diff --git a/pymeasure/experiment/workers.py b/pymeasure/experiment/workers.py
--- a/pymeasure/experiment/workers.py
+++ b/pymeasure/experiment/workers.py
@@ -76,8 +76,27 @@ def __init__(self, results, log_queue=None, log_level=logging.INFO, port=None):
         self.log_queue = log_queue
         self.log_level = log_level
 
+        global log
+        log = logging.getLogger()
+        log.setLevel(self.log_level)
+        # log.handlers = []  # Remove all other handlers
+        # log.addHandler(TopicQueueHandler(self.monitor_queue))
+        # log.addHandler(QueueHandler(self.log_queue))
+
         self.context = None
         self.publisher = None
+        if self.port is not None and zmq is not None:
+            try:
+                self.context = zmq.Context()
+                log.debug("Worker ZMQ Context: %r" % self.context)
+                self.publisher = self.context.socket(zmq.PUB)
+                self.publisher.bind('tcp://*:%d' % self.port)
+                log.info("Worker connected to tcp://*:%d" % self.port)
+                time.sleep(0.3)  # wait so that the socket will be ready before starting to emit messages
+            except Exception:
+                log.exception("Couldn't establish ZMQ publisher!")
+                self.context = None
+                self.publisher = None
 
     def join(self, timeout=0):
         try:
@@ -92,7 +111,10 @@ def emit(self, topic, record):
         log.debug("Emitting message: %s %s", topic, record)
 
         try:
-            self.publisher.send_serialized((topic, record), serialize=cloudpickle.dumps)
+            self.publisher.send_serialized(
+                record,
+                serialize=lambda rec: (topic.encode(), cloudpickle.dumps(rec)),
+            )
         except (NameError, AttributeError):
             pass  # No dumps defined
         if topic == 'results':
@@ -125,14 +147,14 @@ def shutdown(self):
 
         self.recorder.stop()
         self.monitor_queue.put(None)
+        if self.context is not None:
+            # Cleanly close down ZMQ context and associated socket
+            # For some reason, we need to close the socket before the
+            # context, otherwise context termination hangs.
+            self.publisher.close()
+            self.context.term()
 
     def run(self):
-        global log
-        log = logging.getLogger()
-        log.setLevel(self.log_level)
-        # log.handlers = []  # Remove all other handlers
-        # log.addHandler(TopicQueueHandler(self.monitor_queue))
-        # log.addHandler(QueueHandler(self.log_queue))
         log.info("Worker thread started")
 
         self.procedure = self.results.procedure
@@ -146,17 +168,6 @@ def run(self):
         self.procedure.should_stop = self.should_stop
         self.procedure.emit = self.emit
 
-        if self.port is not None and zmq is not None:
-            try:
-                self.context = zmq.Context()
-                log.debug("Worker ZMQ Context: %r" % self.context)
-                self.publisher = self.context.socket(zmq.PUB)
-                self.publisher.bind('tcp://*:%d' % self.port)
-                log.info("Worker connected to tcp://*:%d" % self.port)
-                time.sleep(0.01)
-            except Exception:
-                log.exception("couldn't connect to ZMQ context")
-
         log.info("Worker started running an instance of %r", self.procedure.__class__.__name__)
         self.update_status(Procedure.RUNNING)
         self.emit('progress', 0.)


Test patch:
diff --git a/tests/experiment/test_workers.py b/tests/experiment/test_workers.py
--- a/tests/experiment/test_workers.py
+++ b/tests/experiment/test_workers.py
@@ -21,6 +21,8 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 # THE SOFTWARE.
 #
+import importlib
+import logging
 
 import pytest
 import os
@@ -28,9 +30,13 @@
 from time import sleep
 from importlib.machinery import SourceFileLoader
 
+from pymeasure.experiment import Listener, Procedure
 from pymeasure.experiment.workers import Worker
 from pymeasure.experiment.results import Results
 
+tcp_libs_available = bool(importlib.util.find_spec('cloudpickle')
+                          and importlib.util.find_spec('zmq'))
+
 # Load the procedure, without it being in a module
 data_path = os.path.join(os.path.dirname(__file__), 'data/procedure_for_testing.py')
 RandomProcedure = SourceFileLoader('procedure', data_path).load_module().RandomProcedure
@@ -82,3 +88,55 @@ def test_worker_closes_file_after_finishing():
 
     # Test if the file has been properly closed by removing the file
     os.remove(file)
+
+
+@pytest.mark.skipif(not tcp_libs_available,
+                    reason='TCP communication packages not installed')
+def test_zmq_does_not_crash_worker(caplog):
+    """Check that a ZMQ serialisation usage error does not cause a crash.
+
+    See https://github.com/ralph-group/pymeasure/issues/168
+    """
+    procedure = RandomProcedure()
+    file = tempfile.mktemp()
+    results = Results(procedure, file)
+    # If we define a port here we get ZMQ communication
+    # if cloudpickle is installed
+    worker = Worker(results, port=5888, log_level=logging.DEBUG)
+    worker.start()
+    worker.join(timeout=4.0)  # give it enough time to finish the procedure
+    assert procedure.status == procedure.FINISHED
+    del worker  # make sure to clean up, reduce the possibility of test
+    # dependencies via left-over sockets
+
+
+@pytest.mark.skipif(not tcp_libs_available,
+                    reason='TCP communication packages not installed')
+def test_zmq_topic_filtering_works(caplog):
+
+    class ThreeEmitsProcedure(Procedure):
+        def execute(self):
+            self.emit('results', 'Data 1')
+            self.emit('progress', 33)
+            self.emit('results', 'Data 2')
+            self.emit('progress', 66)
+            self.emit('results', 'Data 3')
+            self.emit('progress', 99)
+
+    procedure = ThreeEmitsProcedure()
+    file = tempfile.mktemp()
+    results = Results(procedure, file)
+    received = []
+    worker = Worker(results, port=5888, log_level=logging.DEBUG)
+    listener = Listener(port=5888, topic='results', timeout=0.1)
+    sleep(0.5)  # leave time for subscriber and publisher to establish a connection
+    worker.start()
+    while True:
+        if not listener.message_waiting():
+            break
+        topic, record = listener.receive()
+        received.append((topic, record))
+    worker.join(timeout=4.0)  # give it enough time to finish the procedure
+    assert procedure.status == procedure.FINISHED
+    assert len(received) == 3
+    assert all([item[0] == 'results' for item in received])