Repository: tomerfiliba-org/rpyc

Issue #354: Race condition when sending messages from another thread
Created: 2019-10-18T15:53:02Z
We're seeing a race that occasionally causes a 30-second lag. The precise scenario:

Run a ThreadedServer
Connection arrives, thread A assigned to serve_all on that connection.
Thread B sends a message on the connection then calls serve() to get the reply.
Thread B waits for a recv_event while thread A is in poll().
Thread A gets the reply.
Thread A posts the recv_event and gives up the GIL while dispatching the reply, before setting _ready.
Thread B wakes up, enters the poll() and gives up the GIL.
Thread A finishes dispatching the reply and sets _ready on the AsyncResult.
30 seconds later, thread B's poll times out
Thread B finds the reply is ready and all is well.

So there's no data loss, no unexpected exceptions, just 30 seconds wasted, once in a while.
Workaround: have the client ping every second, so that the waste is at most 1 second (because in step 9, thread B's poll doesn't time out it just handles the ping).
We haven't minimized the example yet.
Environment

rpyc version 4.0.2
python version 2.7.x
operating system Windows, mac, linux

This issue is presumably platform-independent.

Duplicate issue #449: sporadic deadlock in syncreq
Created: 2021-07-16T08:24:55Z
We are sporadically having an issue stucking in syncreq from variaus calls.
However when the timeout expires, execution continues normaly without any exception.
It can be reproduced when:

running server and client at the same CPU (using taskset command)
creating some background-serving-threads at the client
adding a delay in the _dispatch function
(formerly added some logging here, but a delay with 0 seconds triggered the issue as well)

$ taskset -c 0 python3 client.py 
^CTraceback (most recent call last):
  File "client.py", line 15, in <module>
    conn.execute("import time")
  File "/home/wasc/.local/lib/python3.6/site-packages/rpyc/core/netref.py", line 240, in __call__
    return syncreq(_self, consts.HANDLE_CALL, args, kwargs)
  File "/home/wasc/.local/lib/python3.6/site-packages/rpyc/core/netref.py", line 63, in syncreq
    return conn.sync_request(handler, proxy, *args)
  File "/home/wasc/.local/lib/python3.6/site-packages/rpyc/core/protocol.py", line 474, in sync_request
    return self.async_request(handler, *args, timeout=timeout).value
  File "/home/wasc/.local/lib/python3.6/site-packages/rpyc/core/async_.py", line 100, in value
    self.wait()
  File "/home/wasc/.local/lib/python3.6/site-packages/rpyc/core/async_.py", line 47, in wait
    self._conn.serve(self._ttl)
  File "/home/wasc/.local/lib/python3.6/site-packages/rpyc/core/protocol.py", line 387, in serve
    data = self._channel.poll(timeout) and self._channel.recv()
  File "/home/wasc/.local/lib/python3.6/site-packages/rpyc/core/channel.py", line 47, in poll
    return self.stream.poll(timeout)
  File "/home/wasc/.local/lib/python3.6/site-packages/rpyc/core/stream.py", line 47, in poll
    rl = p.poll(timeout.timeleft())
  File "/home/wasc/.local/lib/python3.6/site-packages/rpyc/lib/compat.py", line 122, in poll
    events = self._poll.poll(timeout)
KeyboardInterrupt

Environment

rpyc 5.0.1 (same with 4.0.2)
python 3.6
operating Ubuntu 18.04 (bionic)

Minimal example
Patch
diff --git a/rpyc/core/protocol.py b/rpyc/core/protocol.py
index baef019..977316b 100644
--- a/rpyc/core/protocol.py
+++ b/rpyc/core/protocol.py
@@ -358,6 +358,7 @@ class Connection(object):
 
     def _dispatch(self, data):  # serving---dispatch?
         msg, seq, args = brine.load(data)
+        time.sleep(0.0)
         if msg == consts.MSG_REQUEST:
             self._dispatch_request(seq, args)
         elif msg == consts.MSG_REPLY:
Server:
taskset -c 0 rpyc_classic.py
Client:
import time
import rpyc

conn = rpyc.classic.connect("localhost")

bg_threads = []

for i in range(3):
    bg_threads.append(rpyc.BgServingThread(conn))

if __name__ == "__main__":
    t0 = time.time()
    conn.execute("import time")
    print(time.time() - t0)
Run Client:
taskset -c 0 python3 client.py

Closing comment (2021-07-16T08:27:49Z): My be same as / related to #354