Repository: ros/ros_comm

Issue:
rospy: high latency when using asynchronous publishing with python 2
When using asynchronous mode in python 2, publishing incurs high latency. This is due to the way python 2 does threading-related timeouts. If you do wait() without timeout it's fine. But if you do use a timeout it will do some silly things, which you can see [here](https://github.com/python/cpython/blob/9d93b00adb12ffd30f451c0f272a877ae4c3aa4f/Lib/threading.py#L344), resulting in high latency between the time notify() is called and the time wait() returns. This happens in [clients/rospy/src/rospy/impl/tcpros_pubsub.py](https://github.com/ros/ros_comm/blob/1.11.10/clients/rospy/src/rospy/impl/tcpros_pubsub.py):

``` python
self._cond_data_available.wait(1.0)
```

Here is an example that demonstrates the problem:
publisher:

``` python
#!/usr/bin/env python
import rospy
from std_msgs.msg import Time

def talker():
    pub = rospy.Publisher('latency', Time, queue_size=10)
    rospy.init_node('talker', anonymous=True)
    rate = rospy.Rate(10)
    while not rospy.is_shutdown():
        pub.publish(rospy.Time.now())
        rate.sleep()

if __name__ == '__main__':
    try:
        talker()
    except rospy.ROSInterruptException:
        pass
```

subscriber:

``` python
#!/usr/bin/env python
import rospy
from std_msgs.msg import Time

def callback(data):
    diff = rospy.Time.now()-data.data
    rospy.loginfo("diff = %.3f ms", diff.to_nsec()/1000000.0)

def listener():
    rospy.init_node('listener', anonymous=True)
    rospy.Subscriber("latency", Time, callback)
    rospy.spin()

if __name__ == '__main__':
    listener()
```

Results with python 2:

```
[INFO] [WallTime: 1421259423.134991] diff = 20.410 ms
[INFO] [WallTime: 1421259423.248932] diff = 34.405 ms
[INFO] [WallTime: 1421259423.363028] diff = 48.533 ms
[INFO] [WallTime: 1421259423.426917] diff = 12.439 ms
[INFO] [WallTime: 1421259423.541052] diff = 26.467 ms
[INFO] [WallTime: 1421259423.655113] diff = 40.537 ms
[INFO] [WallTime: 1421259423.719151] diff = 4.578 ms
[INFO] [WallTime: 1421259423.833174] diff = 18.659 ms
...
```

If we change the wait to not use a timeout, i.e. 

``` python
self._cond_data_available.wait()
```

the results are much better:

```
[INFO] [WallTime: 1421259403.810390] diff = 0.468 ms
[INFO] [WallTime: 1421259403.910442] diff = 0.451 ms
[INFO] [WallTime: 1421259404.010454] diff = 0.527 ms
[INFO] [WallTime: 1421259404.110433] diff = 0.478 ms
[INFO] [WallTime: 1421259404.210471] diff = 0.526 ms
[INFO] [WallTime: 1421259404.310507] diff = 0.510 ms
[INFO] [WallTime: 1421259404.410385] diff = 0.436 ms
[INFO] [WallTime: 1421259404.510414] diff = 0.472 ms
...
```

One solution would be to call wait() without a timeout and have the QueuedConnection register a connection closed callback that calls notify() so that wait() will return when the connection is closed.



Discussion:
Thanks for the detailed report and steps to reproduce the problem. The explanation and propose solution sound solid, and I can confirm the described behavior on my machine using Indigo.

However, a change like this has to be carefully architected and reviewed since it is at the core of so many things. We'll have a look at this as time allows, but it may be a while before we can address this. Also, things typically go faster if you can provide a pull request, even if you are unsure if the approach is correct, it helps to have something we can iterate on.


Resolving patch:
diff --git a/clients/rospy/src/rospy/impl/tcpros_pubsub.py b/clients/rospy/src/rospy/impl/tcpros_pubsub.py
--- a/clients/rospy/src/rospy/impl/tcpros_pubsub.py
+++ b/clients/rospy/src/rospy/impl/tcpros_pubsub.py
@@ -382,12 +382,16 @@ def __init__(self, connection, queue_size):
 
         self._lock = threading.Lock()
         self._cond_data_available = threading.Condition(self._lock)
+        self._connection.set_cleanup_callback(self._closed_connection_callback)
         self._queue = []
         self._error = None
 
         self._thread = threading.Thread(target=self._run)
         self._thread.start()
 
+    def _closed_connection_callback(self, connection):
+        self._cond_data_available.notify()
+
     def __getattr__(self, name):
         if name.startswith('__'):
             raise AttributeError(name)
@@ -415,7 +419,7 @@ def _run(self):
             with self._lock:
                 # wait for available data
                 while not self._queue and not self._connection.done:
-                    self._cond_data_available.wait(1.0)
+                    self._cond_data_available.wait()
                 # take all data from queue for processing outside of the lock
                 if self._queue:
                     queue = self._queue


Test patch: