Repository: ros/ros_comm

Issue:
[bug] python rosnode_ping failure after node restart
Problem:
I have a python node A running on a Ubuntu16 machine which every 10 seconds pings another node B (a service server) to see if it's still alive. B is running on an Android tablet using rosjava. Whenever the tablet loses network connection, A obviously fails to ping B
Then, I programmed A to reinitialize itself and wait for the service server to come online again. Once that happens, A successfully connects to B again but then pinging B out of that python script always fails ("Unknown network error contacting node: timed out").
This makes A to reinitialize itself again and the procedure above restarts. B is available though because otherwise A would not be able to successfully connect to the service server. 
The only work-around to avoid this weird behaviour right now is to restart A.

If I make use of rosnode ping B it also works perfectly.
I had a look at [this](https://github.com/ros/ros_comm/issues/221) issue but it only refers to a situation where a node is stopped properly by the user, not because of network errors.




Discussion:
The description is way to abstract to act on it. Please either share a reproducible example which points to a bug in code of this repository or ask you question on answers.ros.org instead.
Hi,
sorry for the confusion. I'll try explain it more detailed now. I put the issue on github because I think it is related to the issue mentioned in the initial post.

I have a a service client running on a Ubuntu16 machine and a service server running on an Android tablet, using rosjava. For my setup, I create an own local network / Hotspot, which is created by the Ubuntu machine. FYI, everything is working fine until the tablet loses the connection to that Hotspot.

If server and client are running properly and e.g. I kill the app, 'check_for_service_availability' of the python node on the Ubuntu machine obviously fails to ping the server and displays an Error (without crashing) "Error: connection refused to <node>" (rosnode/__init__.py line 356) in the terminal window. Then, the client reinitializes itself, I restart the app and the client reconnects successfully to the "new" server and thus, 'check_for_service_availability' always successfully pings the service server until the server is shutdown at some point. That's all fine and works as expected.

Sometimes the tablet loses the network connection because of too great distance. This is very likely to happen in my use case. The app then returns to a wifi list where the user can choose what network he wants to join. Well, once the user is _again_ in range of the Ubuntu machine's Hotspot, this Hotspot is displayed in the app and the user can join the network _again_. After doing so, all the ros nodes on the app will be initialized again (since the old ones were shutdown because of the network connection loss), including that service server. Now, once the service server is up and running, the service client on Ubuntu side successfully connects to the service server _again_, too. That is, 'wait_for_service' succeeds and 'check_for_service_availability' starts. Until here, everything is still working as expected.

Now, finally what I believe is a bug is the following behaviour:
'rosnode_ping' inside 'check_for_service_availability' then always fails to ping the service server. Because of my code, this leads to a reinitialization of the service client, which then immediately successfully connects to the service server but then for some reason fails to ping it. Then it reinitializes, it connects, it fails pinging, it reinitializes, it connects, it fails pinging and so on.

It looks like rosnode_ping tries to ping the old service server, but still connects to new one successfully. But even if I type in rosnode cleanup to erase the old service server from ros master, the same behaviour occurs. 
What's wrong here?

This is the client's code:


```
class Client:

    def __init__(self):
        self.client_service = None
        self.path = rospy.get_param("~data_directory", "/default/path/")
        self.namespace = rospy.get_namespace()
        self.init_node()

    def init_node(self):
        print('Waiting for app\'s server to come online ...')
        rospy.wait_for_service('/app/server')
        self.client_service = rospy.ServiceProxy('/app/server', MessageType)
        print('Successfully connected to app\'s  server!')
        # After successful connection, I want the client to send the content of a directory on the Ubuntu machine to the server on the app
        self.send_directory_content()
        threading.Thread(target=self.check_for_service_availability).start()

    # This function repeatedly checks if the app's server is still running.
    # If this is not the case this service client reinitializes itself.
    def check_for_service_availability(self):
        while True:
            print('Checking service availability ...')
            // Here, rosnode_ping starts failing after the tablet, which runs the service server, has left and then rejoined the wifi network
            if not rosnode.rosnode_ping('/app/server', 1, False):
                print('Service not available anymore. Reconnecting ...')
                break
            time.sleep(10)
        self.init_node()

    def send_directory_content(self):
        print('Sending directory content')
        # Store files in a list
        files = [f for f in listdir(self.path) if isfile(join(self.path, f))]
        try:
            # Make a call to the service server on the app
            self.client_service(self.namespace, files)

        except rospy.ServiceException:
            print('Service not available anymore')

if __name__ == '__main__':
    rospy.init_node('client')
    service_client = Client()
```


Thanks, Max
I found the source of the problem. 
It is located in 'rosnode/init.py' in function 'get_api_uri' in line 308 (ROS Kinetic).
rosnode_ping usually tries to get a node uri out of a cache before it makes a request to ros master. Ros master must have kept the old uri in this cache even though I used rosnode cleanup. That's why rosnode_ping always tried to ping the "old" service server.
By setting 'skip_cache' parameter of 'get_api_uri' to True, rosnode_ping doesn't make use of this cache and immediately makes a request to ros master. That fixed my problem. 

Maybe it would be good to give the programmer the possibility to skip looking into the cache because otherwise one would have to make use of 'get_api_uri' directly and evaluate its return value. What do you think?
Exposing the `skip_cache` argument of `get_api_uri()` in the `rosnode_ping()` functions sound like a straight forward way to allow the caller to make that decision.

Please consider to contribute a pull request targeting the default branch (`noetic-devel`) to add this argument.
That sounds like an idea. I'd do that if my thoughts below are wrong.

I had some more thoughts about this and even though the above mentioned solution would work, wouldn't it be rather a work-around for another problem?
What I mean is, that cache should have no reference to the old service server uri - at least after rosnode cleanup was called right? I believe, if this was fixed the underlying problem would be solved.
Or is there function which can clear the cache? I couldn't find such one. If yes, one could simply clear the cache every time a node has died and then make use of the cache again.

Otherwise, setting `skip_cache` to True is a little waste of resources as long as no nodes have died.
The is currently no public API to reset the cache.

It is basically stored in a single variable. Due to the nature of Python you are able to clear that dict anyway: https://github.com/ros/ros_comm/blob/7ee09ea5cc215733014f47394e6c55ccff69d814/tools/rosnode/src/rosnode/__init__.py#L87
Okay, you're right. That in combination with exposed `skip_cache` should do the job. PR is coming in the next days.

Resolving patch:
diff --git a/tools/rosnode/src/rosnode/__init__.py b/tools/rosnode/src/rosnode/__init__.py
--- a/tools/rosnode/src/rosnode/__init__.py
+++ b/tools/rosnode/src/rosnode/__init__.py
@@ -406,7 +406,7 @@ def rosnode_ping_all(verbose=False):
     
 def cleanup_master_blacklist(master, blacklist):
     """
-    Remove registrations from ROS Master that match blacklist.    
+    Remove registrations from ROS Master and node cache (_caller_apis) that match blacklist.    
     @param master: rosgraph Master instance
     @type  master: rosgraph.Master
     @param blacklist: list of nodes to scrub
@@ -429,6 +429,7 @@ def cleanup_master_blacklist(master, blacklist):
                 service_api = master.lookupService(s)
                 master_n = rosgraph.Master(n)
                 master_n.unregisterService(s, service_api)
+        _caller_apis.pop(n, None)
 
 def cleanup_master_whitelist(master, whitelist):
     """


Test patch: