Repository: ros/ros_comm

Issue:
Race condition between `roscore` and `roslaunch --wait`
**Steps to reproduce**

```
echo "<launch></launch>" > launch.xml
roscore & roslaunch --wait launch.xml
```
Rarely, both roscore and roslaunch will start the rosout node. If this happens, the rosout nodes will conflict forever as they are both started with `respawn="true"`.

To make this more likely to happen, you can do the following:

```
edit _setup() in roslaunch/launch.py:
        self._launch_master()
        import random
        time.sleep(random.random())
        self._launch_core_nodes() 
edit _wait_for_master() in roslaunch/rlutil.py:
    while not is_running:
        # time.sleep(0.1)
        is_running = m.is_running()
run this script and look for multiple "started core service" lines:
    #!/bin/bash
    export PYTHONUNBUFFERED=yes
    roscore | sed -n "/core service/ {s/^/roscore: /; p}" &
    for i in $(seq 1 100); do
      roslaunch --wait launch.xml | sed -n "/core service/ {s/^/roslaunch $i: /; p}" &
    done
    sleep 5
    pkill -P $$
    wait
```

**Explanation**

If you run `roslaunch --wait`, it will wait for the master before starting, but the `--wait` flag has no effect on the starting of core nodes. In roslaunch/launch.py, we have:

```
        # start up the core: master + core nodes defined in core.xml
        self._launch_master()
        self._launch_core_nodes() 
```

unconditionally of whether `--core` or `--wait` is used. In the `--wait` case, `rlutil._wait_for_master` has already been called, and `_launch_master()` notices that the master is already running so does nothing. However, `_launch_core_nodes` may still try to launch `rosout` if the roscore hasn't started it yet.

**Workaround**

Run this command before `roslaunch --wait`:

```
until rosnode info rosout | grep Pid; do sleep 1; done
```


Discussion:
I guess it is an uncommon use case to start `roscore` as well as `roslaunch` in parallel. Currently that will have the race you described. The right solution for this would be to add an option `--no-core` to `roslaunch` to never attempt to start the core on its own and always rely on it being started externally.

Beside that I am not sure why you want to invoke `roscore` and `roslaunch` in parallel in the first plac. You could just invoke `roslaunch` and it will start the core automatically. Usually the core is only started separately in order to be able to stop / restart the launch file without restarting the core. But since you invoke these commands together that doesn't seem to be your use case. Maybe describing your use case will help to understand why you are trying to do it.

Anyway regarding the mentioned new option it would be great if you could provide a PR implementing it if that is of use for your application.
Thanks for the suggestion!

We have a containerized setup running on Kubernetes. When you create a deployment, it starts a `roscore` container along with containers for any ROS tools that are desired (eg nav, motion planning, network bridging). Some of these ROS tools are just single nodes, so there's no risk, but others use `roslaunch` to bring up multiple nodes, in which case we see this problem.

Does anybody start the `rosmaster` without starting the core nodes? It seems like `roscore` is the only supported way of starting a master - f that's the case, we could just make `--wait` skip the core services, since whatever started the master will start them. This would be simpler and less error prone, since most users wouldn't find out about `--no-core`. What do you think?
The `--wait` option has a different semantic at the moment. It still starts the core but then wait until the core is up and available before starting any node. Existing code will rely on that behavior. So changing the semantic of the existing option would likely break those.

Therefore adding a new option for this is better. Both options could even make sense to be used together or not. E.g. using `--no-core` without `--wait` if you want to start the nodes even though the core might not be ready yet. Or using both options to wait for the external core to be available.
> The --wait option has a different semantic at the moment. It still starts the core but then wait until the core is up and available before starting any node

I don't quite understand this. As I understand, `--wait` means the first thing done is to wait for a master to come up, ie it doesn't wait for the core nodes/params:

        if options.wait_for_master:
            if options.core:
                parser.error("--wait cannot be used with roscore")
            rlutil._wait_for_master()            

From then on, the option is largely ignored, and `_launch_master()` and `_launch_core_nodes()` are both called, even though `_launch_master()` has no effect (since we already waited for an existing master) and `_launch_core_nodes()` introduces the race condition described (but normally has no effect, since `roscore` has already launched the core nodes).

If the `--wait` option waits for the master as before, but doesn't call `_launch_core_nodes()`, code will not break (except for the case that they are running `rosmaster` directly without running `roscore`, and assuming that `roslaunch --wait` will start core nodes without starting a master, which seems unlikely).

I share your caution about changing the behavior of the existing code, but the current behaviour of `--wait` is broken in the case of a simultaneous launch, and leaving it broken by default seems the bigger risk.

> E.g. using --no-core without --wait if you want to start the nodes even though the core might not be ready yet.

This would not be reliable - `roslaunch` can start nodes before the master is running, but it can't set params.



Resolving patch:
diff --git a/tools/roslaunch/src/roslaunch/launch.py b/tools/roslaunch/src/roslaunch/launch.py
--- a/tools/roslaunch/src/roslaunch/launch.py
+++ b/tools/roslaunch/src/roslaunch/launch.py
@@ -382,6 +382,9 @@ def _launch_nodes(self):
     def _launch_master(self):
         """
         Launches master if requested. 
+        @return: True if a master was launched, False if a master was
+        already running.
+        @rtype: bool
         @raise RLException: if master launch fails
         """
         m = self.config.master
@@ -426,6 +429,8 @@ def _launch_master(self):
             self.logger.info("setting /roslaunch/uris/%s__%s' to %s"%(hostname, port, self.server_uri))
             param_server.setParam(_ID, '/roslaunch/uris/%s__%s'%(hostname, port),self.server_uri)
 
+        return not is_running
+
     def _check_and_set_run_id(self, param_server, run_id):
         """
         Initialize self.run_id to existing value or setup parameter
@@ -627,8 +632,9 @@ def _setup(self):
             self.remote_runner.add_process_listener(self.listeners)            
 
         # start up the core: master + core nodes defined in core.xml
-        self._launch_master()
-        self._launch_core_nodes()        
+        launched = self._launch_master()
+        if launched:
+            self._launch_core_nodes()
         
         # run exectuables marked as setup period. this will block
         # until these executables exit. setup executable have to run


Test patch: