Repository: ray-project/ray

Issue:
[autoscaler] bash 4.4 does not support set -i
<!--
General questions should be asked on the mailing list ray-dev@googlegroups.com.

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
Ubuntu 17.10

- **Ray installed from (source or binary)**:
Source
- **Ray version**:
commit b8811cbe3418ab0d3ea10deaa54947d5bb26cecf

- **Python version**:
3.6
- **Exact command to reproduce**:
ray create_or_update example.yaml

<!--
You can obtain the Ray version with

python -c "import ray; print(ray.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

As of bash 4.4, set -i is no longer accepted to create an interactive shell. Consider -t in stead.

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:
Tried using -t, this creates problems down the line. Will try to come up with a better suggestion...
OTOH -- why try to force interactive shells anyway?
Thanks @atypic, I believe the motivation was so that the sequence of commands, e.g.,

https://github.com/ray-project/ray/blob/215d526e0d605e2f090da3c7b1ec66c990bec89c/python/ray/autoscaler/aws/development-example.yaml#L69-L80

would share "state" with each other. E.g., modifying the `PATH` in one command carries over to the next.
@robertnishihara the commands are actually still separate sessions. The original motivation was so that the setup commands can run with a similar PATH as an interactive shell (otherwise, you'd have to prepend all your commands with things like `PATH=~/anaconda3/bin:$PATH`).

You might think that just auto-prepending `source ~/.bashrc` to each command would be sufficient, but it turns out many bashrc scripts early exit if the session is non-interactive. Hence the `set -i`.

One solution might just be to remove the `set -i`, this is not as nice from the user perspective but avoids weird pitfalls.
This stackoverflow post has a better explanation: https://stackoverflow.com/questions/940533/how-do-i-set-path-such-that-ssh-userhost-command-works
I tried to remove set -i and modify the yaml accordingly, however something odd happens to the startup of the redis server, making me unable to connect to the socket. The server is running and all seems fine, but ray.init() fails ("Unable to connect to socket /tmp/<id>"...). I do something similar to 

`source /home/ubuntu/anaconda3/bin/activate MyEnvironment; ray start --head <...>
`
in the head_start_ray_commands. I had to go back to bash 4.3 to get set -i. 

ray start perhaps depends on something in Env/.bashrc? I have yet to investigate :-) 

Anyway...
Somehow depending on the behavior of bash seems wonky, as people like to use a variety of shells even on AWS.

Another solution could perhaps be to have a "PATH" section in the yaml, and use .ssh/environment.

Node provisioning is always such a mess :( 

Resolving patch:
diff --git a/python/ray/autoscaler/updater.py b/python/ray/autoscaler/updater.py
--- a/python/ray/autoscaler/updater.py
+++ b/python/ray/autoscaler/updater.py
@@ -271,9 +271,10 @@ def ssh_cmd(self,
             ssh.append("-tt")
         if emulate_interactive:
             force_interactive = (
-                "set -i || true && source ~/.bashrc && "
+                "true && source ~/.bashrc && "
                 "export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && ")
-            cmd = "bash --login -c {}".format(quote(force_interactive + cmd))
+            cmd = "bash --login -c -i {}".format(
+                quote(force_interactive + cmd))
 
         if port_forward is None:
             ssh_opt = []


Test patch: