Repository: ray-project/ray

Issue:
Docker + AWS fails with no such container error
Latest dev version of ray

```
(vanilla_ray_venv) richard@richard-desktop:~/improbable/vanillas/ray/python/ray/autoscaler/aws$ ray up aws_gpu_dummy.yaml 
2020-08-12 20:12:39,383	INFO config.py:268 -- _configure_iam_role: Role not specified for head node, using arn:aws:iam::179622923911:instance-profile/ray-autoscaler-v1
2020-08-12 20:12:39,612	INFO config.py:346 -- _configure_key_pair: KeyName not specified for nodes, using ray-autoscaler_us-east-1
2020-08-12 20:12:39,745	INFO config.py:407 -- _configure_subnet: SubnetIds not specified for head node, using [('subnet-f737f791', 'us-east-1a')]
2020-08-12 20:12:39,746	INFO config.py:417 -- _configure_subnet: SubnetId not specified for workers, using [('subnet-f737f791', 'us-east-1a')]
2020-08-12 20:12:40,358	INFO config.py:590 -- _create_security_group: Created new security group ray-autoscaler-richard_cluster_gpu_dummy (sg-0061ca6aff182c1bf)
2020-08-12 20:12:40,739	INFO config.py:444 -- _configure_security_group: SecurityGroupIds not specified for head node, using ray-autoscaler-richard_cluster_gpu_dummy (sg-0061ca6aff182c1bf)
2020-08-12 20:12:40,739	INFO config.py:454 -- _configure_security_group: SecurityGroupIds not specified for workers, using ray-autoscaler-richard_cluster_gpu_dummy (sg-0061ca6aff182c1bf)
This will create a new cluster [y/N]: y
2020-08-12 20:12:42,619	INFO commands.py:531 -- get_or_create_head_node: Launching new head node...
2020-08-12 20:12:42,620	INFO node_provider.py:326 -- NodeProvider: calling create_instances with subnet-f737f791 (count=1).
2020-08-12 20:12:44,032	INFO node_provider.py:354 -- NodeProvider: Created instance [id=i-0729c7a86355d5ff8, name=pending, info=pending]
2020-08-12 20:12:44,223	INFO commands.py:570 -- get_or_create_head_node: Updating files on head node...
2020-08-12 20:12:44,320	INFO command_runner.py:331 -- NodeUpdater: i-0729c7a86355d5ff8: Waiting for IP...
2020-08-12 20:12:54,409	INFO command_runner.py:331 -- NodeUpdater: i-0729c7a86355d5ff8: Waiting for IP...
2020-08-12 20:12:54,534	INFO log_timer.py:27 -- NodeUpdater: i-0729c7a86355d5ff8: Got IP  [LogTimer=10310ms]
2020-08-12 20:12:54,534	INFO command_runner.py:468 -- NodeUpdater: i-0729c7a86355d5ff8: Running ssh -tt -i /home/richard/.ssh/ray-autoscaler_us-east-1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_6ae199a93c/cfde1c79f1/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@3.226.253.119 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && command -v docker'
Warning: Permanently added '3.226.253.119' (ECDSA) to the list of known hosts.
/usr/bin/docker
Shared connection to 3.226.253.119 closed.
2020-08-12 20:14:04,587	INFO updater.py:71 -- NodeUpdater: i-0729c7a86355d5ff8: Updating to 6b5fc8ee8c5dcdf3cfabe0bf90ba4e844f65a7c9
2020-08-12 20:14:04,587	INFO updater.py:180 -- NodeUpdater: i-0729c7a86355d5ff8: Waiting for remote shell...
2020-08-12 20:14:04,587	INFO command_runner.py:468 -- NodeUpdater: i-0729c7a86355d5ff8: Running ssh -tt -i /home/richard/.ssh/ray-autoscaler_us-east-1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_6ae199a93c/cfde1c79f1/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@3.226.253.119 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && docker exec -it  pytorch_docker /bin/bash -c '"'"'bash --login -c -i '"'"'"'"'"'"'"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && uptime'"'"'"'"'"'"'"'"''"'"' '
2020-08-12 20:14:04,950	INFO log_timer.py:27 -- AWSNodeProvider: Set tag ray-node-status=waiting-for-ssh on ['i-0729c7a86355d5ff8']  [LogTimer=361ms]
                             Error: No such container: pytorch_docker
Shared connection to 3.226.253.119 closed.
2020-08-12 20:14:21,222	INFO command_runner.py:468 -- NodeUpdater: i-0729c7a86355d5ff8: Running ssh -tt -i /home/richard/.ssh/ray-autoscaler_us-east-1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_6ae199a93c/cfde1c79f1/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@3.226.253.119 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && docker exec -it  pytorch_docker /bin/bash -c '"'"'bash --login -c -i '"'"'"'"'"'"'"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && uptime'"'"'"'"'"'"'"'"''"'"' '
Error: No such container: pytorch_docker
Shared connection to 3.226.253.119 closed.
2020-08-12 20:14:26,417	INFO command_runner.py:468 -- NodeUpdater: i-0729c7a86355d5ff8: Running ssh -tt -i /home/richard/.ssh/ray-autoscaler_us-east-1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_6ae199a93c/cfde1c79f1/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@3.226.253.119 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && docker exec -it  pytorch_docker /bin/bash -c '"'"'bash --login -c -i '"'"'"'"'"'"'"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && uptime'"'"'"'"'"'"'"'"''"'"' '
Error: No such container: pytorch_docker
Shared connection to 3.226.253.119 closed.
2020-08-12 20:14:31,610	INFO command_runner.py:468 -- NodeUpdater: i-0729c7a86355d5ff8: Running ssh -tt -i /home/richard/.ssh/ray-autoscaler_us-east-1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_6ae199a93c/cfde1c79f1/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@3.226.253.119 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && docker exec -it  pytorch_docker /bin/bash -c '"'"'bash --login -c -i '"'"'"'"'"'"'"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && uptime'"'"'"'"'"'"'"'"''"'"' '
Error: No such container: pytorch_docker
Shared connection to 3.226.253.119 closed.
2020-08-12 20:14:36,798	INFO command_runner.py:468 -- NodeUpdater: i-0729c7a86355d5ff8: Running ssh -tt -i /home/richard/.ssh/ray-autoscaler_us-east-1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_6ae199a93c/cfde1c79f1/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@3.226.253.119 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && docker exec -it  pytorch_docker /bin/bash -c '"'"'bash --login -c -i '"'"'"'"'"'"'"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && uptime'"'"'"'"'"'"'"'"''"'"' '
Error: No such container: pytorch_docker
Shared connection to 3.226.253.119 closed.
2020-08-12 20:14:41,986	INFO command_runner.py:468 -- NodeUpdater: i-0729c7a86355d5ff8: Running ssh -tt -i /home/richard/.ssh/ray-autoscaler_us-east-1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_6ae199a93c/cfde1c79f1/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@3.226.253.119 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && docker exec -it  pytorch_docker /bin/bash -c '"'"'bash --login -c -i '"'"'"'"'"'"'"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && uptime'"'"'"'"'"'"'"'"''"'"' '
Error: No such container: pytorch_docker
Shared connection to 3.226.253.119 closed.
2020-08-12 20:14:47,170	INFO command_runner.py:468 -- NodeUpdater: i-0729c7a86355d5ff8: Running ssh -tt -i /home/richard/.ssh/ray-autoscaler_us-east-1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_6ae199a93c/cfde1c79f1/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@3.226.253.119 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && docker exec -it  pytorch_docker /bin/bash -c '"'"'bash --login -c -i '"'"'"'"'"'"'"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && uptime'"'"'"'"'"'"'"'"''"'"' '
Error: No such container: pytorch_docker
Shared connection to 3.226.253.119 closed.
2020-08-12 20:14:52,358	INFO command_runner.py:468 -- NodeUpdater: i-0729c7a86355d5ff8: Running ssh -tt -i /home/richard/.ssh/ray-autoscaler_us-east-1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_6ae199a93c/cfde1c79f1/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@3.226.253.119 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && docker exec -it  pytorch_docker /bin/bash -c '"'"'bash --login -c -i '"'"'"'"'"'"'"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && uptime'"'"'"'"'"'"'"'"''"'"' '
Error: No such container: pytorch_docker
Shared connection to 3.226.253.119 closed.
2020-08-12 20:14:57,554	INFO command_runner.py:468 -- NodeUpdater: i-0729c7a86355d5ff8: Running ssh -tt -i /home/richard/.ssh/ray-autoscaler_us-east-1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_6ae199a93c/cfde1c79f1/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@3.226.253.119 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && docker exec -it  pytorch_docker /bin/bash -c '"'"'bash --login -c -i '"'"'"'"'"'"'"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && uptime'"'"'"'"'"'"'"'"''"'"' '
Error: No such container: pytorch_docker
Shared connection to 3.226.253.119 closed.
2020-08-12 20:15:02,750	INFO command_runner.py:468 -- NodeUpdater: i-0729c7a86355d5ff8: Running ssh -tt -i /home/richard/.ssh/ray-autoscaler_us-east-1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_6ae199a93c/cfde1c79f1/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@3.226.253.119 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && docker exec -it  pytorch_docker /bin/bash -c '"'"'bash --login -c -i '"'"'"'"'"'"'"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && uptime'"'"'"'"'"'"'"'"''"'"' '
Error: No such container: pytorch_docker
Shared connection to 3.226.253.119 closed.
2020-08-12 20:15:07,938	INFO command_runner.py:468 -- NodeUpdater: i-0729c7a86355d5ff8: Running ssh -tt -i /home/richard/.ssh/ray-autoscaler_us-east-1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_6ae199a93c/cfde1c79f1/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@3.226.253.119 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && docker exec -it  pytorch_docker /bin/bash -c '"'"'bash --login -c -i '"'"'"'"'"'"'"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && uptime'"'"'"'"'"'"'"'"''"'"' '
Error: No such container: pytorch_docker
Shared connection to 3.226.253.119 closed.
2020-08-12 20:15:13,126	INFO command_runner.py:468 -- NodeUpdater: i-0729c7a86355d5ff8: Running ssh -tt -i /home/richard/.ssh/ray-autoscaler_us-east-1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_6ae199a93c/cfde1c79f1/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@3.226.253.119 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && docker exec -it  pytorch_docker /bin/bash -c '"'"'bash --login -c -i '"'"'"'"'"'"'"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && uptime'"'"'"'"'"'"'"'"''"'"' '
Error: No such container: pytorch_docker
Shared connection to 3.226.253.119 closed.
2020-08-12 20:15:18,307	INFO command_runner.py:468 -- NodeUpdater: i-0729c7a86355d5ff8: Running ssh -tt -i /home/richard/.ssh/ray-autoscaler_us-east-1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_6ae199a93c/cfde1c79f1/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@3.226.253.119 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && docker exec -it  pytorch_docker /bin/bash -c '"'"'bash --login -c -i '"'"'"'"'"'"'"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && uptime'"'"'"'"'"'"'"'"''"'"' '
Error: No such container: pytorch_docker
Shared connection to 3.226.253.119 closed.
2020-08-12 20:15:23,494	INFO command_runner.py:468 -- NodeUpdater: i-0729c7a86355d5ff8: Running ssh -tt -i /home/richard/.ssh/ray-autoscaler_us-east-1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_6ae199a93c/cfde1c79f1/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@3.226.253.119 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && docker exec -it  pytorch_docker /bin/bash -c '"'"'bash --login -c -i '"'"'"'"'"'"'"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && uptime'"'"'"'"'"'"'"'"''"'"' '
Error: No such container: pytorch_docker
Shared connection to 3.226.253.119 closed.
Shared connection to 3.226.253.119 closed.
2020-08-12 20:19:01,502	INFO command_runner.py:468 -- NodeUpdater: i-0729c7a86355d5ff8: Running ssh -tt -i /home/richard/.ssh/ray-autoscaler_us-east-1.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_6ae199a93c/cfde1c79f1/%C -o ControlPersist=10s -o ConnectTimeout=120s ubuntu@3.226.253.119 bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && docker exec -it  pytorch_docker /bin/bash -c '"'"'bash --login -c -i '"'"'"'"'"'"'"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && uptime'"'"'"'"'"'"'"'"''"'"' '
Error: No such container: pytorch_docker
Shared connection to 3.226.253.119 closed.
2020-08-12 20:19:06,689	INFO log_timer.py:27 -- NodeUpdater: i-0729c7a86355d5ff8: Got remote shell  [LogTimer=302102ms]
2020-08-12 20:19:06,690	INFO log_timer.py:27 -- NodeUpdater: i-0729c7a86355d5ff8: Applied config 6b5fc8ee8c5dcdf3cfabe0bf90ba4e844f65a7c9  [LogTimer=302103ms]
2020-08-12 20:19:06,690	ERROR updater.py:88 -- NodeUpdater: i-0729c7a86355d5ff8: Error executing: Unable to connect to node

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/richard/improbable/vanillas/ray/python/ray/autoscaler/updater.py", line 76, in run
    self.do_update()
  File "/home/richard/improbable/vanillas/ray/python/ray/autoscaler/updater.py", line 232, in do_update
    self.wait_ready(deadline)
  File "/home/richard/improbable/vanillas/ray/python/ray/autoscaler/updater.py", line 224, in wait_ready
    assert False, "Unable to connect to node"
AssertionError: Unable to connect to node

2020-08-12 20:19:06,962	ERROR commands.py:650 -- get_or_create_head_node: Updating 3.226.253.119 failed
2020-08-12 20:19:07,002	INFO log_timer.py:27 -- AWSNodeProvider: Set tag ray-node-status=update-failed on ['i-0729c7a86355d5ff8']  [LogTimer=312ms]

```

YAML file for repo attached.

```
# An unique identifier for the head node and workers of this cluster.
cluster_name: richard_cluster_gpu_dummy

# The minimum number of workers nodes to launch in addition to the head
# node. This number should be >= 0.
min_workers: 1

# The maximum number of workers nodes to launch in addition to the head
# node. This takes precedence over min_workers.
max_workers: 5

# The initial number of worker nodes to launch in addition to the head
# node. When the cluster is first brought up (or when it is refreshed with a
# subsequent `ray up`) this number of nodes will be started.
initial_workers: 1

# Whether or not to autoscale aggressively. If this is enabled, if at any point
#   we would start more workers, we start at least enough to bring us to
#   initial_workers.
autoscaling_mode: default

# This executes all commands on all nodes in the docker efcontainer,
# and opens all the necessary ports to support the Ray cluster.
# Empty string means disabled.
docker:
  image: "pytorch/pytorch:latest" # e.g., tensorflow/tensorflow:1.5.0-py3
  container_name: "pytorch_docker" # e.g. ray_docker
  # If true, pulls latest version of image. Otherwise, `docker run` will only pull the image
  # if no cached version is present.
  pull_before_run: True
  run_options: []
#    - $([ -d /proc/driver ] && echo -n --runtime-nvidia) # Use the nvidia runtime only if nvidia gpu's are installed
  worker_run_options:
    - --runtime=nvidia  # Extra options to pass into "docker run"

  # Example of running a GPU head with CPU workers
  # head_image: "tensorflow/tensorflow:1.13.1-py3"
  # head_run_options:
  #     - --runtime=nvidia

  # worker_image: "ubuntu:18.04"
  # worker_run_options: []

# The autoscaler will scale up the cluster to this target fraction of resource
# usage. For example, if a cluster of 10 nodes is 100% busy and
# target_utilization is 0.8, it would resize the cluster to 13. This fraction
# can be decreased to increase the aggressiveness of upscaling.
# This value must be less than 1.0 for scaling to happen.
target_utilization_fraction: 0.8

# If a node is idle for this many minutes, it will be removed.
idle_timeout_minutes: 5

# Cloud-provider specific configuration.
provider:
  type: aws
  region: us-east-1
  # Availability zone(s), comma-separated, that nodes may be launched in.
  # Nodes are currently spread between zones by a round-robin approach,
  # however this implementation detail should not be relied upon.
  availability_zone: us-east-1a, us-east-1b
  cache_stopped_nodes: False

# How Ray will authenticate with newly launched nodes.
auth:
  ssh_user: ubuntu
# By default Ray creates a new private keypair, but you can also use your own.
# If you do so, make sure to also set "KeyName" in the head and worker node
# configurations below.
#    ssh_private_key: /path/to/your/key.pem

# Provider-specific config for the head node, e.g. instance type. By default
# Ray will auto-configure unspecified fields such as SubnetId and KeyName.
# For more documentation on available fields, see:
# http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances
head_node:
  InstanceType: c4.2xlarge
  ImageId: ami-043f9aeaf108ebc37 # Deep Learning AMI (Ubuntu) Version 24.3
#   You can provision additional disk space with a conf as follows
  BlockDeviceMappings:
    - DeviceName: /dev/sda1
      Ebs:
        VolumeSize: 100

  # Additional options in the boto docs.

# Provider-specific config for worker nodes, e.g. instance type. By default
# Ray will auto-configure unspecified fields such as SubnetId and KeyName.
# For more documentation on available fields, see:
# http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances
worker_nodes:
  InstanceType: p3.2xlarge
  ImageId: ami-043f9aeaf108ebc37 # Deep Learning AMI (Ubuntu) Version 24.3

  # Run workers on spot by default. Comment this out to use on-demand.
  InstanceMarketOptions:
    MarketType: spot
    # Additional options can be found in the boto docs, e.g.
    #   SpotOptions:
    #       MaxPrice: MAX_HOURLY_PRICE

  # Additional options in the boto docs.

# Files or directories to copy to the head and worker nodes. The format is a
# dictionary from REMOTE_PATH: LOCAL_PATH, e.g.
file_mounts: {

}



# List of commands that will be run before `setup_commands`. If docker is
# enabled, these commands will run outside the container and before docker
# is setup.
initialization_commands: []

# List of shell commands to run to set up nodes.
setup_commands:
  # Note: if you're developing Ray, you probably want to create an AMI that
  # has your Ray repo pre-cloned. Then, you can replace the pip installs
  # below with a git checkout <your_sha> (and possibly a recompile).
  - pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.9.0.dev0-cp37-cp37m-manylinux1_x86_64.whl
#  - pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.9.0.dev0-cp37-cp37m-manylinux1_x86_64.whl
  # Consider uncommenting these if you also want to run apt-get commands during setup
  # - sudo pkill -9 apt-get || true
  # - sudo pkill -9 dpkg || true
  # - sudo dpkg --configure -a

# Custom commands that will be run on the head node after common setup.
head_setup_commands:
  - pip install boto3  # 1.4.8 adds InstanceMarketOptions

# Custom commands that will be run on worker nodes after common setup.
worker_setup_commands:
  - pip install boto3  # 1.4.8 adds InstanceMarketOptions

# Command to start ray on the head node. You don't need to change this.
head_start_ray_commands:
  - ray stop
  - ulimit -n 65536; ray start --num-cpus=0 --head --port=6379 --object-manager-port=8076 --autoscaling-config=~/ray_bootstrap_config.yaml

# Command to start ray on worker nodes. You don't need to change this.
worker_start_ray_commands:
  - ray stop
  - ulimit -n 65536; ray start --address=$RAY_HEAD_IP:6379 --object-manager-port=8076
```


Discussion:

Resolving patch:
diff --git a/python/ray/autoscaler/command_runner.py b/python/ray/autoscaler/command_runner.py
--- a/python/ray/autoscaler/command_runner.py
+++ b/python/ray/autoscaler/command_runner.py
@@ -556,6 +556,7 @@ def run(
             ssh_options_override_ssh_key=ssh_options_override_ssh_key)
 
     def run_rsync_up(self, source, target):
+        # TODO(ilr) Expose this to before NodeUpdater::sync_file_mounts
         protected_path = target
         if target.find("/root") == 0:
             target = target.replace("/root", "/tmp/root")
diff --git a/python/ray/autoscaler/updater.py b/python/ray/autoscaler/updater.py
--- a/python/ray/autoscaler/updater.py
+++ b/python/ray/autoscaler/updater.py
@@ -145,8 +145,9 @@ def do_sync(remote_path, local_path, allow_non_existing_paths=False):
 
             with LogTimer(self.log_prefix +
                           "Synced {} to {}".format(local_path, remote_path)):
-                self.cmd_runner.run("mkdir -p {}".format(
-                    os.path.dirname(remote_path)))
+                self.cmd_runner.run(
+                    "mkdir -p {}".format(os.path.dirname(remote_path)),
+                    run_env="host")
                 sync_cmd(local_path, remote_path)
 
                 if remote_path not in nolog_paths:
@@ -188,7 +189,7 @@ def wait_ready(self, deadline):
                                              "{}Waiting for remote shell...",
                                              self.log_prefix)
 
-                        self.cmd_runner.run("uptime")
+                        self.cmd_runner.run("uptime", run_env="host")
                         cli_logger.old_debug(logger, "Uptime succeeded.")
                         cli_logger.success("Success.")
                         return True


Test patch: