Repository: ray-project/ray

Issue:
[k8s][autoscaler] Possible command parsing issue, worker nodes quickly dying
<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->

### What is the problem?
There seem to be multiple bugs in the kuburnetes autoscaler. When running `ray`, commands appear to be parsed incorrectly,  resulting in the failure to start the ray runtime on the head node. After manually entering the pod on the head node and starting the ray runtime, a worker node is briefly spawned and quickly dies.

### Version/System
I'm working off of the latest master branch ray v0.9.0dev0, with Python3.7 on Ubuntu 20.04. I'm running on a preexisting Kubernetes cluster of two c5.2xlarge nodes on AWS.

### Reproduction
Please provide a script that can be run to reproduce the issue. The script should have **no external library dependencies** (i.e., use fake or mock data / environments):

I use the provided example-full.yaml file with the only difference being the following configuration:
```
min_workers: 1
max_workers: 1
initial_workers: 1
```
Started with the following command:
`ray up python/ray/autoscaler/kubernetes/example-full.yaml`

This produces the following output. The ray runtime fails to start. 

```
(myenv) tcw@pop-os:~/projects/ray$ ray up example-full.yaml 
Commands running under a login shell can produce more output than special processing can handle.
Thus, the output from subcommands will be logged as is.
Consider using --use-normal-shells, if you tested your workflow and it is compatible.

Cluster configuration valid
Cluster: default
2020-09-09 20:16:16,175 INFO config.py:68 -- KubernetesNodeProvider: namespace 'ray' not found, attempting to create it
2020-09-09 20:16:16,213 INFO config.py:72 -- KubernetesNodeProvider: successfully created namespace 'ray'
2020-09-09 20:16:16,250 INFO config.py:97 -- KubernetesNodeProvider: autoscaler_service_account 'autoscaler' not found, attempting to create it
2020-09-09 20:16:16,291 INFO config.py:99 -- KubernetesNodeProvider: successfully created autoscaler_service_account 'autoscaler'
2020-09-09 20:16:16,401 INFO config.py:123 -- KubernetesNodeProvider: autoscaler_role 'autoscaler' not found, attempting to create it
2020-09-09 20:16:16,436 INFO config.py:125 -- KubernetesNodeProvider: successfully created autoscaler_role 'autoscaler'
2020-09-09 20:16:16,473 INFO config.py:156 -- KubernetesNodeProvider: autoscaler_role_binding 'autoscaler' not found, attempting to create it
2020-09-09 20:16:16,518 INFO config.py:158 -- KubernetesNodeProvider: successfully created autoscaler_role_binding 'autoscaler'
2020-09-09 20:16:16,554 INFO config.py:189 -- KubernetesNodeProvider: service 'ray-head' not found, attempting to create it
2020-09-09 20:16:16,613 INFO config.py:191 -- KubernetesNodeProvider: successfully created service 'ray-head'
2020-09-09 20:16:16,653 INFO config.py:189 -- KubernetesNodeProvider: service 'ray-workers' not found, attempting to create it
2020-09-09 20:16:16,702 INFO config.py:191 -- KubernetesNodeProvider: successfully created service 'ray-workers'
This will create a new cluster [y/N]: y
2020-09-09 20:16:18,507 INFO commands.py:592 -- get_or_create_head_node: Launching new head node...
2020-09-09 20:16:18,507 INFO node_provider.py:87 -- KubernetesNodeProvider: calling create_namespaced_pod (count=1).
2020-09-09 20:16:18,606 INFO commands.py:631 -- get_or_create_head_node: Updating files on head node...
2020-09-09 20:16:18,609 INFO updater.py:97 -- NodeUpdater: ray-head-xn4b5: Updating to c1ad72dff590afc61824084810dc92021a0f3d39
2020-09-09 20:16:18,697 INFO updater.py:219 -- NodeUpdater: ray-head-xn4b5: Waiting for remote shell...
2020-09-09 20:16:18,729 INFO command_runner.py:223 -- NodeUpdater: ray-head-xn4b5: Running ['kubectl', '-n', 'ray', 'exec', '-it', 'ray-head-xn4b5', '--', 'bash', '--login', '-c', '-i', "'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'"]
kubectl controls the Kubernetes cluster manager.

 Find more information at: https://kubernetes.io/docs/reference/kubectl/overview/

Basic Commands (Beginner):
  create         Create a resource from a file or from stdin.
  expose         Take a replication controller, service, deployment or pod and expose it as a new Kubernetes Service
  run            Run a particular image on the cluster
  set            Set specific features on objects

Basic Commands (Intermediate):
  explain        Documentation of resources
  get            Display one or many resources
  edit           Edit a resource on the server
  delete         Delete resources by filenames, stdin, resources and names, or by resources and label selector

Deploy Commands:
  rollout        Manage the rollout of a resource
  scale          Set a new size for a Deployment, ReplicaSet or Replication Controller
  autoscale      Auto-scale a Deployment, ReplicaSet, or ReplicationController

Cluster Management Commands:
  certificate    Modify certificate resources.
  cluster-info   Display cluster info
  top            Display Resource (CPU/Memory/Storage) usage.
  cordon         Mark node as unschedulable
  uncordon       Mark node as schedulable
  drain          Drain node in preparation for maintenance
  taint          Update the taints on one or more nodes

Troubleshooting and Debugging Commands:
  describe       Show details of a specific resource or group of resources
  logs           Print the logs for a container in a pod
  attach         Attach to a running container
  exec           Execute a command in a container
  port-forward   Forward one or more local ports to a pod
  proxy          Run a proxy to the Kubernetes API server
  cp             Copy files and directories to and from containers.
  auth           Inspect authorization

Advanced Commands:
  diff           Diff live version against would-be applied version
  apply          Apply a configuration to a resource by filename or stdin
  patch          Update field(s) of a resource using strategic merge patch
  replace        Replace a resource by filename or stdin
  wait           Experimental: Wait for a specific condition on one or many resources.
  convert        Convert config files between different API versions
  kustomize      Build a kustomization target from a directory or a remote url.

Settings Commands:
  label          Update the labels on a resource
  annotate       Update the annotations on a resource
  completion     Output shell completion code for the specified shell (bash or zsh)

Other Commands:
  api-resources  Print the supported API resources on the server
  api-versions   Print the supported API versions on the server, in the form of "group/version"
  config         Modify kubeconfig files
  plugin         Provides utilities for interacting with plugins.
  version        Print the client and server version information

Usage:
  kubectl [flags] [options]

Use "kubectl <command> --help" for more information about a given command.
Use "kubectl options" for a list of global command-line options (applies to all commands).
2020-09-09 20:16:18,777 INFO log_timer.py:27 -- NodeUpdater: ray-head-xn4b5: Got remote shell  [LogTimer=80ms]
2020-09-09 20:16:18,905 INFO command_runner.py:223 -- NodeUpdater: ray-head-xn4b5: Running ['kubectl', '-n', 'ray', 'exec', '-it', 'ray-head-xn4b5', '--', 'bash', '--login', '-c', '-i', "'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (mkdir -p ~)'"]
kubectl controls the Kubernetes cluster manager.

 Find more information at: https://kubernetes.io/docs/reference/kubectl/overview/

Basic Commands (Beginner):
  create         Create a resource from a file or from stdin.
  expose         Take a replication controller, service, deployment or pod and expose it as a new Kubernetes Service
  run            Run a particular image on the cluster
  set            Set specific features on objects

Basic Commands (Intermediate):
  explain        Documentation of resources
  get            Display one or many resources
  edit           Edit a resource on the server
  delete         Delete resources by filenames, stdin, resources and names, or by resources and label selector

Deploy Commands:
  rollout        Manage the rollout of a resource
  scale          Set a new size for a Deployment, ReplicaSet or Replication Controller
  autoscale      Auto-scale a Deployment, ReplicaSet, or ReplicationController

Cluster Management Commands:
  certificate    Modify certificate resources.
  cluster-info   Display cluster info
  top            Display Resource (CPU/Memory/Storage) usage.
  cordon         Mark node as unschedulable
  uncordon       Mark node as schedulable
  drain          Drain node in preparation for maintenance
  taint          Update the taints on one or more nodes

Troubleshooting and Debugging Commands:
  describe       Show details of a specific resource or group of resources
  logs           Print the logs for a container in a pod
  attach         Attach to a running container
  exec           Execute a command in a container
  port-forward   Forward one or more local ports to a pod
  proxy          Run a proxy to the Kubernetes API server
  cp             Copy files and directories to and from containers.
  auth           Inspect authorization

Advanced Commands:
  diff           Diff live version against would-be applied version
  apply          Apply a configuration to a resource by filename or stdin
  patch          Update field(s) of a resource using strategic merge patch
  replace        Replace a resource by filename or stdin
  wait           Experimental: Wait for a specific condition on one or many resources.
  convert        Convert config files between different API versions
  kustomize      Build a kustomization target from a directory or a remote url.

Settings Commands:
  label          Update the labels on a resource
  annotate       Update the annotations on a resource
  completion     Output shell completion code for the specified shell (bash or zsh)

Other Commands:
  api-resources  Print the supported API resources on the server
  api-versions   Print the supported API versions on the server, in the form of "group/version"
  config         Modify kubeconfig files
  plugin         Provides utilities for interacting with plugins.
  version        Print the client and server version information

Usage:
  kubectl [flags] [options]

Use "kubectl <command> --help" for more information about a given command.
Use "kubectl options" for a list of global command-line options (applies to all commands).
2020-09-09 20:16:18,985 INFO updater.py:420 -- NodeUpdater: ray-head-xn4b5: Syncing /tmp/ray-bootstrap-ixliccej to ~/ray_bootstrap_config.yaml...
error: unable to upgrade connection: container not found ("ray-node")
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.3]
2020-09-09 20:16:19,670 WARNING command_runner.py:255 -- NodeUpdater: ray-head-xn4b5: rsync failed: 'Command '['/home/tcw/projects/mu0_minimal/myenv/lib/python3.7/site-packages/ray/autoscaler/kubernetes/kubectl-rsync.sh', '-avz', '/tmp/ray-bootstrap-ixliccej', 'ray-head-xn4b5@ray:/root/ray_bootstrap_config.yaml']' returned non-zero exit status 12.'. Falling back to 'kubectl cp'
2020-09-09 20:16:20,912 INFO log_timer.py:27 -- NodeUpdater: ray-head-xn4b5: Synced /tmp/ray-bootstrap-ixliccej to ~/ray_bootstrap_config.yaml  [LogTimer=2006ms]
2020-09-09 20:16:21,012 INFO command_runner.py:223 -- NodeUpdater: ray-head-xn4b5: Running ['kubectl', '-n', 'ray', 'exec', '-it', 'ray-head-xn4b5', '--', 'bash', '--login', '-c', '-i', "'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (ray stop)'"]
kubectl controls the Kubernetes cluster manager.

 Find more information at: https://kubernetes.io/docs/reference/kubectl/overview/

Basic Commands (Beginner):
  create         Create a resource from a file or from stdin.
  expose         Take a replication controller, service, deployment or pod and expose it as a new Kubernetes Service
  run            Run a particular image on the cluster
  set            Set specific features on objects

Basic Commands (Intermediate):
  explain        Documentation of resources
  get            Display one or many resources
  edit           Edit a resource on the server
  delete         Delete resources by filenames, stdin, resources and names, or by resources and label selector

Deploy Commands:
  rollout        Manage the rollout of a resource
  scale          Set a new size for a Deployment, ReplicaSet or Replication Controller
  autoscale      Auto-scale a Deployment, ReplicaSet, or ReplicationController

Cluster Management Commands:
  certificate    Modify certificate resources.
  cluster-info   Display cluster info
  top            Display Resource (CPU/Memory/Storage) usage.
  cordon         Mark node as unschedulable
  uncordon       Mark node as schedulable
  drain          Drain node in preparation for maintenance
  taint          Update the taints on one or more nodes

Troubleshooting and Debugging Commands:
  describe       Show details of a specific resource or group of resources
  logs           Print the logs for a container in a pod
  attach         Attach to a running container
  exec           Execute a command in a container
  port-forward   Forward one or more local ports to a pod
  proxy          Run a proxy to the Kubernetes API server
  cp             Copy files and directories to and from containers.
  auth           Inspect authorization

Advanced Commands:
  diff           Diff live version against would-be applied version
  apply          Apply a configuration to a resource by filename or stdin
  patch          Update field(s) of a resource using strategic merge patch
  replace        Replace a resource by filename or stdin
  wait           Experimental: Wait for a specific condition on one or many resources.
  convert        Convert config files between different API versions
  kustomize      Build a kustomization target from a directory or a remote url.

Settings Commands:
  label          Update the labels on a resource
  annotate       Update the annotations on a resource
  completion     Output shell completion code for the specified shell (bash or zsh)

Other Commands:
  api-resources  Print the supported API resources on the server
  api-versions   Print the supported API versions on the server, in the form of "group/version"
  config         Modify kubeconfig files
  plugin         Provides utilities for interacting with plugins.
  version        Print the client and server version information

Usage:
  kubectl [flags] [options]

Use "kubectl <command> --help" for more information about a given command.
Use "kubectl options" for a list of global command-line options (applies to all commands).
2020-09-09 20:16:21,057 INFO command_runner.py:223 -- NodeUpdater: ray-head-xn4b5: Running ['kubectl', '-n', 'ray', 'exec', '-it', 'ray-head-xn4b5', '--', 'bash', '--login', '-c', '-i', "'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (ulimit -n 65536; ray start --head --num-cpus=$MY_CPU_REQUEST --port=6379 --object-manager-port=8076 --autoscaling-config=~/ray_bootstrap_config.yaml --webui-host 0.0.0.0)'"]
kubectl controls the Kubernetes cluster manager.

 Find more information at: https://kubernetes.io/docs/reference/kubectl/overview/

Basic Commands (Beginner):
  create         Create a resource from a file or from stdin.
  expose         Take a replication controller, service, deployment or pod and expose it as a new Kubernetes Service
  run            Run a particular image on the cluster
  set            Set specific features on objects

Basic Commands (Intermediate):
  explain        Documentation of resources
  get            Display one or many resources
  edit           Edit a resource on the server
  delete         Delete resources by filenames, stdin, resources and names, or by resources and label selector

Deploy Commands:
  rollout        Manage the rollout of a resource
  scale          Set a new size for a Deployment, ReplicaSet or Replication Controller
  autoscale      Auto-scale a Deployment, ReplicaSet, or ReplicationController

Cluster Management Commands:
  certificate    Modify certificate resources.
  cluster-info   Display cluster info
  top            Display Resource (CPU/Memory/Storage) usage.
  cordon         Mark node as unschedulable
  uncordon       Mark node as schedulable
  drain          Drain node in preparation for maintenance
  taint          Update the taints on one or more nodes

Troubleshooting and Debugging Commands:
  describe       Show details of a specific resource or group of resources
  logs           Print the logs for a container in a pod
  attach         Attach to a running container
  exec           Execute a command in a container
  port-forward   Forward one or more local ports to a pod
  proxy          Run a proxy to the Kubernetes API server
  cp             Copy files and directories to and from containers.
  auth           Inspect authorization

Advanced Commands:
  diff           Diff live version against would-be applied version
  apply          Apply a configuration to a resource by filename or stdin
  patch          Update field(s) of a resource using strategic merge patch
  replace        Replace a resource by filename or stdin
  wait           Experimental: Wait for a specific condition on one or many resources.
  convert        Convert config files between different API versions
  kustomize      Build a kustomization target from a directory or a remote url.

Settings Commands:
  label          Update the labels on a resource
  annotate       Update the annotations on a resource
  completion     Output shell completion code for the specified shell (bash or zsh)

Other Commands:
  api-resources  Print the supported API resources on the server
  api-versions   Print the supported API versions on the server, in the form of "group/version"
  config         Modify kubeconfig files
  plugin         Provides utilities for interacting with plugins.
  version        Print the client and server version information

Usage:
  kubectl [flags] [options]

Use "kubectl <command> --help" for more information about a given command.
Use "kubectl options" for a list of global command-line options (applies to all commands).
2020-09-09 20:16:21,096 INFO log_timer.py:27 -- NodeUpdater: ray-head-xn4b5: Ray start commands succeeded [LogTimer=84ms]
2020-09-09 20:16:21,096 INFO log_timer.py:27 -- NodeUpdater: ray-head-xn4b5: Applied config c1ad72dff590afc61824084810dc92021a0f3d39  [LogTimer=2486ms]
2020-09-09 20:16:21,254 INFO commands.py:718 -- get_or_create_head_node: Head node up-to-date, IP address is: 192.168.6.142
To monitor autoscaling activity, you can run:

  ray exec /home/tcw/projects/mu0_minimal/ray/example-full.yaml 'tail -n 100 -f /tmp/ray/session_*/logs/monitor*'

To open a console on the cluster:

  ray attach /home/tcw/projects/mu0_minimal/ray/example-full.yaml

To get a remote shell to the cluster manually, run:

  kubectl -n ray exec -it ray-head-xn4b5 bash

```

Other ray commands like `ray attach` and `ray down` likewise fail.

Upon entering the head node and starting the ray runtime with

```
$ kubectl -n ray exec -it ray-head-xn4b5 bash
(base) root@ray-head-xn4b5:/# ray start --head --num-cpus=3 --port=6379 --object-manager-port=8076 --autoscaling-config=~/ray_bootstrap_config.yaml --webui-host 0.0.0.0
```
and getting logs with 

```
(base) root@ray-head-xn4b5:/# tail -n 100 -f /tmp/ray/session_*/logs/monitor*
```
I received the following output:

```
==> /tmp/ray/session_2020-09-09_17-20-39_779593_74/logs/monitor.err <==
    docker_config = self._get_node_specific_docker_config(node_id)
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 431, in _get_node_specific_docker_config
    node_id, "docker")
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 417, in _get_node_type_specific_fields
    fields = self.config[fields_key]
KeyError: 'docker'
2020-09-09 17:21:11,100 INFO autoscaler.py:520 -- Cluster status: 1/1 target nodes (0 pending)
 - MostDelayedHeartbeats: {'192.168.6.142': 0.13077020645141602}
 - NodeIdleSeconds: Min=29 Mean=29 Max=29
 - NumNodesConnected: 1
 - NumNodesUsed: 0.0
 - ResourceUsage: 0.0/3.0 CPU, 0.0 GiB/1.21 GiB memory, 0.0 GiB/0.42 GiB object_store_memory
 - TimeSinceLastHeartbeat: Min=0 Mean=0 Max=0

2020-09-09 17:21:11,112 INFO autoscaler.py:520 -- Cluster status: 1/1 target nodes (0 pending)
 - MostDelayedHeartbeats: {'192.168.6.142': 0.14269304275512695}
 - NodeIdleSeconds: Min=29 Mean=29 Max=29
 - NumNodesConnected: 1
 - NumNodesUsed: 0.0
 - ResourceUsage: 0.0/3.0 CPU, 0.0 GiB/1.21 GiB memory, 0.0 GiB/0.42 GiB object_store_memory
 - TimeSinceLastHeartbeat: Min=0 Mean=0 Max=0

2020-09-09 17:21:11,128 ERROR autoscaler.py:123 -- StandardAutoscaler: Error during autoscaling.
Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 121, in update
    self._update()
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 250, in _update
    self.should_update(node_id) for node_id in nodes):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 250, in <genexpr>
    self.should_update(node_id) for node_id in nodes):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 456, in should_update
    docker_config = self._get_node_specific_docker_config(node_id)
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 431, in _get_node_specific_docker_config
    node_id, "docker")
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 417, in _get_node_type_specific_fields
    fields = self.config[fields_key]
KeyError: 'docker'
2020-09-09 17:21:11,129 CRITICAL autoscaler.py:130 -- StandardAutoscaler: Too many errors, abort.
Error in monitor loop
Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/monitor.py", line 313, in run
    self._run()
  File "/root/anaconda3/lib/python3.7/site-packages/ray/monitor.py", line 268, in _run
    self.autoscaler.update()
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 132, in update
    raise e
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 121, in update
    self._update()
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 250, in _update
    self.should_update(node_id) for node_id in nodes):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 250, in <genexpr>
    self.should_update(node_id) for node_id in nodes):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 456, in should_update
    docker_config = self._get_node_specific_docker_config(node_id)
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 431, in _get_node_specific_docker_config
    node_id, "docker")
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 417, in _get_node_type_specific_fields
    fields = self.config[fields_key]
KeyError: 'docker'
2020-09-09 17:21:11,129 ERROR autoscaler.py:554 -- StandardAutoscaler: kill_workers triggered
2020-09-09 17:21:11,134 INFO node_provider.py:121 -- KubernetesNodeProvider: calling delete_namespaced_pod
2020-09-09 17:21:11,148 ERROR autoscaler.py:559 -- StandardAutoscaler: terminated 1 node(s)
Error in sys.excepthook:
Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 834, in custom_excepthook
    worker_id = global_worker.worker_id
AttributeError: 'Worker' object has no attribute 'worker_id'

Original exception was:
Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/monitor.py", line 368, in <module>
    monitor.run()
  File "/root/anaconda3/lib/python3.7/site-packages/ray/monitor.py", line 313, in run
    self._run()
  File "/root/anaconda3/lib/python3.7/site-packages/ray/monitor.py", line 268, in _run
    self.autoscaler.update()
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 132, in update
    raise e
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 121, in update
    self._update()
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 250, in _update
    self.should_update(node_id) for node_id in nodes):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 250, in <genexpr>
    self.should_update(node_id) for node_id in nodes):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 456, in should_update
    docker_config = self._get_node_specific_docker_config(node_id)
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 431, in _get_node_specific_docker_config
    node_id, "docker")
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 417, in _get_node_type_specific_fields
    fields = self.config[fields_key]
KeyError: 'docker'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/monitor.py", line 380, in <module>
    redis_client, ray_constants.MONITOR_DIED_ERROR, message)
  File "/root/anaconda3/lib/python3.7/site-packages/ray/utils.py", line 128, in push_error_to_driver_through_redis
    pubsub_msg.SerializeAsString())
AttributeError: SerializeAsString

==> /tmp/ray/session_2020-09-09_17-20-39_779593_74/logs/monitor.out <==
Destroying cluster. Confirm [y/N]: y [automatic, due to --yes]
1 random worker nodes will not be shut down. (due to --keep-min-workers)
The head node will not be shut down. (due to --workers-only)
No nodes remaining.

==> /tmp/ray/session_latest/logs/monitor.err <==
    docker_config = self._get_node_specific_docker_config(node_id)
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 431, in _get_node_specific_docker_config
    node_id, "docker")
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 417, in _get_node_type_specific_fields
    fields = self.config[fields_key]
KeyError: 'docker'
2020-09-09 17:21:11,100 INFO autoscaler.py:520 -- Cluster status: 1/1 target nodes (0 pending)
 - MostDelayedHeartbeats: {'192.168.6.142': 0.13077020645141602}
 - NodeIdleSeconds: Min=29 Mean=29 Max=29
 - NumNodesConnected: 1
 - NumNodesUsed: 0.0
 - ResourceUsage: 0.0/3.0 CPU, 0.0 GiB/1.21 GiB memory, 0.0 GiB/0.42 GiB object_store_memory
 - TimeSinceLastHeartbeat: Min=0 Mean=0 Max=0

2020-09-09 17:21:11,112 INFO autoscaler.py:520 -- Cluster status: 1/1 target nodes (0 pending)
 - MostDelayedHeartbeats: {'192.168.6.142': 0.14269304275512695}
 - NodeIdleSeconds: Min=29 Mean=29 Max=29
 - NumNodesConnected: 1
 - NumNodesUsed: 0.0
 - ResourceUsage: 0.0/3.0 CPU, 0.0 GiB/1.21 GiB memory, 0.0 GiB/0.42 GiB object_store_memory
 - TimeSinceLastHeartbeat: Min=0 Mean=0 Max=0

2020-09-09 17:21:11,128 ERROR autoscaler.py:123 -- StandardAutoscaler: Error during autoscaling.
Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 121, in update
    self._update()
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 250, in _update
    self.should_update(node_id) for node_id in nodes):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 250, in <genexpr>
    self.should_update(node_id) for node_id in nodes):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 456, in should_update
    docker_config = self._get_node_specific_docker_config(node_id)
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 431, in _get_node_specific_docker_config
    node_id, "docker")
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 417, in _get_node_type_specific_fields
    fields = self.config[fields_key]
KeyError: 'docker'
2020-09-09 17:21:11,129 CRITICAL autoscaler.py:130 -- StandardAutoscaler: Too many errors, abort.
Error in monitor loop
Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/monitor.py", line 313, in run
    self._run()
  File "/root/anaconda3/lib/python3.7/site-packages/ray/monitor.py", line 268, in _run
    self.autoscaler.update()
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 132, in update
    raise e
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 121, in update
    self._update()
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 250, in _update
    self.should_update(node_id) for node_id in nodes):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 250, in <genexpr>
    self.should_update(node_id) for node_id in nodes):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 456, in should_update
    docker_config = self._get_node_specific_docker_config(node_id)
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 431, in _get_node_specific_docker_config
    node_id, "docker")
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 417, in _get_node_type_specific_fields
    fields = self.config[fields_key]
KeyError: 'docker'
2020-09-09 17:21:11,129 ERROR autoscaler.py:554 -- StandardAutoscaler: kill_workers triggered
2020-09-09 17:21:11,134 INFO node_provider.py:121 -- KubernetesNodeProvider: calling delete_namespaced_pod
2020-09-09 17:21:11,148 ERROR autoscaler.py:559 -- StandardAutoscaler: terminated 1 node(s)
Error in sys.excepthook:
Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 834, in custom_excepthook
    worker_id = global_worker.worker_id
AttributeError: 'Worker' object has no attribute 'worker_id'

Original exception was:
Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/monitor.py", line 368, in <module>
    monitor.run()
  File "/root/anaconda3/lib/python3.7/site-packages/ray/monitor.py", line 313, in run
    self._run()
  File "/root/anaconda3/lib/python3.7/site-packages/ray/monitor.py", line 268, in _run
    self.autoscaler.update()
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 132, in update
    raise e
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 121, in update
    self._update()
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 250, in _update
    self.should_update(node_id) for node_id in nodes):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 250, in <genexpr>
    self.should_update(node_id) for node_id in nodes):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 456, in should_update
    docker_config = self._get_node_specific_docker_config(node_id)
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 431, in _get_node_specific_docker_config
    node_id, "docker")
  File "/root/anaconda3/lib/python3.7/site-packages/ray/autoscaler/autoscaler.py", line 417, in _get_node_type_specific_fields
    fields = self.config[fields_key]
KeyError: 'docker'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.7/site-packages/ray/monitor.py", line 380, in <module>
    redis_client, ray_constants.MONITOR_DIED_ERROR, message)
  File "/root/anaconda3/lib/python3.7/site-packages/ray/utils.py", line 128, in push_error_to_driver_through_redis
    pubsub_msg.SerializeAsString())
AttributeError: SerializeAsString

==> /tmp/ray/session_latest/logs/monitor.out <==
Destroying cluster. Confirm [y/N]: y [automatic, due to --yes]
1 random worker nodes will not be shut down. (due to --keep-min-workers)
The head node will not be shut down. (due to --workers-only)
No nodes remaining.

```
If we cannot run your script, we cannot fix your issue.

- [x] I have verified my script runs in a clean environment and reproduces the issue.
- [x] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).



Discussion:
@twillkens thanks a bunch for taking the time to open this issue! 

cc @ijrsvt @wuisawesome could you take a look at this? I've marked it as a release blocker for now.

Resolving patch:
diff --git a/python/ray/autoscaler/autoscaler.py b/python/ray/autoscaler/autoscaler.py
--- a/python/ray/autoscaler/autoscaler.py
+++ b/python/ray/autoscaler/autoscaler.py
@@ -430,6 +430,8 @@ def _get_node_type_specific_fields(self, node_id: str,
         return fields
 
     def _get_node_specific_docker_config(self, node_id):
+        if "docker" not in self.config:
+            return {}
         docker_config = copy.deepcopy(self.config.get("docker", {}))
         node_specific_docker = self._get_node_type_specific_fields(
             node_id, "docker")


Test patch:
diff --git a/python/ray/tests/test_resource_demand_scheduler.py b/python/ray/tests/test_resource_demand_scheduler.py
--- a/python/ray/tests/test_resource_demand_scheduler.py
+++ b/python/ray/tests/test_resource_demand_scheduler.py
@@ -605,6 +605,32 @@ def testUpdateConfig(self):
         autoscaler.update()
         self.waitForNodes(0)
 
+    def testEmptyDocker(self):
+        config = MULTI_WORKER_CLUSTER.copy()
+        del config["docker"]
+        config["min_workers"] = 0
+        config["max_workers"] = 10
+        config_path = self.write_config(config)
+        self.provider = MockProvider()
+        runner = MockProcessRunner()
+        autoscaler = StandardAutoscaler(
+            config_path,
+            LoadMetrics(),
+            max_failures=0,
+            process_runner=runner,
+            update_interval_s=0)
+        assert len(self.provider.non_terminated_nodes({})) == 0
+        autoscaler.update()
+        self.waitForNodes(0)
+        autoscaler.request_resources([{"CPU": 1}])
+        autoscaler.update()
+        self.waitForNodes(1)
+        assert self.provider.mock_nodes[0].node_type == "m4.large"
+        autoscaler.request_resources([{"GPU": 8}])
+        autoscaler.update()
+        self.waitForNodes(2)
+        assert self.provider.mock_nodes[1].node_type == "p2.8xlarge"
+
 
 if __name__ == "__main__":
     import sys