Repository: ray-project/ray

Issue:
[autoscaler] cluster_synced_files doesn't support `~`
### What is the problem?

`~/filename` is marked as "nonexistent" despite actually existing on the cluster.
*Ray version and other system information (Python version, TensorFlow version, OS):* master


### Reproduction (REQUIRED)
Please provide a script that can be run to reproduce the issue. The script should have **no external library dependencies** (i.e., use fake or mock data / environments):

```
sent 10,834 bytes  received 637 bytes  22,942.00 bytes/sec
total size is 684,320  speedup is 59.66
    /home/ubuntu/.local/lib/python3.7/site-packages/ray/autoscaler/ from /home/ubuntu/.local/lib/python3.7/site-packages/ray/autoscaler/
  [4/6] Processing worker file mounts
synced files:  ['~/my-cluster-synced-file']
Does ~/my-cluster-synced-file exist? False
```


```
cluster_name: SYNC_TEST
min_workers: 2
max_workers: 2
initial_workers: 2
autoscaling_mode: default
target_utilization_fraction: 0.8
idle_timeout_minutes: 60
provider:
  type: aws
  region: us-east-1
  cache_stopped_nodes: false
auth:
  ssh_user: ubuntu
head_node:
  BlockDeviceMappings:
  - DeviceName: /dev/sda1
    Ebs:
      VolumeSize: 350
      DeleteOnTermination: true
      VolumeType: gp2
  ImageId: latest_dlami
  InstanceType: t3.xlarge
worker_nodes:
  BlockDeviceMappings:
  - DeviceName: /dev/sda1
    Ebs:
      VolumeSize: 300
      DeleteOnTermination: true
      VolumeType: gp2
  ImageId: latest_dlami
  InstanceType: t3.xlarge

cluster_synced_files:
- ~/my-cluster-synced-file
file_mounts_sync_continuously: true
initialization_commands: []
setup_commands:
- pip install --user https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.9.0.dev0-cp37-cp37m-manylinux1_x86_64.whl
head_setup_commands:
- tmux new-session -d "python3 -m http.server"
- pip install --user  https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.9.0.dev0-cp37-cp37m-manylinux1_x86_64.whl
- sleep 5
- curl localhost:8000 -o ~/my-cluster-synced-file
worker_setup_commands: []

```

If we cannot run your script, we cannot fix your issue.

- [ ] I have verified my script runs in a clean environment and reproduces the issue.
- [ ] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).



Discussion:

Resolving patch:
diff --git a/python/ray/autoscaler/updater.py b/python/ray/autoscaler/updater.py
--- a/python/ray/autoscaler/updater.py
+++ b/python/ray/autoscaler/updater.py
@@ -78,17 +78,28 @@ def __init__(self,
         self.process_runner = process_runner
         self.node_id = node_id
         self.provider = provider
+        # Some node providers don't specify empty structures as
+        # defaults. Better to be defensive.
+        file_mounts = file_mounts or {}
         self.file_mounts = {
             remote: os.path.expanduser(local)
             for remote, local in file_mounts.items()
         }
+
         self.initialization_commands = initialization_commands
         self.setup_commands = setup_commands
         self.ray_start_commands = ray_start_commands
         self.node_resources = node_resources
         self.runtime_hash = runtime_hash
         self.file_mounts_contents_hash = file_mounts_contents_hash
-        self.cluster_synced_files = cluster_synced_files
+        # TODO (Alex): This makes the assumption that $HOME on the head and
+        # worker nodes is the same. Also note that `cluster_synced_files` is
+        # set on the head -> worker updaters only (so `expanduser` is only run
+        # on the head node).
+        cluster_synced_files = cluster_synced_files or []
+        self.cluster_synced_files = [
+            os.path.expanduser(path) for path in cluster_synced_files
+        ]
         self.auth_config = auth_config
         self.is_head_node = is_head_node
 
@@ -166,6 +177,8 @@ def sync_file_mounts(self, sync_cmd, step_numbers=(0, 2)):
 
         def do_sync(remote_path, local_path, allow_non_existing_paths=False):
             if allow_non_existing_paths and not os.path.exists(local_path):
+                cli_logger.print("sync: {} does not exist. Skipping.",
+                                 local_path)
                 # Ignore missing source files. In the future we should support
                 # the --delete-missing-args command to delete files that have
                 # been removed
@@ -199,17 +212,21 @@ def do_sync(remote_path, local_path, allow_non_existing_paths=False):
                 _numbered=("[]", previous_steps + 1, total_steps)):
             for remote_path, local_path in self.file_mounts.items():
                 do_sync(remote_path, local_path)
+            previous_steps += 1
 
         if self.cluster_synced_files:
             with cli_logger.group(
                     "Processing worker file mounts",
-                    _numbered=("[]", previous_steps + 2, total_steps)):
+                    _numbered=("[]", previous_steps + 1, total_steps)):
+                cli_logger.print("synced files: {}",
+                                 str(self.cluster_synced_files))
                 for path in self.cluster_synced_files:
                     do_sync(path, path, allow_non_existing_paths=True)
+                previous_steps += 1
         else:
             cli_logger.print(
                 "No worker file mounts to sync",
-                _numbered=("[]", previous_steps + 2, total_steps))
+                _numbered=("[]", previous_steps + 1, total_steps))
 
     def wait_ready(self, deadline):
         with cli_logger.group(


Test patch: