Repository: skypilot-org/skypilot
Pull request: Support file_mounts destination paths that require root access.
Fixes #39.

Both these should now work:
```
/data/<25GB dir>: /data/<25GB dir>
/data/<25GB dir>: gs://<dir>
```
Added `examples/using_file_mounts.yaml` as a user guide.

- [x] test resnet_app file_mounts again

Problem statement:
Error when creating directory during filemount
I specify file_mounts like below
```
file_mounts = {
        '/data/weichiang/dataset/': '/data/weichiang/dataset/',
    }
```
the autoscaler setup failed and I got the permission denied error. 
```
mkdir: cannot create directory ‘/data’: Permission denied           
```
I had to change all the data paths in my codebase to make it work, which may be too inconvenient for users.
Can we grant root permission during filemount?

Hints from issue discussion:
file_mounts is synced before setup/init commands in autoscaler design: https://github.com/ray-project/ray/blob/333d0b43fdce13bc46b1f2b43d0b9f4d758ab1e9/python/ray/autoscaler/_private/updater.py#L349-L361
so unfortunately we can't sudo make that dir first before file_mounts are synced. 

How about
```
file_mounts = {
        '~/data/weichiang/dataset/': '/data/weichiang/dataset/',
}
setup = 'sudo ln -s ~/data /data && sudo chmod 777 /data && <rest of setup>'
```

@romilbhardwaj another reason for docker.

Commit message: Support file_mounts destination paths that require root access.

Fixes #39.

Patch to review:
diff --git a/prototype/config/aws-ray.yml.j2 b/prototype/config/aws-ray.yml.j2
index fb81c684e00..42a9b9eb120 100644
--- a/prototype/config/aws-ray.yml.j2
+++ b/prototype/config/aws-ray.yml.j2
@@ -65,6 +65,13 @@ file_mounts: {
 {%- endfor %}
 }
 
+{%- if initialization_commands is not none %}
+initialization_commands:
+{%- for cmd in initialization_commands %}
+  - {{cmd}}
+{%- endfor %}
+{%- endif %}
+
 # Patterns for files to exclude when running rsync up or rsync down
 # rsync_exclude:
 #   - "**/.git"
diff --git a/prototype/config/gcp-ray.yml.j2 b/prototype/config/gcp-ray.yml.j2
index 694a011dcdd..dc0877c9e6b 100644
--- a/prototype/config/gcp-ray.yml.j2
+++ b/prototype/config/gcp-ray.yml.j2
@@ -87,6 +87,13 @@ file_mounts: {
 {%- endfor %}
 }
 
+{%- if initialization_commands is not none %}
+initialization_commands:
+{%- for cmd in initialization_commands %}
+  - {{cmd}}
+{%- endfor %}
+{%- endif %}
+
 # Patterns for files to exclude when running rsync up or rsync down
 # rsync_exclude:
 #   - "**/.git"
diff --git a/prototype/examples/root_file_mount.yaml b/prototype/examples/root_file_mount.yaml
new file mode 100644
index 00000000000..4500f90cf46
--- /dev/null
+++ b/prototype/examples/root_file_mount.yaml
@@ -0,0 +1,15 @@
+# Example: file_mounts destination paths that require root access
+#
+# Sky will use sudo to create destination paths and run 'chown' on them, so that
+# the 'run' command (logged in as non-root) can still access them.
+resources:
+  cloud: aws
+
+# <VM path> : <local path>
+file_mounts:
+  /setup.py: ./setup.py
+  /sky: sky
+
+run: |
+  ls -l /setup.py
+  ls -lH /sky
diff --git a/prototype/sky/backends/backend_utils.py b/prototype/sky/backends/backend_utils.py
index 38278b3f067..6b038973dab 100644
--- a/prototype/sky/backends/backend_utils.py
+++ b/prototype/sky/backends/backend_utils.py
@@ -28,6 +28,9 @@
 IP_ADDR_REGEX = r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}'
 SKY_LOGS_DIRECTORY = './sky_logs'
 
+# Do not use /tmp because it gets cleared on VM restart.
+_SKY_REMOTE_FILE_MOUNTS_DIR = '~/sky_file_mounts/'
+
 
 def _get_rel_path(path: str) -> str:
     cwd = os.getcwd()
@@ -52,6 +55,20 @@ def _fill_template(template_path: str,
     return output_path
 
 
+def _wrap_file_mount(remote: str) -> str:
+    """Prepends ~/<opaque dir>/ to a path to work around permission issues.
+
+    Examples:
+      /root/hello.txt -> ~/<opaque dir>/root/hello.txt
+      local.txt -> ~/<opaque dir>/local.txt
+
+    In initialization_commands of the ray autoscaler YAML, we can then create a
+    symlink to this wrapped path.
+    """
+    return os.path.join(_SKY_REMOTE_FILE_MOUNTS_DIR, remote.lstrip('/'))
+
+
+# TODO: too many things happening here - leaky abstraction. Refactor.
 def write_cluster_config(run_id: RunId,
                          task: task.Task,
                          cluster_config_template: str,
@@ -104,6 +121,25 @@ def write_cluster_config(run_id: RunId,
             f.write(codegen)
         setup_sh_path = f.name
 
+    # File mounts handling:
+    #  (1) in 'file_mounts' sections, add <prefix> to all target paths.
+    #  (2) then, create symlinks from '/root/file' to '<prefix>/root/file'.
+    # We need to do these since as of Ray 1.8, this is not supported natively
+    # (Docker works though, of course):
+    #  https://github.com/ray-project/ray/pull/9332
+    #  https://github.com/ray-project/ray/issues/9326
+    mounts = task.get_local_to_remote_file_mounts()
+    wrapped_file_mounts = {}
+    initialization_commands = []
+    if mounts is not None:
+        for remote, local in mounts.items():
+            wrapped_remote = _wrap_file_mount(remote)
+            wrapped_file_mounts[wrapped_remote] = local
+            # NOTE: the use of sudo and the assumption of ubuntu.
+            symlink = (f'sudo ln -sf {wrapped_remote} {remote} && '
+                       f'sudo chown ubuntu {remote}')
+            initialization_commands.append(symlink)
+
     yaml_path = _fill_template(
         cluster_config_template,
         dict(
@@ -115,7 +151,9 @@ def write_cluster_config(run_id: RunId,
                 'docker_image': task.docker_image,
                 'container_name': task.container_name,
                 'num_nodes': task.num_nodes,
-                'file_mounts': task.get_local_to_remote_file_mounts() or {},
+                # File mounts handling.
+                'file_mounts': wrapped_file_mounts,
+                'initialization_commands': initialization_commands or None,
                 # Region/zones.
                 'region': region,
                 'zones': ','.join(zones),
diff --git a/prototype/sky/backends/cloud_vm_ray_backend.py b/prototype/sky/backends/cloud_vm_ray_backend.py
index 41afeba9b43..3ca7c330e0f 100644
--- a/prototype/sky/backends/cloud_vm_ray_backend.py
+++ b/prototype/sky/backends/cloud_vm_ray_backend.py
@@ -321,7 +321,7 @@ def provision_with_retries(self, task: App, to_provision: Resources,
                         raise e
             cluster_config_file = config_dict['ray']
 
-            tail_cmd = f'tail -n100 -f {log_path}'
+            tail_cmd = f'tail -f {log_path}'
             logger.info(
                 f'To view progress: {Style.BRIGHT}{tail_cmd}{Style.RESET_ALL}')
             # Redirect stdout/err to the file and streaming (if stream_logs).


Code review comments:
diff_hunk: @@ -104,6 +121,25 @@ def write_cluster_config(run_id: RunId,
             f.write(codegen)
         setup_sh_path = f.name
 
+    # File mounts handling:
+    #  (1) in 'file_mounts' sections, add <prefix> to all target paths.
+    #  (2) then, create symlinks from '/root/file' to '<prefix>/root/file'.
+    # We need to do these since as of Ray 1.8, this is not supported natively
+    # (Docker works though, of course):
+    #  https://github.com/ray-project/ray/pull/9332
+    #  https://github.com/ray-project/ray/issues/9326
+    mounts = task.get_local_to_remote_file_mounts()
+    wrapped_file_mounts = {}
+    initialization_commands = []
+    if mounts is not None:
+        for remote, local in mounts.items():
+            wrapped_remote = _wrap_file_mount(remote)
+            wrapped_file_mounts[wrapped_remote] = local
+            # NOTE: the use of sudo and the assumption of ubuntu.
+            symlink = (f'sudo ln -sf {wrapped_remote} {remote} && '
+                       f'sudo chown ubuntu {remote}')
path: prototype/sky/backends/backend_utils.py
text: @user1:
I'm not super familiar with ray autoscaler works, but if it uses the same user for running initialization_commands, should we change this to 'sudo chown $USER {remote}'?

diff_hunk: @@ -28,6 +28,9 @@
 IP_ADDR_REGEX = r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}'
 SKY_LOGS_DIRECTORY = './sky_logs'
 
+# Do not use /tmp because it gets cleared on VM restart.
+_SKY_REMOTE_FILE_MOUNTS_DIR = '~/sky_file_mounts/'
path: prototype/sky/backends/backend_utils.py
text: @user1:
Considering there's state we'll be maintaining on remote VMs, should we start using a `.sky` directory in the home dir? This would then be something like `~/.sky/file_mounts/`

@author:
Great point!  How about a `~/_sky` or `~/_sky_`?  Reasons: (1) we may want to ease debugging, `.sky` won't show on plain `ls`. (2) `~/sky` could work too, but may be "unprotected" and collide with user paths.  Wdyt?

@user1:
Good point about `.sky` being hidden. `_sky` sounds good, lets go with that

@author:
Decided to go with ~/.sky to be consistent with tons of other tools.