Repository: skypilot-org/skypilot
Pull request: Enhance the logging info for pending task
Closes #892.

Problem statement:
[Job] Pending Jobs Hanging
Suppose the user launches a cluster of 4 V100 and submits two jobs with 2 V100s.  The cluster is now full.

Submitting a 3rd job with 2 V100s will be placed as PENDING for the Ray autoscaler. However, `sky launch -c asdf job.yaml` will hang, since the job is waiting for other two jobs to finish.

Hints from issue discussion:
This is expected?  User can pass -d to detach.

On Thu, Jun 2, 2022 at 13:04 Michael Luo ***@***.***> wrote:

> Suppose the user launches a cluster of 4 V100 and submits two jobs with 2
> V100s. The cluster is now full.
>
> Submitting a 3rd job with 2 V100s will be placed as PENDING for the Ray
> autoscaler. However, sky launch -c asdf job.yaml will hang, since the job
> is waiting for other two jobs to finish.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/sky-proj/sky/issues/892>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAEQWHVSTQFAP3EGQKDWBB3VNEHT5ANCNFSM5XWCIGTA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>

Yes, but users aren't aware of it the first time they encounter a pending job. Sky should detect that a job is pending and close tail logs in that ccase.
That complicates the logic, and I can see some race condition in that.  @Michaelvll thought?
I would prefer to show the PENDING information as a `SKY INFO` when the logs are tailed if the current behavior really confuses the user.

Commit message: Merge branch 'master' of github.com:concretevitamin/sky-experiments into enhance-log-info

Patch to review:
diff --git a/sky/backends/cloud_vm_ray_backend.py b/sky/backends/cloud_vm_ray_backend.py
index afa4313932c..39e64122ae3 100644
--- a/sky/backends/cloud_vm_ray_backend.py
+++ b/sky/backends/cloud_vm_ray_backend.py
@@ -251,7 +251,9 @@ def add_gang_scheduling_placement_group(
                 pg = ray_util.placement_group({json.dumps(bundles)}, {pack_mode!r})
                 plural = 's' if {num_nodes} > 1 else ''
                 node_str = f'{num_nodes} node' + plural + '.'
-                print('SKY INFO: Reserving task slots on ' + node_str,
+                print(('SKY INFO: Waiting for task resources on ' + node_str + 
+                        'This will keep blocked if the cluster is full. '
+                        'Use Ctrl-C to exit log streaming (task will not be killed)'),
                       file=sys.stderr,
                       flush=True)
                 # FIXME: This will print the error message from autoscaler if


Code review comments:
diff_hunk: @@ -251,7 +251,9 @@ def add_gang_scheduling_placement_group(
                 pg = ray_util.placement_group({json.dumps(bundles)}, {pack_mode!r})
                 plural = 's' if {num_nodes} > 1 else ''
                 node_str = f'{num_nodes} node' + plural + '.'
-                print('SKY INFO: Reserving task slots on ' + node_str,
+                print(('SKY INFO: Waiting for task resources on ' + node_str + 
+                        'This will keep blocked if the cluster is full. '
path: sky/backends/cloud_vm_ray_backend.py
text: @user1:
nit: "This will block ..."? 

@author:
Fixed. Thanks!