Repository: skypilot-org/skypilot
Pull request: [Core] Avoid PENDING job to be set to FAILED and speed up job scheduling
<!-- Describe the changes in this PR -->

Fixes #4263 

It also speed up the job scheduling by avoid update status for all the jobs:
Previously, gap between jobs to be scheduled is ~4-5s, ie with 60s duration of a task, there will be 12 jobs running in parallel (the additional RUNNING (with more than 1 min job duration) ones below are invoking the scheduling step for jobs which takes a long time waiting for the job scheduler to finish).
```
221  sky-cmd  11 mins ago  -               -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-19-30-10-870719  
220  sky-cmd  11 mins ago  -               -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-19-30-09-092916  
219  sky-cmd  11 mins ago  -               -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-19-30-07-104709  
218  sky-cmd  11 mins ago  -               -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-19-30-06-950815  
217  sky-cmd  11 mins ago  -               -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-19-30-06-835127  
216  sky-cmd  11 mins ago  -               -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-19-30-04-966880  
215  sky-cmd  11 mins ago  a few secs ago  1s        1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-30-04-499591  
214  sky-cmd  11 mins ago  a few secs ago  5s        1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-30-03-997782  
213  sky-cmd  11 mins ago  a few secs ago  9s        1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-30-03-401201  
212  sky-cmd  11 mins ago  14 secs ago     14s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-30-02-683537  
211  sky-cmd  11 mins ago  18 secs ago     18s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-58-437107  
210  sky-cmd  11 mins ago  22 secs ago     22s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-58-121545  
209  sky-cmd  11 mins ago  27 secs ago     27s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-58-030503  
208  sky-cmd  11 mins ago  30 secs ago     30s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-55-595568  
207  sky-cmd  11 mins ago  36 secs ago     36s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-55-308832  
206  sky-cmd  11 mins ago  41 secs ago     41s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-54-752160  
205  sky-cmd  11 mins ago  46 secs ago     46s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-54-840566  
204  sky-cmd  11 mins ago  52 secs ago     52s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-54-647147  
203  sky-cmd  11 mins ago  56 secs ago     56s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-51-826498  
202  sky-cmd  11 mins ago  1 min ago       1m        1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-51-525987  
201  sky-cmd  11 mins ago  1 min ago       1m 4s     1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-51-529238  
200  sky-cmd  11 mins ago  1 min ago       1m 12s    1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-49-392753  
199  sky-cmd  11 mins ago  1 min ago       1m 16s    1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-48-211998  
198  sky-cmd  11 mins ago  1 min ago       1m 20s    1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-47-018499  
197  sky-cmd  11 mins ago  1 min ago       1m 24s    1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-46-732420  
196  sky-cmd  11 mins ago  1 min ago       1m 29s    1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-46-527903  
195  sky-cmd  11 mins ago  1 min ago       1m 34s    1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-44-218424  
194  sky-cmd  11 mins ago  1 min ago       1m 40s    1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-43-986796  
193  sky-cmd  11 mins ago  1 min ago       1m 44s    1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-19-29-44-039565  
192  sky-cmd  11 mins ago  1 min ago       1m 45s    1x[CPU:1+]  SUCCEEDED  ~/sky_logs/sky-2024-11-05-19-29-42-571895 
```

Now, it takes ~2-3 seconds for each job to be submitted, so there can be 21 parallel jobs, and there are no jobs with duration significantly longer than 1 minute.
```
339   sky-cmd  11 mins ago  -               -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-23-31-46-294901  
338   sky-cmd  11 mins ago  a few secs ago  2s        1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-46-151382  
337   sky-cmd  11 mins ago  a few secs ago  5s        1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-44-053912  
336   sky-cmd  11 mins ago  a few secs ago  8s        1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-41-961098  
335   sky-cmd  11 mins ago  11 secs ago     11s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-41-919819  
334   sky-cmd  11 mins ago  14 secs ago     14s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-41-611173  
333   sky-cmd  11 mins ago  16 secs ago     16s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-41-550143  
332   sky-cmd  11 mins ago  19 secs ago     19s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-41-136605  
331   sky-cmd  11 mins ago  22 secs ago     22s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-40-268279  
330   sky-cmd  11 mins ago  25 secs ago     25s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-40-298456  
329   sky-cmd  11 mins ago  28 secs ago     28s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-38-036070  
328   sky-cmd  11 mins ago  30 secs ago     30s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-35-625810  
327   sky-cmd  11 mins ago  33 secs ago     33s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-35-368378  
326   sky-cmd  11 mins ago  36 secs ago     36s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-35-547889  
325   sky-cmd  11 mins ago  39 secs ago     39s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-35-271033  
324   sky-cmd  11 mins ago  41 secs ago     41s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-34-871366  
323   sky-cmd  11 mins ago  44 secs ago     44s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-34-202901  
322   sky-cmd  11 mins ago  47 secs ago     47s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-34-022552  
321   sky-cmd  11 mins ago  50 secs ago     50s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-31-843038  
320   sky-cmd  11 mins ago  53 secs ago     53s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-29-105083  
319   sky-cmd  11 mins ago  56 secs ago     56s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-29-240834  
318   sky-cmd  11 mins ago  58 secs ago     58s       1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-23-31-28-474097  
317   sky-cmd  11 mins ago  1 min ago       1m        1x[CPU:1+]  SUCCEEDED  ~/sky_logs/sky-2024-11-05-23-31-28-665341
```



<!-- Describe the tests ran -->
<!-- Unit tests (tests/test_*.py) are part of GitHub CI; below are tests that launch on the cloud. -->

Tested (run the relevant ones):

- [ ] Code formatting: `bash format.sh`
- [x] Any manual or new tests for this PR (please specify below)
  - [x] Reproducible script in #4263 
- [x] All smoke tests: `pytest tests/test_smoke.py --aws` 
- [ ] Relevant individual smoke tests: `pytest tests/test_smoke.py::test_fill_in_the_name` 
  - [x] `pytest tests/test_smoke.py::test_multi_echo`
- [x] Backward compatibility tests: `conda deactivate; bash -i tests/backward_compatibility_tests.sh`


Problem statement:
[Core] Submitting 1000 jobs to a cluster
<!-- Describe the bug report / feature request here -->


`sky launch -c test-many-jobs --cloud aws --cpus 16 --region us-east-1`
and submit jobs:
```
import subprocess
from multiprocessing.pool import ThreadPool

def run_task(task):
    print(f'Running task {task}')
    subprocess.run(
        f'sky exec test-many-jobs -d "echo hi {task}; sleep 60"', shell=True
    )

with ThreadPool(8) as pool:
    pool.map(run_task, range(1000))

```

There are some pending jobs directly in `FAILED` status:

<details>
<summary>sky job queue</summary>

```
397   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-36-653878  
396   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-36-241842  
395   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-35-938395  
394   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-35-242534  
393   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-34-388982  
392   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-33-484291  
391   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-33-343155  
390   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-30-894991  
389   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-30-271563  
388   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-29-496423  
387   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-29-210465  
386   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-29-456516  
385   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-28-390933  
384   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-27-188816  
383   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-27-098285  
382   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-24-541945  
381   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-24-229395  
380   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-23-402925  
379   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-23-071930  
378   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-23-145018  
377   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-22-539985  
376   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-21-025510  
375   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-20-997171  
374   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-18-671169  
373   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-17-896233  
372   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-17-097363  
371   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-16-872947  
370   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-17-040533  
369   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-16-321652  
368   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-14-689782  
367   sky-cmd  47 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-14-612180  
366   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-12-561881  
365   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-11-489980  
364   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-11-074665  
363   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  FAILED     ~/sky_logs/sky-2024-11-05-08-02-10-755273  
362   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-10-400760  
361   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-10-216905  
360   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-08-709074  
359   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-08-474316  
358   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-06-822676  
357   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-04-858041  
356   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-04-921609  
355   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-04-662728  
354   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-03-636612  
353   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-03-776459  
352   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-01-931069  
351   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-01-599564  
350   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-02-00-654291  
349   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-57-863715  
348   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-57-476013  
347   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-57-198596  
346   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-56-759002  
345   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-56-728631  
344   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-55-006414  
343   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-55-220696  
342   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-54-064863  
341   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-52-218586  
340   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-51-040224  
339   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-50-935741  
338   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-50-288325  
337   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-50-478773  
336   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-49-099053  
335   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-48-550194  
334   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-48-217828  
333   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-46-255106  
332   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-44-558566  
331   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-44-467170  
330   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-43-892493  
329   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-43-716273  
328   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-42-848585  
327   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-41-999266  
326   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-42-235626  
325   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-40-082665  
324   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-38-614665  
323   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-38-487463  
322   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-38-032037  
321   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-37-467258  
320   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-36-202257  
319   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-35-780035  
318   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-35-966845  
317   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-34-124504  
316   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-32-308747  
315   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-32-096878  
314   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-31-733177  
313   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-31-403574  
312   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-30-537735  
311   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-29-387918  
310   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-29-582632  
309   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-28-219573  
308   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-26-278180  
307   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-25-500778  
306   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-25-306906  
305   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-25-386088  
304   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-24-443275  
303   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-23-497353  
302   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-23-165759  
301   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-21-960793  
300   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-20-041373  
299   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-19-224309  
298   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-19-441115  
297   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-18-773858  
296   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-18-322541  
295   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-17-440484  
294   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-16-785337  
293   sky-cmd  48 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-15-770487  
292   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-13-659089  
291   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-13-018024  
290   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-12-633196  
289   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-12-947045  
288   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-12-198609  
287   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-11-250759  
286   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-11-006672  
285   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-09-793409  
284   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-07-327897  
283   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-06-646837  
282   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-05-628923  
281   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-05-771241  
280   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-05-473983  
279   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-04-536272  
278   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-04-354728  
277   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-03-428077  
276   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-01-00-486087  
275   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-59-334567  
274   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-59-151814  
273   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-58-950464  
272   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-58-310687  
271   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-57-206339  
270   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-56-941714  
269   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-56-535521  
268   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-54-329514  
267   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-53-071108  
266   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-52-941368  
265   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-52-881440  
264   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-52-200498  
263   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-50-945052  
262   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-50-920789  
261   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-50-449763  
260   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-48-121061  
259   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-46-976192  
258   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-46-598611  
257   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-46-553419  
256   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-46-181387  
255   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-45-163721  
254   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-44-567216  
253   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-44-087470  
252   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-42-091929  
251   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-41-113196  
250   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-40-579688  
249   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-40-087820  
248   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-40-248555  
247   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-38-861945  
246   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-38-294149  
245   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-38-258045  
244   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-35-599548  
243   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-34-534367  
242   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-34-526143  
241   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-34-189236  
240   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-33-647800  
239   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-32-618731  
238   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-31-954582  
237   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-31-920088  
236   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-29-534488  
235   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-28-269088  
234   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-27-661199  
233   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-27-713331  
232   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-27-701794  
231   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-26-492482  
230   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-25-519140  
229   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-25-408865  
228   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-23-518706  
227   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-22-218402  
226   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-21-805968  
225   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-21-519246  
224   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-21-255874  
223   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-20-310804  
222   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-19-435243  
221   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-19-274018  
220   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-17-704161  
219   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-16-072318  
218   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-15-431482  
217   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-15-276972  
216   sky-cmd  49 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-14-791237  
215   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-14-053555  
214   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-13-367657  
213   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-13-333595  
212   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-11-804236  
211   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-09-733314  
210   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-09-401792  
209   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-09-036496  
208   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-08-437905  
207   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-07-792564  
206   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-07-012027  
205   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-06-712460  
204   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-05-829213  
203   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-02-724472  
202   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-01-926897  
201   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-01-949462  
200   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-01-155412  
199   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-08-00-00-635174  
198   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-07-59-59-770938  
197   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-07-59-59-791625  
196   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-07-59-58-527490  
195   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-07-59-56-311420  
194   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-07-59-55-741381  
193   sky-cmd  50 mins ago  -            -         1x[CPU:1+]  PENDING    ~/sky_logs/sky-2024-11-05-07-59-55-649851  
192   sky-cmd  50 mins ago  1 min ago    1m 3s     1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-07-59-54-725529  
191   sky-cmd  50 mins ago  1 min ago    1m 6s     1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-07-59-54-654723  
190   sky-cmd  50 mins ago  1 min ago    1m 9s     1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-07-59-53-962026  
189   sky-cmd  50 mins ago  1 min ago    1m 12s    1x[CPU:1+]  RUNNING    ~/sky_logs/sky-2024-11-05-07-59-53-403442  
```

</details>



<!-- If relevant, fill in versioning info to help us troubleshoot -->
_Version & Commit info:_
* `sky -v`: PLEASE_FILL_IN
* `sky -c`: PLEASE_FILL_IN

Hints from issue discussion:
https://gist.github.com/Michaelvll/9325ff98e6bd2b454dfd661fb9c0ebab

Commit message: Merge branch 'master' of github.com:skypilot-org/skypilot into longer-wait-for-pending-to-failed

Patch to review:
diff --git a/examples/multi_echo.py b/examples/multi_echo.py
index a9fe49fad2d..7f310a4bb6b 100644
--- a/examples/multi_echo.py
+++ b/examples/multi_echo.py
@@ -31,13 +31,13 @@ def run(cluster: Optional[str] = None, cloud: Optional[str] = None):
 
     # Submit multiple tasks in parallel to trigger queueing behaviors.
     def _exec(i):
-        task = sky.Task(run=f'echo {i}; sleep 5')
-        resources = sky.Resources(accelerators={'T4': 0.5})
+        task = sky.Task(run=f'echo {i}; sleep 60')
+        resources = sky.Resources(accelerators={'T4': 0.05})
         task.set_resources(resources)
         sky.exec(task, cluster_name=cluster, detach_run=True)
 
     with pool.ThreadPool(8) as p:
-        list(p.imap(_exec, range(32)))
+        list(p.imap(_exec, range(150)))
 
 
 if __name__ == '__main__':
diff --git a/sky/skylet/events.py b/sky/skylet/events.py
index b6e99707dab..10d598a45a8 100644
--- a/sky/skylet/events.py
+++ b/sky/skylet/events.py
@@ -65,6 +65,7 @@ class JobSchedulerEvent(SkyletEvent):
 
     def _run(self):
         job_lib.scheduler.schedule_step(force_update_jobs=True)
+        logger.info('Triggerred job status update and scheduling.')
 
 
 class ManagedJobUpdateEvent(SkyletEvent):
diff --git a/sky/skylet/job_lib.py b/sky/skylet/job_lib.py
index 12d42d8c79c..012b2225c25 100644
--- a/sky/skylet/job_lib.py
+++ b/sky/skylet/job_lib.py
@@ -182,13 +182,18 @@ def _run_job(self, job_id: int, run_cmd: str):
 
     def schedule_step(self, force_update_jobs: bool = False) -> None:
         jobs = self._get_jobs()
-        if len(jobs) > 0 or force_update_jobs:
+        if force_update_jobs:
             update_status()
         # TODO(zhwu, mraheja): One optimization can be allowing more than one
         # job staying in the pending state after ray job submit, so that to be
         # faster to schedule a large amount of jobs.
         for job_id, run_cmd, submit, created_time in jobs:
             with filelock.FileLock(_get_lock_path(job_id)):
+                # We don't have to refresh the job status before checking, as
+                # the job status will only be stale in rare cases where ray job
+                # crashes; or the job stays in INIT state for a long time.
+                # In those cases, the periodic JobSchedulerEvent event will
+                # update the job status every 300 seconds.
                 status = get_status_no_lock(job_id)
                 if (status not in _PRE_RESOURCE_STATUSES or
                         created_time < psutil.boot_time()):
@@ -537,22 +542,9 @@ def update_job_status(job_ids: List[int],
     if len(job_ids) == 0:
         return []
 
-    # TODO: if too slow, directly query against redis.
     ray_job_ids = [make_ray_job_id(job_id) for job_id in job_ids]
-
     job_client = _create_ray_job_submission_client()
 
-    # In ray 2.4.0, job_client.list_jobs returns a list of JobDetails,
-    # which contains the job status (str) and submission_id (str).
-    ray_job_query_time = time.time()
-    job_detail_lists: List['ray_pydantic.JobDetails'] = job_client.list_jobs()
-
-    job_details = {}
-    ray_job_ids_set = set(ray_job_ids)
-    for job_detail in job_detail_lists:
-        if job_detail.submission_id in ray_job_ids_set:
-            job_details[job_detail.submission_id] = job_detail
-
     statuses = []
     for job_id, ray_job_id in zip(job_ids, ray_job_ids):
         # Per-job status lock is required because between the job status
@@ -560,9 +552,34 @@ def update_job_status(job_ids: List[int],
         # can be modified by the generated ray program.
         with filelock.FileLock(_get_lock_path(job_id)):
             status = None
-            if ray_job_id in job_details:
-                ray_status = job_details[ray_job_id].status
-                status = _RAY_TO_JOB_STATUS_MAP[ray_status]
+            job_record = _get_jobs_by_ids([job_id])[0]
+            original_status = job_record['status']
+            job_submitted_at = job_record['submitted_at']
+
+            ray_job_query_time = time.time()
+            if original_status == JobStatus.INIT:
+                if (job_submitted_at >= psutil.boot_time() and job_submitted_at
+                        >= ray_job_query_time - _PENDING_SUBMIT_GRACE_PERIOD):
+                    # The job id is reserved, but the job is not submitted yet.
+                    # We should keep it in INIT.
+                    status = JobStatus.INIT
+                else:
+                    # The job id is reserved, but stale.
+                    status = JobStatus.FAILED
+
+            try:
+                # Querying status within the lock is safer than querying
+                # outside, as it avoids the race condition when job table is
+                # updated after the ray job status query.
+                # Also, getting per-job status is faster than querying all jobs,
+                # when there are significant number of finished jobs.
+                # TODO: if too slow, directly query against redis.
+                ray_job_status = job_client.get_job_status(ray_job_id)
+                status = _RAY_TO_JOB_STATUS_MAP[ray_job_status.value]
+            except RuntimeError:
+                # Job not found.
+                pass
+
             pending_job = _get_pending_job(job_id)
             if pending_job is not None:
                 if pending_job['created_time'] < psutil.boot_time():
@@ -583,7 +600,6 @@ def update_job_status(job_ids: List[int],
                     # as stale.
                     status = JobStatus.PENDING
 
-            original_status = get_status_no_lock(job_id)
             assert original_status is not None, (job_id, status)
             if status is None:
                 status = original_status
@@ -611,6 +627,7 @@ def update_job_status(job_ids: List[int],
                 # DB) would already have that value. So we take the max here to
                 # keep it at later status.
                 status = max(status, original_status)
+                assert status is not None, (job_id, status, original_status)
                 if status != original_status:  # Prevents redundant update.
                     _set_status_no_lock(job_id, status)
                     if not silent:
diff --git a/tests/test_smoke.py b/tests/test_smoke.py
index cdfd9dfc7cb..3cd065fe5a0 100644
--- a/tests/test_smoke.py
+++ b/tests/test_smoke.py
@@ -1813,7 +1813,18 @@ def test_multi_echo(generic_cloud: str):
         'multi_echo',
         [
             f'python examples/multi_echo.py {name} {generic_cloud}',
-            'sleep 120',
+            f's=$(sky queue {name}); echo "$s"; echo; echo; echo "$s" | grep "FAILED" && exit 1 || true',
+            'sleep 10',
+            f's=$(sky queue {name}); echo "$s"; echo; echo; echo "$s" | grep "FAILED" && exit 1 || true',
+            'sleep 30',
+            f's=$(sky queue {name}); echo "$s"; echo; echo; echo "$s" | grep "FAILED" && exit 1 || true',
+            'sleep 30',
+            # Make sure that our job scheduler is fast enough to have at least
+            # 10 RUNNING jobs in parallel.
+            f's=$(sky queue {name}); echo "$s"; echo; echo; echo "$s" | grep "RUNNING" | wc -l | awk \'{{if ($1 < 10) exit 1}}\'',
+            'sleep 30',
+            f's=$(sky queue {name}); echo "$s"; echo; echo; echo "$s" | grep "FAILED" && exit 1 || true',
+            f'until sky logs {name} 32 --status; do echo "Waiting for job 32 to finish..."; sleep 1; done',
         ] +
         # Ensure jobs succeeded.
         [f'sky logs {name} {i + 1} --status' for i in range(32)] +


Code review comments:
diff_hunk: @@ -537,32 +542,44 @@ def update_job_status(job_ids: List[int],
     if len(job_ids) == 0:
         return []
 
-    # TODO: if too slow, directly query against redis.
     ray_job_ids = [make_ray_job_id(job_id) for job_id in job_ids]
-
     job_client = _create_ray_job_submission_client()
 
-    # In ray 2.4.0, job_client.list_jobs returns a list of JobDetails,
-    # which contains the job status (str) and submission_id (str).
-    ray_job_query_time = time.time()
-    job_detail_lists: List['ray_pydantic.JobDetails'] = job_client.list_jobs()
-
-    job_details = {}
-    ray_job_ids_set = set(ray_job_ids)
-    for job_detail in job_detail_lists:
-        if job_detail.submission_id in ray_job_ids_set:
-            job_details[job_detail.submission_id] = job_detail
-
     statuses = []
     for job_id, ray_job_id in zip(job_ids, ray_job_ids):
         # Per-job status lock is required because between the job status
         # query and the job status update, the job status in the databse
         # can be modified by the generated ray program.
         with filelock.FileLock(_get_lock_path(job_id)):
             status = None
-            if ray_job_id in job_details:
-                ray_status = job_details[ray_job_id].status
-                status = _RAY_TO_JOB_STATUS_MAP[ray_status]
+            job_record = _get_jobs_by_ids([job_id])[0]
+            original_status = job_record['status']
+            job_submitted_at = job_record['submitted_at']
+
+            ray_job_query_time = time.time()
+            if original_status == JobStatus.INIT:
+                if (job_submitted_at >= psutil.boot_time() and job_submitted_at
+                        >= ray_job_query_time - _PENDING_SUBMIT_GRACE_PERIOD):
+                    # The job id is reserved, but the job is not submitted yet.
+                    # We should keep it in INIT.
+                    status = JobStatus.INIT
+                else:
+                    # The job id is reserved, but stale.
+                    status = JobStatus.FAILED
path: sky/skylet/job_lib.py
text: @user1:
Why is this set to FAILED? I see it only gets used in the ray job status query below doesn't return any job.

@author:
We always immediately submit job after the job id is allocated, i.e. INIT -> PENDING, if a job stays in `INIT` for too long, it is likely the job submission process was killed before the job is submitted. We should set it to FAILED then.

@user1:
Can use this in the comment.