Repository: skypilot-org/skypilot
Pull request: Timeout for multi-node launching
This is to resolve #502. We will timeout the gang scheduling if we fail to launch at least 1 node in 30 seconds when there is any node that should be launched (launch means: provision without any setup). The fail case relates to quota limit or resources availability limit. We can further reduce the `timeout` to simulate the AWS UI behavior as mentioned in #502 (head node still needs to be provisioned first), but it will disable the retry for resources availability.

This also fixes a problem in #413, where `ray exec 'ray status'` fails, by caching the head_ip by ourselves.

PS: I changed the `upscaling_speed` to `num_nodes - 1`, so that ray will try to launch all the worker nodes when the heads are ready (increasing the parallelism), instead of `upscaling_speed * ready_nodes` each time. (Side effect: it will launch as many instances as possible and then teardown them, if the resources is not enough).

Tested (the following cluster `scale-n` indicates cluster with `num_nodes=n`:
- [x] `sky launch -c scale-3 examples/minimal.yaml` (it fails from the first region in 4 min (start from `sky launch`) due to the quota limit, and then successfully launched on the second region in min).
- [x] `sky launch -c scale-1000 examples/minimal.yaml` (it fails all the aws regions in 12 min)
- [x] `sky launch -c scale-18 examples/minimal.yaml` (Daniel's usecase, it takes 10 min (4min of it fails from the first region) to provision the instances)
  ```
  sky launch -c scale-18 --cloud aws examples/minimal.yaml  28.11s user 8.40s system 5% cpu 10:13.99 total
  ```
- [x] `sky launch -c scale-3 --cloud gcp examples/minimal.yaml` 
- [x] `sky launch -c scale-3 --cloud azure examples/minimal.yaml` 
- [x] `sky launch -c scale-1000 --cloud gcp examples/minimal.yaml` (timeout all the region/zones within 1:40:00)
- [x] `sky launch -c scale-1000 --cloud azure examples/minimal.yaml` (timeout all the region/zones within 4:15:00, the mainly reason is that launching and terminating on azure is very slow)
- [x] `examples/run_smoke_tests.sh` (Tested with azure_start_stop disabled)

Note: the time for scale-1000 is still not good especially for azure and gcp, but at least it is better than before, which will hang forever.

Future TODO: check the quota before we call the `ray up` so we can get rid of the long waiting time for the quota limit.

Problem statement:
[Azure] `sky launch azure-start-stop.yaml` fails due to dependency 
I failed to launch two azure nodes with the following error during launching the head node.
```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
fbprophet 0.7.1 requires cmdstanpy==0.9.5, which is not installed.
fbprophet 0.7.1 requires setuptools-git>=1.2, which is not installed.
azure-storage-file 2.1.0 requires azure-storage-common~=2.1, but you have azure-storage-common 1.4.2 which is incompatible.
```

That error causes the problem when ray tries to launch the worker node, and `ray status` generates the following error.
```
I 02-22 14:38:39 backend_utils.py:547] Fetched IP: 20.110.200.1
I 02-22 14:38:39 backend_utils.py:547] 
I 02-22 14:38:57 backend_utils.py:547] No cluster status.
I 02-22 14:38:57 backend_utils.py:547] The autoscaler failed with the following error:
I 02-22 14:38:57 backend_utils.py:547] Traceback (most recent call last):
I 02-22 14:38:57 backend_utils.py:547]   File "/anaconda/envs/py38_default/lib/python3.8/site-packages/ray/autoscaler/_private/monitor.py", line 435, in run
I 02-22 14:38:57 backend_utils.py:547]     self._run()
I 02-22 14:38:57 backend_utils.py:547]   File "/anaconda/envs/py38_default/lib/python3.8/site-packages/ray/autoscaler/_private/monitor.py", line 325, in _run
I 02-22 14:38:57 backend_utils.py:547]     status["autoscaler_report"] = asdict(self.autoscaler.summary())
I 02-22 14:38:57 backend_utils.py:547]   File "/anaconda/envs/py38_default/lib/python3.8/site-packages/ray/autoscaler/_private/autoscaler.py", line 1195, in summary
I 02-22 14:38:57 backend_utils.py:547]     for node_id in self.non_terminated_nodes.all_node_ids:
I 02-22 14:38:57 backend_utils.py:547] AttributeError: 'NoneType' object has no attribute 'all_node_ids'
I 02-22 14:38:57 backend_utils.py:547] 
I 02-22 14:38:57 backend_utils.py:547] Loaded cached provider configuration
I 02-22 14:38:57 backend_utils.py:547] If you experience issues with the cloud provider, try re-running the command with --no-config-cache.
```
@suquark Do you know why this happens?

Launching multiple nodes will wait the worker node forever when quota exceeds
I have 3 m4.2xlarge instances running on us-east-1, and my quota for this instance on that region is 32 vCPUs (4 m4.2xlarge). When I tried to `sky launch -c env ./examples/env_check.yaml` (2 nodes), it keeps showing the following `ray status` due to the quota exceeding for the worker node. I am going to make multi-node timeout PR happen to mitigate this problem.

```
=====
I 03-07 15:54:39 backend_utils.py:503] Node status
I 03-07 15:54:39 backend_utils.py:503] ---------------------------------------------------------------
I 03-07 15:54:39 backend_utils.py:503] Healthy:
I 03-07 15:54:39 backend_utils.py:503]  1 ray.head.default
I 03-07 15:54:39 backend_utils.py:503] Pending:
I 03-07 15:54:39 backend_utils.py:503]  ray.worker.default, 1 launching
I 03-07 15:54:39 backend_utils.py:503] Recent failures:
I 03-07 15:54:39 backend_utils.py:503]  (no failures)
I 03-07 15:54:39 backend_utils.py:503] 
I 03-07 15:54:39 backend_utils.py:503] Resources
I 03-07 15:54:39 backend_utils.py:503] ---------------------------------------------------------------
I 03-07 15:54:39 backend_utils.py:503] Usage:
I 03-07 15:54:39 backend_utils.py:503]  0.0/8.0 CPU
I 03-07 15:54:39 backend_utils.py:503]  0.00/18.612 GiB memory
I 03-07 15:54:39 backend_utils.py:503]  0.00/9.306 GiB object_store_memory
I 03-07 15:54:39 backend_utils.py:503] 
I 03-07 15:54:39 backend_utils.py:503] Demands:
I 03-07 15:54:39 backend_utils.py:503]  (no resource demands)
I 03-07 15:54:39 backend_utils.py:503] Loaded cached provider configuration
I 03-07 15:54:39 backend_utils.py:503] If you experience issues with the cloud provider, try re-running the command with --no-config-cache.
I 03-07 15:54:39 backend_utils.py:503] Fetched IP: 3.90.43.174
I 03-07 15:54:39 backend_utils.py:503] 
```

Hints from issue discussion:
+1, having the same issue. Here's the complete log, if that's any help:

```
(base) romilb@romilbx1yoga:/mnt/d/Romil/Berkeley/Research/sky-experiments$ sky launch -c azure-start-stop ./examples/azure_start_stop.yaml
Task from YAML spec: ./examples/azure_start_stop.yaml
Running task on cluster azure-start-stop...
I 02-23 00:52:44 optimizer.py:213] Defaulting estimated time to 1 hr. Call Task.set_time_estimator() to override.
I 02-23 00:52:47 optimizer.py:323] Optimizer - plan minimizing cost (~$0.9):
I 02-23 00:52:47 optimizer.py:337]
I 02-23 00:52:47 optimizer.py:337] TASK              BEST_RESOURCE
I 02-23 00:52:47 optimizer.py:337] azure-start-stop  2x Azure(Standard_D8_v4)
I 02-23 00:52:47 optimizer.py:337]
I 02-23 00:52:47 cloud_vm_ray_backend.py:1002] Creating a new cluster: "azure-start-stop" [2x Azure(Standard_D8_v4)].
I 02-23 00:52:47 cloud_vm_ray_backend.py:1002] Tip: to reuse an existing cluster, specify --cluster-name (-c) in the CLI or use sky.launch(.., cluster_name=..) in the Python API. Run `sky status` to see ex
isting clusters.
I 02-23 00:52:47 cloud_vm_ray_backend.py:654] To view detailed progress: tail -n100 -f sky_logs/sky-2022-02-23-00-52-44-998243/provision.log
I 02-23 00:52:47 cloud_vm_ray_backend.py:664]
I 02-23 00:52:47 cloud_vm_ray_backend.py:664] Launching on Azure eastus ()
W 02-23 00:53:55 cloud_vm_ray_backend.py:503] Got error(s) in eastus:
W 02-23 00:53:55 cloud_vm_ray_backend.py:505]   azure.core.exceptions.HttpResponseError: (InvalidTemplateDeployment) The template deployment 'ray-vm-ray-azure-start-stop-head' is not valid according to the
 validation procedure. The tracking id is '98e15b82-b3ee-483f-b8e3-82c807c67079'. See inner errors for details.
W 02-23 00:53:55 cloud_vm_ray_backend.py:505]   Code: InvalidTemplateDeployment
W 02-23 00:53:55 cloud_vm_ray_backend.py:505]   Exception Details:      (ResourceCountExceedsLimitDueToTemplate) Subscription aa86df77-e703-453e-b2f4-955c3b33e534 has a quota of 100 for resources of type P
ublicIpAddress with sku Basic. Subscription currently has 100 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for
resource type PublicIpAddress
I 02-23 00:53:56 cloud_vm_ray_backend.py:664]
I 02-23 00:53:56 cloud_vm_ray_backend.py:664] Launching on Azure eastus2 ()
Shared connection to 20.65.39.229 closed.
To run a command as administrator (user "root"), use "sudo <command>".
Shared connection to 20.65.39.229 closed.
See "man sudo_root" for details.

To run a command as administrator (user "root"), use "sudo <command>".
Shared connection to 20.65.39.229 closed.
See "man sudo_root" for details.

To run a command as administrator (user "root"), use "sudo <command>".
Shared connection to 20.65.39.229 closed.
See "man sudo_root" for details.

To run a command as administrator (user "root"), use "sudo <command>".
Shared connection to 20.65.39.229 closed.
See "man sudo_root" for details.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

Collecting ray[default]==1.10.0
  Downloading ray-1.10.0-cp38-cp38-manylinux2014_x86_64.whl (59.3 MB)
     |████████████████████████████████| 59.3 MB 34 kB/s
Requirement already satisfied: grpcio>=1.28.1 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]==1.10.0) (1.41.1)
Requirement already satisfied: pyyaml in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]==1.10.0) (6.0)
Collecting redis>=3.5.0
  Downloading redis-4.1.4-py3-none-any.whl (175 kB)
     |████████████████████████████████| 175 kB 107.7 MB/s
Collecting filelock
  Downloading filelock-3.6.0-py3-none-any.whl (10.0 kB)
Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]==1.10.0) (1.0.2)
Requirement already satisfied: protobuf>=3.15.3 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]==1.10.0) (3.19.1)
Requirement already satisfied: numpy>=1.16 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]==1.10.0) (1.21.3)
Requirement already satisfied: click>=7.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]==1.10.0) (8.0.3)
Requirement already satisfied: jsonschema in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]==1.10.0) (3.2.0)
Requirement already satisfied: attrs in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]==1.10.0) (21.2.0)
Collecting gpustat>=1.0.0b1
  Downloading gpustat-1.0.0b1.tar.gz (82 kB)
     |████████████████████████████████| 82 kB 3.2 kB/s
  Preparing metadata (setup.py) ... done
Collecting aiohttp-cors
  Downloading aiohttp_cors-0.7.0-py3-none-any.whl (27 kB)
Collecting aiosignal
  Downloading aiosignal-1.2.0-py3-none-any.whl (8.2 kB)
Collecting aioredis<2
  Downloading aioredis-1.3.1-py3-none-any.whl (65 kB)
     |████████████████████████████████| 65 kB 921 kB/s
Collecting colorful
  Downloading colorful-0.5.4-py2.py3-none-any.whl (201 kB)
     |████████████████████████████████| 201 kB 70 kB/s
Requirement already satisfied: smart-open in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]==1.10.0) (5.2.1)
Collecting aiohttp>=3.7
  Downloading aiohttp-3.8.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.3 MB)
     |████████████████████████████████| 1.3 MB 65.0 MB/s
Collecting frozenlist
  Downloading frozenlist-1.3.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (158 kB)
     |████████████████████████████████| 158 kB 79.0 MB/s
Requirement already satisfied: requests in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]==1.10.0) (2.26.0)
Collecting py-spy>=0.2.0
  Downloading py_spy-0.3.11-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (3.0 MB)
     |████████████████████████████████| 3.0 MB 75.8 MB/s
Collecting opencensus
  Downloading opencensus-0.8.0-py2.py3-none-any.whl (128 kB)
     |████████████████████████████████| 128 kB 80.4 MB/s
Requirement already satisfied: prometheus-client>=0.7.1 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]==1.10.0) (0.11.0)
Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from aiohttp>=3.7->ray[default]==1.10.0) (2.0.7)
Collecting async-timeout<5.0,>=4.0.0a3
  Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Collecting yarl<2.0,>=1.0
  Downloading yarl-1.7.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (308 kB)
     |████████████████████████████████| 308 kB 75.2 MB/s
Collecting multidict<7.0,>=4.5
  Downloading multidict-6.0.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (121 kB)
     |████████████████████████████████| 121 kB 108.7 MB/s
Collecting hiredis
  Downloading hiredis-2.0.0-cp38-cp38-manylinux2010_x86_64.whl (85 kB)
     |████████████████████████████████| 85 kB 96 kB/s
Requirement already satisfied: six>=1.7 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from gpustat>=1.0.0b1->ray[default]==1.10.0) (1.15.0)
Collecting nvidia-ml-py3>=7.352.0
  Downloading nvidia-ml-py3-7.352.0.tar.gz (19 kB)
  Preparing metadata (setup.py) ... done
Collecting psutil
  Downloading psutil-5.9.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (283 kB)
     |████████████████████████████████| 283 kB 97.0 MB/s
Collecting blessed>=1.17.1
  Downloading blessed-1.19.1-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 205 kB/s
Requirement already satisfied: packaging>=20.4 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from redis>=3.5.0->ray[default]==1.10.0) (21.0)
Collecting deprecated>=1.2.3
  Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)
Requirement already satisfied: setuptools in /anaconda/envs/py38_default/lib/python3.8/site-packages (from jsonschema->ray[default]==1.10.0) (58.0.4)
Requirement already satisfied: pyrsistent>=0.14.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from jsonschema->ray[default]==1.10.0) (0.17.3)
Collecting opencensus-context==0.1.2
  Downloading opencensus_context-0.1.2-py2.py3-none-any.whl (4.4 kB)
Collecting google-api-core<3.0.0,>=1.0.0
  Downloading google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)
     |████████████████████████████████| 111 kB 112.8 MB/s
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from requests->ray[default]==1.10.0) (1.26.7)
Requirement already satisfied: idna<4,>=2.5 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from requests->ray[default]==1.10.0) (3.3)
Requirement already satisfied: certifi>=2017.4.17 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from requests->ray[default]==1.10.0) (2021.10.8)
Requirement already satisfied: wcwidth>=0.1.4 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from blessed>=1.17.1->gpustat>=1.0.0b1->ray[default]==1.10.0) (0.2.5)
Requirement already satisfied: wrapt<2,>=1.10 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from deprecated>=1.2.3->redis>=3.5.0->ray[default]==1.10.0) (1.12.1)
Collecting googleapis-common-protos<2.0dev,>=1.52.0
  Downloading googleapis_common_protos-1.54.0-py2.py3-none-any.whl (207 kB)
     |████████████████████████████████| 207 kB 100.1 MB/s
Requirement already satisfied: google-auth<3.0dev,>=1.25.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]==1.10.0) (1.35.0)
Requirement already satisfied: pyparsing>=2.0.2 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from packaging>=20.4->redis>=3.5.0->ray[default]==1.10.0) (3.0.4)
Requirement already satisfied: rsa<5,>=3.1.4 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]==1.10.0) (
4.7.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]==1
.10.0) (0.2.8)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]==
1.10.0) (4.2.4)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth<3.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->openc
ensus->ray[default]==1.10.0) (0.4.8)
Building wheels for collected packages: gpustat, nvidia-ml-py3
  Building wheel for gpustat (setup.py) ... done
  Created wheel for gpustat: filename=gpustat-1.0.0b1-py3-none-any.whl size=15979 sha256=0aa15beff2052200bdd9fda3a88e7f9bd2244c62cf3314bb2eb908bef0f24bee
  Stored in directory: /home/azureuser/.cache/pip/wheels/80/87/f2/ed4a55705071f3671434e31c704883425b71bbc90b0c0cfa1e
  Building wheel for nvidia-ml-py3 (setup.py) ... done
  Created wheel for nvidia-ml-py3: filename=nvidia_ml_py3-7.352.0-py3-none-any.whl size=19191 sha256=bb766f715776a2fb3a6b89261bfedb663a246faa28fecaf11479d36779d0e652
  Stored in directory: /home/azureuser/.cache/pip/wheels/b9/b1/68/cb4feab29709d4155310d29a421389665dcab9eb3b679b527b
Successfully built gpustat nvidia-ml-py3
Installing collected packages: multidict, frozenlist, yarl, googleapis-common-protos, deprecated, async-timeout, aiosignal, redis, psutil, opencensus-context, nvidia-ml-py3, hiredis, google-api-core, filel
ock, blessed, aiohttp, ray, py-spy, opencensus, gpustat, colorful, aioredis, aiohttp-cors
Successfully installed aiohttp-3.8.1 aiohttp-cors-0.7.0 aioredis-1.3.1 aiosignal-1.2.0 async-timeout-4.0.2 blessed-1.19.1 colorful-0.5.4 deprecated-1.2.13 filelock-3.6.0 frozenlist-1.3.0 google-api-core-2.
5.0 googleapis-common-protos-1.54.0 gpustat-1.0.0b1 hiredis-2.0.0 multidict-6.0.2 nvidia-ml-py3-7.352.0 opencensus-0.8.0 opencensus-context-0.1.2 psutil-5.9.0 py-spy-0.3.11 ray-1.10.0 redis-4.1.4 yarl-1.7.
2
WARNING: You are using pip version 21.3.1; however, version 22.0.3 is available.
Shared connection to 20.65.39.229 closed.
You should consider upgrading via the '/anaconda/envs/py38_default/bin/python -m pip install --upgrade pip' command.
Processing ./.sky/sky_wheels/sky-0.1.dev0-py3-none-any.whl
Requirement already satisfied: grpcio<=1.43.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from sky==0.1.dev0) (1.41.1)
Collecting pendulum
  Downloading pendulum-2.1.2-cp38-cp38-manylinux1_x86_64.whl (155 kB)
     |████████████████████████████████| 155 kB 9.4 MB/s
Requirement already satisfied: networkx in /anaconda/envs/py38_default/lib/python3.8/site-packages (from sky==0.1.dev0) (2.6.3)
Requirement already satisfied: wheel in /anaconda/envs/py38_default/lib/python3.8/site-packages (from sky==0.1.dev0) (0.37.0)
Requirement already satisfied: jinja2 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from sky==0.1.dev0) (3.0.2)
Requirement already satisfied: cryptography in /anaconda/envs/py38_default/lib/python3.8/site-packages (from sky==0.1.dev0) (35.0.0)
Requirement already satisfied: pandas in /anaconda/envs/py38_default/lib/python3.8/site-packages (from sky==0.1.dev0) (1.3.4)
Collecting PrettyTable
  Downloading prettytable-3.1.1-py3-none-any.whl (26 kB)
Requirement already satisfied: colorama in /anaconda/envs/py38_default/lib/python3.8/site-packages (from sky==0.1.dev0) (0.4.4)
Requirement already satisfied: ray[default] in /anaconda/envs/py38_default/lib/python3.8/site-packages (from sky==0.1.dev0) (1.10.0)
Collecting oauth2client
  Downloading oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
     |████████████████████████████████| 98 kB 791 kB/s
Requirement already satisfied: Click in /anaconda/envs/py38_default/lib/python3.8/site-packages (from sky==0.1.dev0) (8.0.3)
Collecting pycryptodome==3.12.0
  Downloading pycryptodome-3.12.0-cp35-abi3-manylinux2010_x86_64.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 99.2 MB/s
Requirement already satisfied: tabulate in /anaconda/envs/py38_default/lib/python3.8/site-packages (from sky==0.1.dev0) (0.8.9)
Requirement already satisfied: six>=1.5.2 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from grpcio<=1.43.0->sky==0.1.dev0) (1.15.0)
Requirement already satisfied: cffi>=1.12 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from cryptography->sky==0.1.dev0) (1.14.6)
Requirement already satisfied: MarkupSafe>=2.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from jinja2->sky==0.1.dev0) (2.0.1)
Requirement already satisfied: pyasn1>=0.1.7 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from oauth2client->sky==0.1.dev0) (0.4.8)
Requirement already satisfied: pyasn1-modules>=0.0.5 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from oauth2client->sky==0.1.dev0) (0.2.8)
Collecting httplib2>=0.9.1
  Downloading httplib2-0.20.4-py3-none-any.whl (96 kB)
     |████████████████████████████████| 96 kB 553 kB/s
Requirement already satisfied: rsa>=3.1.4 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from oauth2client->sky==0.1.dev0) (4.7.2)
Requirement already satisfied: python-dateutil>=2.7.3 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from pandas->sky==0.1.dev0) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from pandas->sky==0.1.dev0) (2021.3)
Requirement already satisfied: numpy>=1.17.3 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from pandas->sky==0.1.dev0) (1.21.3)
Collecting pytzdata>=2020.1
  Downloading pytzdata-2020.1-py2.py3-none-any.whl (489 kB)
     |████████████████████████████████| 489 kB 70.1 MB/s
Requirement already satisfied: wcwidth in /anaconda/envs/py38_default/lib/python3.8/site-packages (from PrettyTable->sky==0.1.dev0) (0.2.5)
Requirement already satisfied: attrs in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (21.2.0)
Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (1.0.2)
Requirement already satisfied: pyyaml in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (6.0)
Requirement already satisfied: redis>=3.5.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (4.1.4)
Requirement already satisfied: protobuf>=3.15.3 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (3.19.1)
Requirement already satisfied: filelock in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (3.6.0)
Requirement already satisfied: jsonschema in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (3.2.0)
Requirement already satisfied: aiohttp>=3.7 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (3.8.1)
Requirement already satisfied: smart-open in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (5.2.1)
Requirement already satisfied: aioredis<2 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (1.3.1)
Requirement already satisfied: py-spy>=0.2.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (0.3.11)
Requirement already satisfied: aiosignal in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (1.2.0)
Requirement already satisfied: gpustat>=1.0.0b1 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (1.0.0b1)
Requirement already satisfied: colorful in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (0.5.4)
Requirement already satisfied: opencensus in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (0.8.0)
Requirement already satisfied: aiohttp-cors in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (0.7.0)
Requirement already satisfied: frozenlist in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (1.3.0)
Requirement already satisfied: prometheus-client>=0.7.1 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (0.11.0)
Requirement already satisfied: requests in /anaconda/envs/py38_default/lib/python3.8/site-packages (from ray[default]->sky==0.1.dev0) (2.26.0)
Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from aiohttp>=3.7->ray[default]->sky==0.1.dev0) (2.0.7)
Requirement already satisfied: multidict<7.0,>=4.5 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from aiohttp>=3.7->ray[default]->sky==0.1.dev0) (6.0.2)
Requirement already satisfied: yarl<2.0,>=1.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from aiohttp>=3.7->ray[default]->sky==0.1.dev0) (1.7.2)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from aiohttp>=3.7->ray[default]->sky==0.1.dev0) (4.0.2)
Requirement already satisfied: hiredis in /anaconda/envs/py38_default/lib/python3.8/site-packages (from aioredis<2->ray[default]->sky==0.1.dev0) (2.0.0)
Requirement already satisfied: pycparser in /anaconda/envs/py38_default/lib/python3.8/site-packages (from cffi>=1.12->cryptography->sky==0.1.dev0) (2.20)
Requirement already satisfied: psutil in /anaconda/envs/py38_default/lib/python3.8/site-packages (from gpustat>=1.0.0b1->ray[default]->sky==0.1.dev0) (5.9.0)
Requirement already satisfied: blessed>=1.17.1 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from gpustat>=1.0.0b1->ray[default]->sky==0.1.dev0) (1.19.1)
Requirement already satisfied: nvidia-ml-py3>=7.352.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from gpustat>=1.0.0b1->ray[default]->sky==0.1.dev0) (7.352.0)
Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from httplib2>=0.9.1->oauth2client->sky==0.1.dev0) (3.0.4)
Requirement already satisfied: deprecated>=1.2.3 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from redis>=3.5.0->ray[default]->sky==0.1.dev0) (1.2.13)
Requirement already satisfied: packaging>=20.4 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from redis>=3.5.0->ray[default]->sky==0.1.dev0) (21.0)
Requirement already satisfied: pyrsistent>=0.14.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from jsonschema->ray[default]->sky==0.1.dev0) (0.17.3)
Requirement already satisfied: setuptools in /anaconda/envs/py38_default/lib/python3.8/site-packages (from jsonschema->ray[default]->sky==0.1.dev0) (58.0.4)
Requirement already satisfied: opencensus-context==0.1.2 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from opencensus->ray[default]->sky==0.1.dev0) (0.1.2)
Requirement already satisfied: google-api-core<3.0.0,>=1.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from opencensus->ray[default]->sky==0.1.dev0) (2.5.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from requests->ray[default]->sky==0.1.dev0) (1.26.7)
Requirement already satisfied: certifi>=2017.4.17 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from requests->ray[default]->sky==0.1.dev0) (2021.10.8)
Requirement already satisfied: idna<4,>=2.5 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from requests->ray[default]->sky==0.1.dev0) (3.3)
Requirement already satisfied: wrapt<2,>=1.10 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from deprecated>=1.2.3->redis>=3.5.0->ray[default]->sky==0.1.dev0) (1.12.1)
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.52.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->sky==0.1.de
v0) (1.54.0)
Requirement already satisfied: google-auth<3.0dev,>=1.25.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->sky==0.1.dev0) (1.35.0)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->
sky==0.1.dev0) (4.2.4)
Installing collected packages: pytzdata, httplib2, pycryptodome, PrettyTable, pendulum, oauth2client, sky
Successfully installed PrettyTable-3.1.1 httplib2-0.20.4 oauth2client-4.1.3 pendulum-2.1.2 pycryptodome-3.12.0 pytzdata-2020.1 sky-0.1.dev0
WARNING: You are using pip version 21.3.1; however, version 22.0.3 is available.
You should consider upgrading via the '/anaconda/envs/py38_default/bin/python -m pip install --upgrade pip' command.
Collecting azure-cli==2.30.0
  Downloading azure_cli-2.30.0-py3-none-any.whl (2.3 MB)
     |████████████████████████████████| 2.3 MB 22 kB/s
Requirement already satisfied: azure-mgmt-compute~=23.1.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (23.1.0)
Collecting azure-mgmt-containerservice~=16.1.0
  Downloading azure_mgmt_containerservice-16.1.0-py2.py3-none-any.whl (1.7 MB)
     |████████████████████████████████| 1.7 MB 99.8 MB/s
Requirement already satisfied: azure-mgmt-recoveryservicesbackup~=3.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (3.0.0)
Collecting antlr4-python3-runtime~=4.7.2
  Downloading antlr4-python3-runtime-4.7.2.tar.gz (112 kB)
     |████████████████████████████████| 112 kB 96.8 MB/s
  Preparing metadata (setup.py) ... done
Requirement already satisfied: distro in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (1.6.0)
Collecting azure-identity
  Downloading azure_identity-1.7.1-py2.py3-none-any.whl (129 kB)
     |████████████████████████████████| 129 kB 91.4 MB/s
Requirement already satisfied: azure-mgmt-datalake-store~=0.5.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (0.5.0)
Collecting azure-mgmt-sqlvirtualmachine~=1.0.0b1
  Downloading azure_mgmt_sqlvirtualmachine-1.0.0b1-py2.py3-none-any.whl (55 kB)
     |████████████████████████████████| 55 kB 210 kB/s
Requirement already satisfied: azure-mgmt-iothub==2.1.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (2.1.0)
Requirement already satisfied: azure-mgmt-marketplaceordering==1.1.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (1.1.0)
Collecting azure-mgmt-servicefabricmanagedclusters~=1.0.0
  Downloading azure_mgmt_servicefabricmanagedclusters-1.0.0-py2.py3-none-any.whl (112 kB)
     |████████████████████████████████| 112 kB 116.6 MB/s
Requirement already satisfied: azure-mgmt-batch~=16.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (16.0.0)
Collecting azure-mgmt-botservice~=0.3.0
  Downloading azure_mgmt_botservice-0.3.0-py2.py3-none-any.whl (37 kB)
Collecting azure-mgmt-batchai~=7.0.0b1
  Downloading azure_mgmt_batchai-7.0.0b1-py2.py3-none-any.whl (99 kB)
     |████████████████████████████████| 99 kB 493 kB/s
Requirement already satisfied: azure-mgmt-recoveryservices~=2.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (2.0.0)
Collecting azure-mgmt-reservations==0.6.0
  Downloading azure_mgmt_reservations-0.6.0-py2.py3-none-any.whl (33 kB)
Collecting azure-mgmt-deploymentmanager~=0.2.0
  Downloading azure_mgmt_deploymentmanager-0.2.0-py2.py3-none-any.whl (41 kB)
     |████████████████████████████████| 41 kB 30 kB/s
Collecting azure-mgmt-media~=7.0
  Downloading azure_mgmt_media-7.0.0-py2.py3-none-any.whl (219 kB)
     |████████████████████████████████| 219 kB 112.4 MB/s
Requirement already satisfied: azure-batch~=11.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (11.0.0)
Requirement already satisfied: azure-mgmt-cognitiveservices~=12.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (12.0.0)
Collecting azure-mgmt-network~=19.1.0
  Downloading azure_mgmt_network-19.1.0-py2.py3-none-any.whl (22.0 MB)
     |████████████████████████████████| 22.0 MB 105 kB/s
Collecting azure-mgmt-datalake-analytics~=0.2.1
  Downloading azure_mgmt_datalake_analytics-0.2.1-py2.py3-none-any.whl (146 kB)
     |████████████████████████████████| 146 kB 108.2 MB/s
Requirement already satisfied: azure-mgmt-redis~=13.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (13.0.0)
Collecting azure-mgmt-msi~=0.2
  Downloading azure_mgmt_msi-0.2.0-py2.py3-none-any.whl (17 kB)
Collecting azure-mgmt-databoxedge~=1.0.0
  Downloading azure_mgmt_databoxedge-1.0.0-py2.py3-none-any.whl (1.3 MB)
     |████████████████████████████████| 1.3 MB 115.3 MB/s
Collecting azure-mgmt-privatedns~=1.0.0
  Downloading azure_mgmt_privatedns-1.0.0-py2.py3-none-any.whl (43 kB)
     |████████████████████████████████| 43 kB 47 kB/s
Collecting azure-multiapi-storage~=0.7.0
  Downloading azure_multiapi_storage-0.7.0-py2.py3-none-any.whl (5.5 MB)
     |████████████████████████████████| 5.5 MB 88.6 MB/s
Collecting azure-mgmt-iothubprovisioningservices~=0.3.0
  Downloading azure_mgmt_iothubprovisioningservices-0.3.0-py2.py3-none-any.whl (32 kB)
Collecting azure-synapse-artifacts~=0.8.0
  Downloading azure_synapse_artifacts-0.8.0-py2.py3-none-any.whl (997 kB)
     |████████████████████████████████| 997 kB 91.5 MB/s
Requirement already satisfied: azure-mgmt-containerinstance~=9.1.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (9.1.0)
Requirement already satisfied: azure-mgmt-cdn==11.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (11.0.0)
Requirement already satisfied: azure-mgmt-billing==6.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (6.0.0)
Requirement already satisfied: azure-mgmt-rdbms~=10.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (10.0.0)
Collecting PyGithub~=1.38
  Downloading PyGithub-1.55-py3-none-any.whl (291 kB)
     |████████████████████████████████| 291 kB 94.8 MB/s
Requirement already satisfied: azure-mgmt-sql~=3.0.1 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (3.0.1)
Collecting azure-cli-core==2.30.0
  Downloading azure_cli_core-2.30.0-py3-none-any.whl (181 kB)
     |████████████████████████████████| 181 kB 110.1 MB/s
Collecting azure-mgmt-hdinsight~=9.0.0
  Downloading azure_mgmt_hdinsight-9.0.0-py2.py3-none-any.whl (125 kB)
     |████████████████████████████████| 125 kB 115.3 MB/s
Collecting azure-mgmt-servicelinker==1.0.0b1
  Downloading azure_mgmt_servicelinker-1.0.0b1-py2.py3-none-any.whl (32 kB)
Collecting azure-mgmt-web~=4.0.0
  Downloading azure_mgmt_web-4.0.0-py2.py3-none-any.whl (3.9 MB)
     |████████████████████████████████| 3.9 MB 69.4 MB/s
Collecting pytz==2019.1
  Downloading pytz-2019.1-py2.py3-none-any.whl (510 kB)
     |████████████████████████████████| 510 kB 81.5 MB/s
Collecting azure-mgmt-iotcentral~=9.0.0b1
  Downloading azure_mgmt_iotcentral-9.0.0-py2.py3-none-any.whl (31 kB)
Collecting azure-mgmt-appconfiguration~=2.0.0
  Downloading azure_mgmt_appconfiguration-2.0.0-py2.py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 3.5 kB/s
Collecting javaproperties~=0.5.1
  Downloading javaproperties-0.5.2-py2.py3-none-any.whl (19 kB)
Collecting azure-keyvault-administration==4.0.0b3
  Downloading azure_keyvault_administration-4.0.0b3-py2.py3-none-any.whl (77 kB)
     |████████████████████████████████| 77 kB 2.7 MB/s
Collecting PyNaCl~=1.4.0
  Downloading PyNaCl-1.4.0-cp35-abi3-manylinux1_x86_64.whl (961 kB)
     |████████████████████████████████| 961 kB 88.6 MB/s
Collecting azure-mgmt-managementgroups~=0.1
  Downloading azure_mgmt_managementgroups-0.2.0-py2.py3-none-any.whl (59 kB)
     |████████████████████████████████| 59 kB 1.7 MB/s
Collecting azure-synapse-spark~=0.2.0
  Downloading azure_synapse_spark-0.2.0-py2.py3-none-any.whl (29 kB)
Requirement already satisfied: azure-mgmt-trafficmanager~=0.51.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (0.51.0)
Requirement already satisfied: azure-mgmt-search~=8.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (8.0.0)
Requirement already satisfied: six>=1.10.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (1.15.0)
Collecting azure-appconfiguration~=1.1.1
  Downloading azure_appconfiguration-1.1.1-py2.py3-none-any.whl (37 kB)
Collecting azure-graphrbac~=0.60.0
  Downloading azure_graphrbac-0.60.0-py2.py3-none-any.whl (139 kB)
     |████████████████████████████████| 139 kB 112.7 MB/s
Collecting websocket-client~=0.56.0
  Downloading websocket_client-0.56.0-py2.py3-none-any.whl (200 kB)
     |████████████████████████████████| 200 kB 96.7 MB/s
Collecting azure-mgmt-redhatopenshift==1.0.0
  Downloading azure_mgmt_redhatopenshift-1.0.0-py2.py3-none-any.whl (41 kB)
     |████████████████████████████████| 41 kB 4.1 kB/s
Collecting azure-mgmt-apimanagement~=0.2.0
  Downloading azure_mgmt_apimanagement-0.2.0-py2.py3-none-any.whl (275 kB)
     |████████████████████████████████| 275 kB 81.3 MB/s
Collecting azure-mgmt-kusto~=0.3.0
  Downloading azure_mgmt_kusto-0.3.0-py2.py3-none-any.whl (73 kB)
     |████████████████████████████████| 73 kB 25 kB/s
Requirement already satisfied: azure-mgmt-datamigration~=10.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (10.0.0)
Requirement already satisfied: azure-mgmt-signalr~=1.0.0b2 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (1.0.0)
Collecting azure-keyvault~=1.1.0
  Downloading azure_keyvault-1.1.0-py2.py3-none-any.whl (352 kB)
     |████████████████████████████████| 352 kB 87.4 MB/s
Requirement already satisfied: azure-mgmt-eventgrid==9.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (9.0.0)
Collecting azure-mgmt-resource==19.0.0
  Downloading azure_mgmt_resource-19.0.0-py2.py3-none-any.whl (2.2 MB)
     |████████████████████████████████| 2.2 MB 78.6 MB/s
Collecting azure-mgmt-relay~=0.1.0
  Downloading azure_mgmt_relay-0.1.0-py2.py3-none-any.whl (36 kB)
Collecting azure-mgmt-devtestlabs~=4.0
  Downloading azure_mgmt_devtestlabs-4.0.0-py2.py3-none-any.whl (137 kB)
     |████████████████████████████████| 137 kB 99.5 MB/s
Collecting azure-mgmt-imagebuilder~=0.4.0
  Downloading azure_mgmt_imagebuilder-0.4.0-py2.py3-none-any.whl (29 kB)
Collecting azure-keyvault-keys~=4.5.0b4
  Downloading azure_keyvault_keys-4.5.0b6-py3-none-any.whl (372 kB)
     |████████████████████████████████| 372 kB 96.0 MB/s
Collecting azure-mgmt-managedservices~=1.0
  Downloading azure_mgmt_managedservices-1.0.0-py2.py3-none-any.whl (38 kB)
Requirement already satisfied: azure-mgmt-loganalytics~=11.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (11.0.0)
Requirement already satisfied: azure-mgmt-maps~=2.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (2.0.0)
Collecting azure-mgmt-consumption~=2.0
  Downloading azure_mgmt_consumption-2.0.0-py2.py3-none-any.whl (46 kB)
     |████████████████████████████████| 46 kB 197 kB/s
Requirement already satisfied: azure-loganalytics~=0.1.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (0.1.0)
Requirement already satisfied: azure-mgmt-advisor==9.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (9.0.0)
Requirement already satisfied: azure-mgmt-dns~=8.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (8.0.0)
Requirement already satisfied: azure-mgmt-applicationinsights~=1.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (1.0.0)
Collecting azure-mgmt-security~=2.0.0b1
  Downloading azure_mgmt_security-2.0.0b1-py2.py3-none-any.whl (344 kB)
     |████████████████████████████████| 344 kB 92.5 MB/s
Collecting sshtunnel~=0.1.4
  Downloading sshtunnel-0.1.5-py2.py3-none-any.whl (23 kB)
Requirement already satisfied: azure-datalake-store~=0.0.49 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (0.0.52)
Collecting jsondiff~=1.2.0
  Downloading jsondiff-1.2.0.tar.gz (8.0 kB)
  Preparing metadata (setup.py) ... done
Collecting azure-cosmos>=3.0.2,~=3.0
  Downloading azure_cosmos-3.2.0-py2.py3-none-any.whl (106 kB)
     |████████████████████████████████| 106 kB 70.4 MB/s
Collecting xmltodict~=0.12
  Downloading xmltodict-0.12.0-py2.py3-none-any.whl (9.2 kB)
Collecting packaging~=20.9
  Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB 1.5 MB/s
Collecting azure-mgmt-servicebus~=6.0.0
  Downloading azure_mgmt_servicebus-6.0.0-py2.py3-none-any.whl (120 kB)
     |████████████████████████████████| 120 kB 103.8 MB/s
Collecting azure-synapse-managedprivateendpoints~=0.3.0
  Downloading azure_synapse_managedprivateendpoints-0.3.0-py2.py3-none-any.whl (19 kB)
Collecting azure-mgmt-eventhub~=9.1.0
  Downloading azure_mgmt_eventhub-9.1.0-py2.py3-none-any.whl (383 kB)
     |████████████████████████████████| 383 kB 88.3 MB/s
Collecting azure-mgmt-extendedlocation~=1.0.0b2
  Downloading azure_mgmt_extendedlocation-1.0.0-py2.py3-none-any.whl (64 kB)
     |████████████████████████████████| 64 kB 60 kB/s
Collecting azure-mgmt-netapp~=5.1.0
  Downloading azure_mgmt_netapp-5.1.0-py2.py3-none-any.whl (127 kB)
     |████████████████████████████████| 127 kB 102.2 MB/s
Collecting azure-mgmt-synapse~=2.1.0b2
  Downloading azure_mgmt_synapse-2.1.0b4-py3-none-any.whl (514 kB)
     |████████████████████████████████| 514 kB 100.2 MB/s
Collecting azure-storage-common~=1.4
  Downloading azure_storage_common-1.4.2-py2.py3-none-any.whl (47 kB)
     |████████████████████████████████| 47 kB 109 kB/s
Collecting azure-synapse-accesscontrol~=0.5.0
  Downloading azure_synapse_accesscontrol-0.5.0-py2.py3-none-any.whl (30 kB)
Collecting azure-mgmt-containerregistry==8.2.0
  Downloading azure_mgmt_containerregistry-8.2.0-py2.py3-none-any.whl (928 kB)
     |████████████████████████████████| 928 kB 93.1 MB/s
Requirement already satisfied: azure-mgmt-policyinsights~=1.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (1.0.0)
Requirement already satisfied: azure-mgmt-monitor~=2.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli==2.30.0) (2.0.0)
Collecting fabric~=2.4
  Downloading fabric-2.6.0-py2.py3-none-any.whl (53 kB)
     |████████████████████████████████| 53 kB 55 kB/s
Collecting azure-mgmt-storage~=19.0.0
  Downloading azure_mgmt_storage-19.0.0-py2.py3-none-any.whl (1.8 MB)
     |████████████████████████████████| 1.8 MB 91.2 MB/s
Collecting chardet~=3.0.4
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
     |████████████████████████████████| 133 kB 97.9 MB/s
Collecting azure-mgmt-cosmosdb~=7.0.0b2
  Downloading azure_mgmt_cosmosdb-7.0.0b3-py3-none-any.whl (385 kB)
     |████████████████████████████████| 385 kB 69.9 MB/s
Collecting scp~=0.13.2
  Downloading scp-0.13.6-py2.py3-none-any.whl (8.2 kB)
Collecting azure-mgmt-keyvault==9.2.0
  Downloading azure_mgmt_keyvault-9.2.0-py2.py3-none-any.whl (412 kB)
     |████████████████████████████████| 412 kB 26.5 MB/s
Collecting azure-mgmt-servicefabric~=1.0.0
  Downloading azure_mgmt_servicefabric-1.0.0-py2.py3-none-any.whl (139 kB)
     |████████████████████████████████| 139 kB 108.6 MB/s
Collecting semver==2.13.0
  Downloading semver-2.13.0-py2.py3-none-any.whl (12 kB)
Collecting azure-mgmt-authorization~=0.61.0
  Downloading azure_mgmt_authorization-0.61.0-py2.py3-none-any.whl (94 kB)
     |████████████████████████████████| 94 kB 256 kB/s
Collecting pkginfo>=1.5.0.1
  Downloading pkginfo-1.8.2-py2.py3-none-any.whl (26 kB)
Collecting jmespath
  Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting paramiko<3.0.0,>=2.0.8
  Downloading paramiko-2.9.2-py2.py3-none-any.whl (210 kB)
     |████████████████████████████████| 210 kB 85.2 MB/s
Collecting azure-cli-telemetry==1.0.6.*
  Downloading azure_cli_telemetry-1.0.6-py3-none-any.whl (10 kB)
Collecting msal-extensions<0.4,>=0.3.0
  Downloading msal_extensions-0.3.1-py2.py3-none-any.whl (18 kB)
Collecting argcomplete~=1.8
  Downloading argcomplete-1.12.3-py2.py3-none-any.whl (38 kB)
Requirement already satisfied: urllib3[secure]>=1.26.5 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli-core==2.30.0->azure-cli==2.30.0) (1.26.7)
Collecting knack~=0.8.2
  Downloading knack-0.8.2-py3-none-any.whl (59 kB)
     |████████████████████████████████| 59 kB 90 kB/s
Requirement already satisfied: azure-mgmt-core<2,>=1.2.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli-core==2.30.0->azure-cli==2.30.0) (1.3.0)
Collecting requests[socks]~=2.25.1
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 70 kB/s
Collecting humanfriendly<10.0,>=4.7
  Downloading humanfriendly-9.2-py2.py3-none-any.whl (86 kB)
     |████████████████████████████████| 86 kB 32 kB/s
Requirement already satisfied: pyopenssl>=17.1.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli-core==2.30.0->azure-cli==2.30.0) (21.0.0)
Requirement already satisfied: cryptography in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli-core==2.30.0->azure-cli==2.30.0) (35.0.0)
Requirement already satisfied: psutil~=5.8 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli-core==2.30.0->azure-cli==2.30.0) (5.9.0)
Collecting msal<2.0.0,>=1.15.0
  Downloading msal-1.17.0-py2.py3-none-any.whl (79 kB)
     |████████████████████████████████| 79 kB 85 kB/s
Requirement already satisfied: PyJWT>=2.1.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cli-core==2.30.0->azure-cli==2.30.0) (2.3.0)
Requirement already satisfied: azure-core<2.0.0,>=1.7.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-keyvault-administration==4.0.0b3->azure-cli==2.30.0) (1.19.1)
Requirement already satisfied: azure-common~=1.1 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-keyvault-administration==4.0.0b3->azure-cli==2.30.0) (1.1.27)
Requirement already satisfied: msrest>=0.6.21 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-keyvault-administration==4.0.0b3->azure-cli==2.30.0) (0.6.21)
Requirement already satisfied: msrestazure<2.0.0,>=0.4.32 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-mgmt-reservations==0.6.0->azure-cli==2.30.0) (0.6.4)
Collecting portalocker~=1.2
  Downloading portalocker-1.7.1-py2.py3-none-any.whl (10 kB)
Collecting applicationinsights<0.12,>=0.11.1
  Downloading applicationinsights-0.11.10-py2.py3-none-any.whl (55 kB)
     |████████████████████████████████| 55 kB 1.7 MB/s
Requirement already satisfied: requests>=2.10.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-cosmos>=3.0.2,~=3.0->azure-cli==2.30.0) (2.26.0)
Requirement already satisfied: cffi in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-datalake-store~=0.0.49->azure-cli==2.30.0) (1.14.6)
Requirement already satisfied: adal>=0.4.2 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-datalake-store~=0.0.49->azure-cli==2.30.0) (1.2.7)
Requirement already satisfied: azure-nspkg>=2.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-keyvault~=1.1.0->azure-cli==2.30.0) (3.0.2)
Collecting azure-core<2.0.0,>=1.7.0
  Downloading azure_core-1.22.1-py3-none-any.whl (178 kB)
     |████████████████████████████████| 178 kB 111.3 MB/s
Requirement already satisfied: azure-mgmt-nspkg>=2.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-mgmt-consumption~=2.0->azure-cli==2.30.0) (3.0.2)
Requirement already satisfied: azure-mgmt-datalake-nspkg>=2.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-mgmt-datalake-analytics~=0.2.1->azure-cli==2.30.0) (3.0.1)
Requirement already satisfied: python-dateutil in /anaconda/envs/py38_default/lib/python3.8/site-packages (from azure-multiapi-storage~=0.7.0->azure-cli==2.30.0) (2.8.2)
Collecting invoke<2.0,>=1.3
  Downloading invoke-1.6.0-py3-none-any.whl (212 kB)
     |████████████████████████████████| 212 kB 114.9 MB/s
Collecting pathlib2
  Downloading pathlib2-2.3.7.post1-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: pyparsing>=2.0.2 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from packaging~=20.9->azure-cli==2.30.0) (3.0.4)
Requirement already satisfied: deprecated in /anaconda/envs/py38_default/lib/python3.8/site-packages (from PyGithub~=1.38->azure-cli==2.30.0) (1.2.13)
Requirement already satisfied: pycparser in /anaconda/envs/py38_default/lib/python3.8/site-packages (from cffi->azure-datalake-store~=0.0.49->azure-cli==2.30.0) (2.20)
Requirement already satisfied: pyyaml in /anaconda/envs/py38_default/lib/python3.8/site-packages (from knack~=0.8.2->azure-cli-core==2.30.0->azure-cli==2.30.0) (6.0)
Requirement already satisfied: colorama in /anaconda/envs/py38_default/lib/python3.8/site-packages (from knack~=0.8.2->azure-cli-core==2.30.0->azure-cli==2.30.0) (0.4.4)
Requirement already satisfied: tabulate in /anaconda/envs/py38_default/lib/python3.8/site-packages (from knack~=0.8.2->azure-cli-core==2.30.0->azure-cli==2.30.0) (0.8.9)
Requirement already satisfied: pygments in /anaconda/envs/py38_default/lib/python3.8/site-packages (from knack~=0.8.2->azure-cli-core==2.30.0->azure-cli==2.30.0) (2.10.0)
Requirement already satisfied: certifi>=2017.4.17 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from msrest>=0.6.21->azure-keyvault-administration==4.0.0b3->azure-cli==2.30.0) (2021.10.8)
Requirement already satisfied: isodate>=0.6.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from msrest>=0.6.21->azure-keyvault-administration==4.0.0b3->azure-cli==2.30.0) (0.6.0)
Requirement already satisfied: requests-oauthlib>=0.5.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from msrest>=0.6.21->azure-keyvault-administration==4.0.0b3->azure-cli==2.30.0) (1.3.0)
Collecting bcrypt>=3.1.3
  Downloading bcrypt-3.2.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (61 kB)
     |████████████████████████████████| 61 kB 28 kB/s
Requirement already satisfied: charset-normalizer~=2.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from requests>=2.10.0->azure-cosmos>=3.0.2,~=3.0->azure-cli==2.30.0) (2.0.7)
Requirement already satisfied: idna<4,>=2.5 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from requests>=2.10.0->azure-cosmos>=3.0.2,~=3.0->azure-cli==2.30.0) (3.3)
Collecting idna<4,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 786 kB/s
Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from requests[socks]~=2.25.1->azure-cli-core==2.30.0->azure-cli==2.30.0) (1.7.1)
Requirement already satisfied: wrapt<2,>=1.10 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from deprecated->PyGithub~=1.38->azure-cli==2.30.0) (1.12.1)
Requirement already satisfied: oauthlib>=3.0.0 in /anaconda/envs/py38_default/lib/python3.8/site-packages (from requests-oauthlib>=0.5.0->msrest>=0.6.21->azure-keyvault-administration==4.0.0b3->azure-cli==
2.30.0) (3.1.1)
Building wheels for collected packages: antlr4-python3-runtime, jsondiff
  Building wheel for antlr4-python3-runtime (setup.py) ... done
  Created wheel for antlr4-python3-runtime: filename=antlr4_python3_runtime-4.7.2-py3-none-any.whl size=140948 sha256=992b381878cdb8f3b0ec770793b8e9126f257e11f80d8677eb7af4cec73c602e
  Stored in directory: /home/azureuser/.cache/pip/wheels/63/93/52/480c17789fbf685f5d0e33e83c311f4a844cd6e2f5cf036c03
  Building wheel for jsondiff (setup.py) ... done
  Created wheel for jsondiff: filename=jsondiff-1.2.0-py3-none-any.whl size=6552 sha256=9699c8e58c16f68f137e26713cf5c3e8f5ea7eb86d683e236de5e68b4ddb4c25
  Stored in directory: /home/azureuser/.cache/pip/wheels/e0/d6/20/2604c7c4e34cc466610fbb736b96042cbc42919291a76ebaee
Successfully built antlr4-python3-runtime jsondiff
Installing collected packages: idna, chardet, requests, PyNaCl, portalocker, msal, jmespath, bcrypt, azure-core, argcomplete, applicationinsights, pkginfo, pathlib2, paramiko, msal-extensions, knack, invok
e, humanfriendly, azure-cli-telemetry, xmltodict, websocket-client, sshtunnel, semver, scp, pytz, PyGithub, packaging, jsondiff, javaproperties, fabric, azure-synapse-spark, azure-synapse-managedprivateend
points, azure-synapse-artifacts, azure-synapse-accesscontrol, azure-storage-common, azure-multiapi-storage, azure-mgmt-web, azure-mgmt-synapse, azure-mgmt-storage, azure-mgmt-sqlvirtualmachine, azure-mgmt-
servicelinker, azure-mgmt-servicefabricmanagedclusters, azure-mgmt-servicefabric, azure-mgmt-servicebus, azure-mgmt-security, azure-mgmt-resource, azure-mgmt-reservations, azure-mgmt-relay, azure-mgmt-redh
atopenshift, azure-mgmt-privatedns, azure-mgmt-network, azure-mgmt-netapp, azure-mgmt-msi, azure-mgmt-media, azure-mgmt-managementgroups, azure-mgmt-managedservices, azure-mgmt-kusto, azure-mgmt-keyvault,
azure-mgmt-iothubprovisioningservices, azure-mgmt-iotcentral, azure-mgmt-imagebuilder, azure-mgmt-hdinsight, azure-mgmt-extendedlocation, azure-mgmt-eventhub, azure-mgmt-devtestlabs, azure-mgmt-deploymentm
anager, azure-mgmt-datalake-analytics, azure-mgmt-databoxedge, azure-mgmt-cosmosdb, azure-mgmt-containerservice, azure-mgmt-containerregistry, azure-mgmt-consumption, azure-mgmt-botservice, azure-mgmt-batc
hai, azure-mgmt-authorization, azure-mgmt-appconfiguration, azure-mgmt-apimanagement, azure-keyvault-keys, azure-keyvault-administration, azure-keyvault, azure-identity, azure-graphrbac, azure-cosmos, azur
e-cli-core, azure-appconfiguration, antlr4-python3-runtime, azure-cli
  Attempting uninstall: idna
    Found existing installation: idna 3.3
    Uninstalling idna-3.3:
      Successfully uninstalled idna-3.3
  Attempting uninstall: chardet
    Found existing installation: chardet 4.0.0
    Uninstalling chardet-4.0.0:
      Successfully uninstalled chardet-4.0.0
  Attempting uninstall: requests
    Found existing installation: requests 2.26.0
    Uninstalling requests-2.26.0:
      Successfully uninstalled requests-2.26.0
  Attempting uninstall: azure-core
    Found existing installation: azure-core 1.19.1
    Uninstalling azure-core-1.19.1:
      Successfully uninstalled azure-core-1.19.1
  Attempting uninstall: websocket-client
    Found existing installation: websocket-client 1.2.1
    Uninstalling websocket-client-1.2.1:
      Successfully uninstalled websocket-client-1.2.1
  Attempting uninstall: pytz
    Found existing installation: pytz 2021.3
    Uninstalling pytz-2021.3:
      Successfully uninstalled pytz-2021.3
  Attempting uninstall: packaging
    Found existing installation: packaging 21.0
    Uninstalling packaging-21.0:
      Successfully uninstalled packaging-21.0
  Attempting uninstall: azure-storage-common
    Found existing installation: azure-storage-common 2.1.0
    Uninstalling azure-storage-common-2.1.0:
      Successfully uninstalled azure-storage-common-2.1.0
  Attempting uninstall: azure-mgmt-web
    Found existing installation: azure-mgmt-web 5.0.0
    Uninstalling azure-mgmt-web-5.0.0:
      Successfully uninstalled azure-mgmt-web-5.0.0
  Attempting uninstall: azure-mgmt-servicefabric
    Found existing installation: azure-mgmt-servicefabric 2.0.0
    Uninstalling azure-mgmt-servicefabric-2.0.0:
      Successfully uninstalled azure-mgmt-servicefabric-2.0.0
  Attempting uninstall: azure-mgmt-servicebus
    Found existing installation: azure-mgmt-servicebus 7.1.0
    Uninstalling azure-mgmt-servicebus-7.1.0:
      Successfully uninstalled azure-mgmt-servicebus-7.1.0
  Attempting uninstall: azure-mgmt-reservations
    Found existing installation: azure-mgmt-reservations 1.0.0
    Uninstalling azure-mgmt-reservations-1.0.0:
      Successfully uninstalled azure-mgmt-reservations-1.0.0
  Attempting uninstall: azure-mgmt-relay
    Found existing installation: azure-mgmt-relay 1.1.0
    Uninstalling azure-mgmt-relay-1.1.0:
      Successfully uninstalled azure-mgmt-relay-1.1.0
  Attempting uninstall: azure-mgmt-msi
    Found existing installation: azure-mgmt-msi 1.0.0
    Uninstalling azure-mgmt-msi-1.0.0:
      Successfully uninstalled azure-mgmt-msi-1.0.0
  Attempting uninstall: azure-mgmt-media
    Found existing installation: azure-mgmt-media 8.0.0
    Uninstalling azure-mgmt-media-8.0.0:
      Successfully uninstalled azure-mgmt-media-8.0.0
  Attempting uninstall: azure-mgmt-managementgroups
    Found existing installation: azure-mgmt-managementgroups 1.0.0
    Uninstalling azure-mgmt-managementgroups-1.0.0:
      Successfully uninstalled azure-mgmt-managementgroups-1.0.0
  Attempting uninstall: azure-mgmt-iothubprovisioningservices
    Found existing installation: azure-mgmt-iothubprovisioningservices 1.0.0
    Uninstalling azure-mgmt-iothubprovisioningservices-1.0.0:
      Successfully uninstalled azure-mgmt-iothubprovisioningservices-1.0.0
  Attempting uninstall: azure-mgmt-iotcentral
    Found existing installation: azure-mgmt-iotcentral 4.1.0
    Uninstalling azure-mgmt-iotcentral-4.1.0:
      Successfully uninstalled azure-mgmt-iotcentral-4.1.0
  Attempting uninstall: azure-mgmt-eventhub
    Found existing installation: azure-mgmt-eventhub 10.0.0
    Uninstalling azure-mgmt-eventhub-10.0.0:
      Successfully uninstalled azure-mgmt-eventhub-10.0.0
  Attempting uninstall: azure-mgmt-devtestlabs
    Found existing installation: azure-mgmt-devtestlabs 9.0.0
    Uninstalling azure-mgmt-devtestlabs-9.0.0:
      Successfully uninstalled azure-mgmt-devtestlabs-9.0.0
  Attempting uninstall: azure-mgmt-datalake-analytics
    Found existing installation: azure-mgmt-datalake-analytics 0.6.0
    Uninstalling azure-mgmt-datalake-analytics-0.6.0:
      Successfully uninstalled azure-mgmt-datalake-analytics-0.6.0
  Attempting uninstall: azure-mgmt-cosmosdb
    Found existing installation: azure-mgmt-cosmosdb 6.4.0
    Uninstalling azure-mgmt-cosmosdb-6.4.0:
      Successfully uninstalled azure-mgmt-cosmosdb-6.4.0
  Attempting uninstall: azure-mgmt-containerservice
    Found existing installation: azure-mgmt-containerservice 16.3.0
    Uninstalling azure-mgmt-containerservice-16.3.0:
      Successfully uninstalled azure-mgmt-containerservice-16.3.0
  Attempting uninstall: azure-mgmt-consumption
    Found existing installation: azure-mgmt-consumption 8.0.0
    Uninstalling azure-mgmt-consumption-8.0.0:
      Successfully uninstalled azure-mgmt-consumption-8.0.0
  Attempting uninstall: azure-keyvault-keys
    Found existing installation: azure-keyvault-keys 4.4.0
    Uninstalling azure-keyvault-keys-4.4.0:
      Successfully uninstalled azure-keyvault-keys-4.4.0
  Attempting uninstall: azure-keyvault
    Found existing installation: azure-keyvault 4.1.0
    Uninstalling azure-keyvault-4.1.0:
      Successfully uninstalled azure-keyvault-4.1.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
fbprophet 0.7.1 requires cmdstanpy==0.9.5, which is not installed.
fbprophet 0.7.1 requires setuptools-git>=1.2, which is not installed.
azure-storage-file 2.1.0 requires azure-storage-common~=2.1, but you have azure-storage-common 1.4.2 which is incompatible.
Successfully installed PyGithub-1.55 PyNaCl-1.4.0 antlr4-python3-runtime-4.7.2 applicationinsights-0.11.10 argcomplete-1.12.3 azure-appconfiguration-1.1.1 azure-cli-2.30.0 azure-cli-core-2.30.0 azure-cli-t
elemetry-1.0.6 azure-core-1.22.1 azure-cosmos-3.2.0 azure-graphrbac-0.60.0 azure-identity-1.7.1 azure-keyvault-1.1.0 azure-keyvault-administration-4.0.0b3 azure-keyvault-keys-4.5.0b6 azure-mgmt-apimanageme
nt-0.2.0 azure-mgmt-appconfiguration-2.0.0 azure-mgmt-authorization-0.61.0 azure-mgmt-batchai-7.0.0b1 azure-mgmt-botservice-0.3.0 azure-mgmt-consumption-2.0.0 azure-mgmt-containerregistry-8.2.0 azure-mgmt-
containerservice-16.1.0 azure-mgmt-cosmosdb-7.0.0b3 azure-mgmt-databoxedge-1.0.0 azure-mgmt-datalake-analytics-0.2.1 azure-mgmt-deploymentmanager-0.2.0 azure-mgmt-devtestlabs-4.0.0 azure-mgmt-eventhub-9.1.
0 azure-mgmt-extendedlocation-1.0.0 azure-mgmt-hdinsight-9.0.0 azure-mgmt-imagebuilder-0.4.0 azure-mgmt-iotcentral-9.0.0 azure-mgmt-iothubprovisioningservices-0.3.0 azure-mgmt-keyvault-9.2.0 azure-mgmt-kus
to-0.3.0 azure-mgmt-managedservices-1.0.0 azure-mgmt-managementgroups-0.2.0 azure-mgmt-media-7.0.0 azure-mgmt-msi-0.2.0 azure-mgmt-netapp-5.1.0 azure-mgmt-network-19.1.0 azure-mgmt-privatedns-1.0.0 azure-m
gmt-redhatopenshift-1.0.0 azure-mgmt-relay-0.1.0 azure-mgmt-reservations-0.6.0 azure-mgmt-resource-19.0.0 azure-mgmt-security-2.0.0b1 azure-mgmt-servicebus-6.0.0 azure-mgmt-servicefabric-1.0.0 azure-mgmt-s
ervicefabricmanagedclusters-1.0.0 azure-mgmt-servicelinker-1.0.0b1 azure-mgmt-sqlvirtualmachine-1.0.0b1 azure-mgmt-storage-19.0.0 azure-mgmt-synapse-2.1.0b4 azure-mgmt-web-4.0.0 azure-multiapi-storage-0.7.
0 azure-storage-common-1.4.2 azure-synapse-accesscontrol-0.5.0 azure-synapse-artifacts-0.8.0 azure-synapse-managedprivateendpoints-0.3.0 azure-synapse-spark-0.2.0 bcrypt-3.2.0 chardet-3.0.4 fabric-2.6.0 hu
manfriendly-9.2 idna-3.1 invoke-1.6.0 javaproperties-0.5.2 jmespath-0.10.0 jsondiff-1.2.0 knack-0.8.2 msal-1.17.0 msal-extensions-0.3.1 packaging-20.9 paramiko-2.9.2 pathlib2-2.3.7.post1 pkginfo-1.8.2 port
alocker-1.7.1 pytz-2019.1 requests-2.25.1 scp-0.13.6 semver-2.13.0 sshtunnel-0.1.5 websocket-client-0.57.0 xmltodict-0.12.0
WARNING: You are using pip version 21.3.1; however, version 22.0.3 is available.
You should consider upgrading via the '/anaconda/envs/py38_default/bin/python -m pip install --upgrade pip' command.
Did not find any active Ray processes.
Shared connection to 20.65.39.229 closed.
Local node IP: 10.185.0.4
2022-02-23 09:04:17,174 INFO services.py:1374 -- View the Ray dashboard at http://127.0.0.1:8265

--------------------
Ray runtime started.
--------------------

Next steps
  To connect to this Ray runtime from another node, run
    ray start --address='10.185.0.4:6379' --redis-password='5241590000000000'

  Alternatively, use the following Python code:
    import ray
    ray.init(address='auto', _redis_password='5241590000000000')

  To connect to this Ray runtime from outside of the cluster, for example to
  connect to a remote cluster from your laptop directly, use the following
  Python code:
    import ray
    ray.init(address='ray://<head_node_ip_address>:10001')

  If connection fails, check your firewall settings and network configuration.

  To terminate the Ray runtime, run
Shared connection to 20.65.39.229 closed.
Shared connection to 20.65.39.229 closed.
    ray stop
Cluster: azure-start-stop

Checking External environment settings
No head node found. Launching a new cluster. Confirm [y/N]: y [automatic, due to --yes]

Acquiring an up-to-date head node
  Launched a new head node
  Fetching the new head node

<1/1> Setting up head node
  Prepared bootstrap config
  New status: waiting-for-ssh
  [1/7] Waiting for SSH to become available
    Running `uptime` as a test.
    Fetched IP: 20.65.39.229
    SSH still not available (SSH command failed.), retrying in 5 seconds.
    SSH still not available (SSH command failed.), retrying in 5 seconds.
    SSH still not available (SSH command failed.), retrying in 5 seconds.
    Success.
  Updating cluster configuration. [hash=b4bccd9862f61dd056adec49d0b8674e8af69b6a]
  New status: syncing-files
  [2/7] Processing file mounts
    ~/.sky/sky_wheels/ from /tmp/sky_wheels_romilb/
    ~/.ssh/sky-key.pub from /home/romilb/.ssh/sky-key.pub
  [3/7] No worker file mounts to sync
  New status: setting-up
  [4/7] No initialization commands to run.
  [5/7] Initalizing command runner
  [6/7] Running setup commands
    (0/3) (which pip3 > /dev/null 2>&1 |...
    (1/3) pip3 uninstall sky -y &> /dev/...
    (2/3) which conda > /dev/null 2>&1 |...
  [7/7] Starting the Ray runtime
  New status: up-to-date

Useful commands
  Monitor autoscaling with
    ray exec /home/romilb/.sky/generated/azure-start-stop.yml 'tail -n 100 -f /tmp/ray/session_latest/logs/monitor*'
  Connect to a terminal on the cluster head:
    ray attach /home/romilb/.sky/generated/azure-start-stop.yml
  Get a remote shell to the cluster manually:
    ssh -o IdentitiesOnly=yes -i ~/.ssh/sky-key azureuser@20.65.39.229
I 02-23 01:04:20 cloud_vm_ray_backend.py:802] Successfully provisioned or found existing head VM. Waiting for workers.
I 02-23 01:04:36 backend_utils.py:492] No cluster status.
I 02-23 01:04:36 backend_utils.py:492] The autoscaler failed with the following error:
I 02-23 01:04:36 backend_utils.py:492] Traceback (most recent call last):
I 02-23 01:04:36 backend_utils.py:492]   File "/anaconda/envs/py38_default/lib/python3.8/site-packages/ray/autoscaler/_private/monitor.py", line 435, in run
I 02-23 01:04:36 backend_utils.py:492]     self._run()
I 02-23 01:04:36 backend_utils.py:492]   File "/anaconda/envs/py38_default/lib/python3.8/site-packages/ray/autoscaler/_private/monitor.py", line 325, in _run
I 02-23 01:04:36 backend_utils.py:492]     status["autoscaler_report"] = asdict(self.autoscaler.summary())
I 02-23 01:04:36 backend_utils.py:492]   File "/anaconda/envs/py38_default/lib/python3.8/site-packages/ray/autoscaler/_private/autoscaler.py", line 1195, in summary
I 02-23 01:04:36 backend_utils.py:492]     for node_id in self.non_terminated_nodes.all_node_ids:
I 02-23 01:04:36 backend_utils.py:492] AttributeError: 'NoneType' object has no attribute 'all_node_ids'
I 02-23 01:04:36 backend_utils.py:492]
I 02-23 01:04:36 backend_utils.py:492] Loaded cached provider configuration
I 02-23 01:04:36 backend_utils.py:492] If you experience issues with the cloud provider, try re-running the command with --no-config-cache.
I 02-23 01:04:36 backend_utils.py:492] Fetched IP: 20.65.39.229
I 02-23 01:04:36 backend_utils.py:492]
I 02-23 01:04:58 backend_utils.py:492] No cluster status.
I 02-23 01:04:58 backend_utils.py:492] The autoscaler failed with the following error:
I 02-23 01:04:58 backend_utils.py:492] Traceback (most recent call last):
I 02-23 01:04:58 backend_utils.py:492]   File "/anaconda/envs/py38_default/lib/python3.8/site-packages/ray/autoscaler/_private/monitor.py", line 435, in run
I 02-23 01:04:58 backend_utils.py:492]     self._run()
I 02-23 01:04:58 backend_utils.py:492]   File "/anaconda/envs/py38_default/lib/python3.8/site-packages/ray/autoscaler/_private/monitor.py", line 325, in _run
I 02-23 01:04:58 backend_utils.py:492]     status["autoscaler_report"] = asdict(self.autoscaler.summary())
I 02-23 01:04:58 backend_utils.py:492]   File "/anaconda/envs/py38_default/lib/python3.8/site-packages/ray/autoscaler/_private/autoscaler.py", line 1195, in summary
I 02-23 01:04:58 backend_utils.py:492]     for node_id in self.non_terminated_nodes.all_node_ids:
I 02-23 01:04:58 backend_utils.py:492] AttributeError: 'NoneType' object has no attribute 'all_node_ids'
I 02-23 01:04:58 backend_utils.py:492]
I 02-23 01:04:58 backend_utils.py:492] Loaded cached provider configuration
I 02-23 01:04:58 backend_utils.py:492] If you experience issues with the cloud provider, try re-running the command with --no-config-cache.
I 02-23 01:04:58 backend_utils.py:492] Fetched IP: 20.65.39.229
I 02-23 01:04:58 backend_utils.py:492]
I 02-23 01:05:20 backend_utils.py:492] No cluster status.
I 02-23 01:05:20 backend_utils.py:492] The autoscaler failed with the following error:
I 02-23 01:05:20 backend_utils.py:492] Traceback (most recent call last):
I 02-23 01:05:20 backend_utils.py:492]   File "/anaconda/envs/py38_default/lib/python3.8/site-packages/ray/autoscaler/_private/monitor.py", line 435, in run
I 02-23 01:05:20 backend_utils.py:492]     self._run()
I 02-23 01:05:20 backend_utils.py:492]   File "/anaconda/envs/py38_default/lib/python3.8/site-packages/ray/autoscaler/_private/monitor.py", line 325, in _run
I 02-23 01:05:20 backend_utils.py:492]     status["autoscaler_report"] = asdict(self.autoscaler.summary())
I 02-23 01:05:20 backend_utils.py:492]   File "/anaconda/envs/py38_default/lib/python3.8/site-packages/ray/autoscaler/_private/autoscaler.py", line 1195, in summary
I 02-23 01:05:20 backend_utils.py:492]     for node_id in self.non_terminated_nodes.all_node_ids:
I 02-23 01:05:20 backend_utils.py:492] AttributeError: 'NoneType' object has no attribute 'all_node_ids'
I 02-23 01:05:20 backend_utils.py:492]
I 02-23 01:05:20 backend_utils.py:492] Loaded cached provider configuration
I 02-23 01:05:20 backend_utils.py:492] If you experience issues with the cloud provider, try re-running the command with --no-config-cache.
I 02-23 01:05:20 backend_utils.py:492] Fetched IP: 20.65.39.229
I 02-23 01:05:20 backend_utils.py:492]
```

I tried investigating and removed the hard version requirement `azure-cli==2.30.0` in our autoscaler yml:

https://github.com/sky-proj/sky/blob/2d033aab51416a7252ec70fd23f85347203112b6/sky/templates/azure-ray.yml.j2#L90

That removed the fbprophet error, but still getting:
```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following
 dependency conflicts.
azure-storage-file 2.1.0 requires azure-storage-common~=2.1, but you have azure-storage-common 1.4.2 which is incompatible.
```

EDIT: If I comment out line 90 in azure-ray.yml.j2 (i.e., don't install sky and azure-cli), pip doesn't throw up any errors...
I think there were no dependencies issue before. It should be because some other dependency break.
@romilbhardwaj @Michaelvll what's your command (and local ray version)? I cannot reproduce it with `sky cpunode --cloud=azure`. I did see the requirement conflicts, but there are no failures.
This is likely related to a issue in Ray: https://github.com/ray-project/ray/pull/22570
> @romilbhardwaj @Michaelvll what's your command (and local ray version)? I cannot reproduce it with `sky cpunode --cloud=azure`. I did see the requirement conflicts, but there are no failures.

I used the commands `sky launch azure-start-stop.yaml` and my local ray version is `1.9.0`.
Tried on latest master:
```
...
I 03-03 22:48:57 cloud_vm_ray_backend.py:820] Launching on Azure eastus2 ()
I 03-03 22:56:58 cloud_vm_ray_backend.py:831] Successfully provisioned or found existing head VM. Waiting for workers.
I 03-03 22:57:07 backend_utils.py:495] No cluster status.
I 03-03 22:57:07 backend_utils.py:495] Loaded cached provider configuration
I 03-03 22:57:07 backend_utils.py:495] If you experience issues with the cloud provider, try re-running the command with --no-config-cache.
I 03-03 22:57:07 backend_utils.py:495] Fetched IP: 20.122.62.90
I 03-03 22:57:07 backend_utils.py:495]
Traceback (most recent call last):
  File "/Users/zongheng/Dropbox/workspace/riselab/sky-computing/sky/execution.py", line 121, in _execute
    cluster_name=cluster_name)
  File "/Users/zongheng/Dropbox/workspace/riselab/sky-computing/sky/backends/cloud_vm_ray_backend.py", line 1081, in provision
    cluster_name)
  File "/Users/zongheng/Dropbox/workspace/riselab/sky-computing/sky/backends/cloud_vm_ray_backend.py", line 896, in provision_with_retries
    cluster_name=cluster_name)
  File "/Users/zongheng/Dropbox/workspace/riselab/sky-computing/sky/backends/cloud_vm_ray_backend.py", line 719, in _retry_region_zones
    log_abs_path, stream_logs, logging_info)
  File "/Users/zongheng/Dropbox/workspace/riselab/sky-computing/sky/backends/cloud_vm_ray_backend.py", line 840, in _gang_schedule_ray_up
    to_provision_cloud, cluster_config_file, num_nodes)
  File "/Users/zongheng/Dropbox/workspace/riselab/sky-computing/sky/backends/backend_utils.py", line 493, in wait_until_ray_cluster_ready
    stderr=subprocess.PIPE)
  File "/Users/zongheng/anaconda/envs/py37/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'ray exec /Users/zongheng/.sky/generated/sky-a3f8-zongheng.yml "ray status"' returned non-zero exit status 1.
```
I then ran `ray exec /Users/zongheng/.sky/generated/sky-a3f8-zongheng.yml "ray status"` manually and got:
```
...
No cluster status.
The autoscaler failed with the following error:
Traceback (most recent call last):
  File "/anaconda/envs/py38_default/lib/python3.8/site-packages/ray/autoscaler/_private/monitor.py", line 435, in run
    self._run()
  File "/anaconda/envs/py38_default/lib/python3.8/site-packages/ray/autoscaler/_private/monitor.py", line 325, in _run
    status["autoscaler_report"] = asdict(self.autoscaler.summary())
  File "/anaconda/envs/py38_default/lib/python3.8/site-packages/ray/autoscaler/_private/autoscaler.py", line 1195, in summary
    for node_id in self.non_terminated_nodes.all_node_ids:
AttributeError: 'NoneType' object has no attribute 'all_node_ids'
```
@suquark do you see any workarounds before we ship to Kevin?  If not, maybe we should fail early for >1-node Azure clusters.
On the latest master, `sky launch examples/azure_start_stop.yaml` succeeded for me.  Can someone else also confirm the issue has disappeared so we close this?
I got a different error against the previous one during `sky launch examples/azure_start_stop.yaml`.
```
E 03-07 23:10:27 backend_utils.py:729] 2022-03-07 23:10:26,225  INFO util.py:282 -- setting max workers for head node type to 0
E 03-07 23:10:27 backend_utils.py:729] Traceback (most recent call last):
E 03-07 23:10:27 backend_utils.py:729]   File "/Users/zhwu/miniconda3/envs/sky-dev/bin/ray", line 8, in <module>
E 03-07 23:10:27 backend_utils.py:729]     sys.exit(main())
E 03-07 23:10:27 backend_utils.py:729]   File "/Users/zhwu/miniconda3/envs/sky-dev/lib/python3.8/site-packages/ray/scripts/scripts.py", line 1989, in main
E 03-07 23:10:27 backend_utils.py:729]     return cli()
E 03-07 23:10:27 backend_utils.py:729]   File "/Users/zhwu/miniconda3/envs/sky-dev/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
E 03-07 23:10:27 backend_utils.py:729]     return self.main(*args, **kwargs)
E 03-07 23:10:27 backend_utils.py:729]   File "/Users/zhwu/miniconda3/envs/sky-dev/lib/python3.8/site-packages/click/core.py", line 1053, in main
E 03-07 23:10:27 backend_utils.py:729]     rv = self.invoke(ctx)
E 03-07 23:10:27 backend_utils.py:729]   File "/Users/zhwu/miniconda3/envs/sky-dev/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
E 03-07 23:10:27 backend_utils.py:729]     return _process_result(sub_ctx.command.invoke(sub_ctx))
E 03-07 23:10:27 backend_utils.py:729]   File "/Users/zhwu/miniconda3/envs/sky-dev/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
E 03-07 23:10:27 backend_utils.py:729]     return ctx.invoke(self.callback, **ctx.params)
E 03-07 23:10:27 backend_utils.py:729]   File "/Users/zhwu/miniconda3/envs/sky-dev/lib/python3.8/site-packages/click/core.py", line 754, in invoke
E 03-07 23:10:27 backend_utils.py:729]     return __callback(*args, **kwargs)
E 03-07 23:10:27 backend_utils.py:729]   File "/Users/zhwu/miniconda3/envs/sky-dev/lib/python3.8/site-packages/ray/scripts/scripts.py", line 1351, in exec
E 03-07 23:10:27 backend_utils.py:729]     exec_cluster(
E 03-07 23:10:27 backend_utils.py:729]   File "/Users/zhwu/miniconda3/envs/sky-dev/lib/python3.8/site-packages/ray/autoscaler/_private/commands.py", line 949, in exec_cluster
E 03-07 23:10:27 backend_utils.py:729]     head_node = _get_running_head_node(
E 03-07 23:10:27 backend_utils.py:729]   File "/Users/zhwu/miniconda3/envs/sky-dev/lib/python3.8/site-packages/ray/autoscaler/_private/commands.py", line 1257, in _get_running_head_node
E 03-07 23:10:27 backend_utils.py:729]     raise RuntimeError("Head node of cluster ({}) not found!".format(
E 03-07 23:10:27 backend_utils.py:729] RuntimeError: Head node of cluster (az-multi-node) not found!
E 03-07 23:10:27 backend_utils.py:729] 
E 03-07 23:10:27 backend_utils.py:734] Command failed with code 1: ray exec /Users/zhwu/.sky/generated/az-multi-node.yml "ray status"
```
I upgraded my local ray version from `1.9.0` to `1.10.0`, the latest commit (bfe0f4d98) works. Should we enforce the local ray version to 1.10.0? I am trying to see if we can replace `ray exec`.
Is it possible to solve a related problem in one go?  E.g., requesting 1000 VMs, it'll also wait forever but doing in EC2 console the user will immediately get rejected (which is good).
> Is it possible to solve a related problem in one go? E.g., requesting 1000 VMs, it'll also wait forever but doing in EC2 console the user will immediately get rejected (which is good).

Good point! Let me check the configurations in the ray yaml to see if they provide that kind of logic. If not, we may need to consider replacing ray autoscaler with some other provision tool, e.g. terraform.

As a long term fix, multi-region for multi-node sounds like a viable option.

Commit message: Add retry for head ip fetching at the first time

Patch to review:
diff --git a/sky/backends/backend_utils.py b/sky/backends/backend_utils.py
index bcb26d916d5..cda5a261389 100644
--- a/sky/backends/backend_utils.py
+++ b/sky/backends/backend_utils.py
@@ -6,6 +6,7 @@
 from multiprocessing import pool
 import os
 import pathlib
+import re
 import shlex
 import subprocess
 import sys
@@ -46,7 +47,14 @@
 
 # Do not use /tmp because it gets cleared on VM restart.
 _SKY_REMOTE_FILE_MOUNTS_DIR = '~/.sky/file_mounts/'
-# Keep the following two fields in sync with the cluster template:
+
+_LAUNCHED_WORKER_PATTERN = re.compile(r'(\d+) ray[.|_]worker[.|_]default')
+# Intentionally not using prefix 'rf' for the string format because yapf have a
+# bug with python=3.6.
+# 10.133.0.5: ray.worker.default,
+_LAUNCHING_IP_PATTERN = re.compile(
+    r'({}): ray[.|_]worker[.|_]default'.format(IP_ADDR_REGEX))
+WAIT_HEAD_NODE_IP_RETRY_COUNT = 3
 
 
 def _fill_template(template_name: str,
@@ -479,34 +487,84 @@ def get_run_timestamp() -> str:
     return 'sky-' + datetime.datetime.now().strftime('%Y-%m-%d-%H-%M-%S-%f')
 
 
-def wait_until_ray_cluster_ready(cloud: clouds.Cloud, cluster_config_file: str,
-                                 num_nodes: int) -> bool:
+def wait_until_ray_cluster_ready(
+        cluster_config_file: str,
+        num_nodes: int,
+        log_path: str,
+        per_node_timeout: Optional[int] = None) -> bool:
     """Returns whether the entire ray cluster is ready."""
     # FIXME: It may takes a while for the cluster to be available for ray,
     # especially for Azure, causing `ray exec` to fail.
     if num_nodes <= 1:
         return
+
+    # Manually fetching head ip instead of using `ray exec` to avoid the bug
+    # that `ray exec` fails to connect to the head node after some workers
+    # launched especially for Azure.
+    head_ip = get_head_ip(cluster_config_file,
+                          use_cached_head_ip=False,
+                          retry_count=WAIT_HEAD_NODE_IP_RETRY_COUNT)
+
     expected_worker_count = num_nodes - 1
-    if isinstance(cloud, (clouds.AWS, clouds.Azure)):
-        worker_str = 'ray.worker.default'
-    elif isinstance(cloud, clouds.GCP):
-        worker_str = 'ray_worker_default'
-    else:
-        assert False, f'No support for distributed clusters for {cloud}.'
+
+    ssh_user, ssh_key = ssh_credential_from_yaml(cluster_config_file)
+    last_fetched_workers = 0
+    start = time.time()
     while True:
-        proc = subprocess.run(f'ray exec {cluster_config_file} "ray status"',
-                              shell=True,
-                              check=True,
-                              stdout=subprocess.PIPE,
-                              stderr=subprocess.PIPE)
-        output = proc.stdout.decode('ascii')
+        rc, output, stderr = run_command_on_ip_via_ssh(head_ip,
+                                                       'ray status',
+                                                       ssh_user=ssh_user,
+                                                       ssh_private_key=ssh_key,
+                                                       log_path=log_path,
+                                                       stream_logs=False,
+                                                       require_outputs=True)
+
+        handle_returncode(rc, 'ray status',
+                          'Failed to run ray status on head node.', stderr)
         logger.info(output)
-        if f'{expected_worker_count} {worker_str}' in output:
+
+        # Healthy workers
+        result = _LAUNCHED_WORKER_PATTERN.findall(output)
+        healthy_workers = 0
+        if result:
+            assert len(result) == 1, result
+            healthy_workers = int(result[0])
+
+        if healthy_workers == expected_worker_count:
+            # All workers are up.
             break
+
+        # Pending workers that have been launched by ray up.
+        found_ips = _LAUNCHING_IP_PATTERN.findall(output)
+        pending_workers = len(found_ips) if found_ips else 0
+
+        fetched_workers = healthy_workers + pending_workers
+
+        # Check the number of nodes that are fetched. Timeout if no new
+        # nodes fetched in a while (per_node_timeout), though number of
+        # fetched_workers is still not as expected.
+        if fetched_workers != last_fetched_workers:
+            # Reset the start time if the number of launching nodes
+            # changes, i.e. new nodes are launched.
+            logger.debug('Reset start time, as new nodes are launched. '
+                         f'({last_fetched_workers} -> {fetched_workers})')
+            start = time.time()
+            last_fetched_workers = fetched_workers
+        elif (per_node_timeout is not None and
+              time.time() - start > per_node_timeout and
+              fetched_workers != expected_worker_count):
+            logger.error(f'{colorama.Fore.RED}Got Timedout in waiting for '
+                         f'cluster to be ready.{colorama.Style.RESET_ALL}')
+            return False  # failed
+
         if '(no pending nodes)' in output and '(no failures)' in output:
             # Bug in ray autoscaler: e.g., on GCP, if requesting 2 nodes that
             # GCP can satisfy only by half, the worker node would be forgotten.
             # The correct behavior should be for it to error out.
+            logger.error(
+                f'{colorama.Fore.RED}Failed to launch multiple nodes on '
+                'GCP due to a nondeterministic bug in ray autoscaler.'
+                f'{colorama.Style.RESET_ALL}')
             return False  # failed
         time.sleep(10)
     return True  # success
@@ -569,6 +627,15 @@ def _ssh_control_path(ssh_control_filename: Optional[str]) -> Optional[str]:
     return path
 
 
+def ssh_credential_from_yaml(cluster_yaml: str) -> str:
+    """Returns ssh_user and ssh_private_key."""
+    config = read_yaml(cluster_yaml)
+    auth_section = config['auth']
+    ssh_user = auth_section['ssh_user'].strip()
+    ssh_private_key = auth_section.get('ssh_private_key')
+    return ssh_user, ssh_private_key
+
+
 class SshMode(enum.Enum):
     """Enum for SSH mode."""
     # Do not allocating pseudo-tty to avoid user input corrupting the output.
@@ -773,6 +840,68 @@ def generate_cluster_name():
     return f'sky-{uuid.uuid4().hex[:4]}-{getpass.getuser()}'
 
 
+def get_node_ips(cluster_yaml: str,
+                 expected_num_nodes: int,
+                 return_private_ips: bool = False) -> List[str]:
+    """Returns the IPs of all nodes in the cluster."""
+    yaml_handle = cluster_yaml
+    if return_private_ips:
+        config = read_yaml(yaml_handle)
+        # Add this field to a temp file to get private ips.
+        config['provider']['use_internal_ips'] = True
+        yaml_handle = cluster_yaml + '.tmp'
+        dump_yaml(yaml_handle, config)
+
+    out = run(f'ray get-head-ip {yaml_handle}',
+              stdout=subprocess.PIPE).stdout.decode().strip()
+    head_ip = re.findall(IP_ADDR_REGEX, out)
+    assert 1 == len(head_ip), out
+
+    out = run(f'ray get-worker-ips {yaml_handle}',
+              stdout=subprocess.PIPE).stdout.decode()
+    worker_ips = re.findall(IP_ADDR_REGEX, out)
+    assert expected_num_nodes - 1 == len(worker_ips), (expected_num_nodes - 1,
+                                                       out)
+    if return_private_ips:
+        os.remove(yaml_handle)
+    return head_ip + worker_ips
+
+
+def get_head_ip(
+    handle: Union[backends.Backend.ResourceHandle, str],
+    use_cached_head_ip: bool = True,
+    retry_count: int = 1,
+) -> str:
+    """Returns the ip of the head node"""
+    assert not use_cached_head_ip or retry_count == 1, (
+        'Cannot use cached_head_ip when retry_count is not 1')
+    assert not use_cached_head_ip or isinstance(
+        handle, str), ('Cannot use cached_head_ip when handle is None')
+    if use_cached_head_ip:
+        if handle.head_ip is None:
+            # This happens for INIT clusters (e.g., exit 1 in setup).
+            raise ValueError(
+                'Cluster\'s head IP not found; is it up? To fix: '
+                'run a successful launch first (`sky launch`) to ensure'
+                ' the cluster status is UP (`sky status`).')
+        head_ip = handle.head_ip
+    else:
+        for i in range(retry_count):
+            try:
+                out = run(f'ray get-head-ip {handle}',
+                          stdout=subprocess.PIPE).stdout.decode().strip()
+                head_ip = re.findall(IP_ADDR_REGEX, out)
+                assert 1 == len(head_ip), out
+                head_ip = head_ip[0]
+                break
+            except subprocess.CalledProcessError as e:
+                if i == retry_count - 1:
+                    raise e
+                # Retry if the cluster is not up yet.
+                time.sleep(5)
+    return head_ip
+
+
 def get_backend_from_handle(
         handle: backends.Backend.ResourceHandle) -> backends.Backend:
     """Gets a Backend object corresponding to a handle.
diff --git a/sky/backends/cloud_vm_ray_backend.py b/sky/backends/cloud_vm_ray_backend.py
index 614b357d362..40c275e8016 100644
--- a/sky/backends/cloud_vm_ray_backend.py
+++ b/sky/backends/cloud_vm_ray_backend.py
@@ -49,6 +49,9 @@
 
 _PATH_SIZE_MEGABYTES_WARN_THRESHOLD = 256
 
+# Timeout for provision a cluster and wait for it to be ready in seconds.
+_CLUSTER_PER_NODE_PROVISION_TIMEOUT = 30
+
 
 def _check_cluster_name_is_valid(cluster_name: str) -> None:
     """Errors out on invalid cluster names not supported by cloud providers.
@@ -714,34 +717,34 @@ def _retry_region_zones(self, to_provision: Resources, num_nodes: int,
                 'region_name': region.name,
                 'zone_str': zone_str,
             }
-            gang_failed, rc, stdout, stderr = self._gang_schedule_ray_up(
+            gang_succeeded, rc, stdout, stderr = self._gang_schedule_ray_up(
                 to_provision.cloud, num_nodes, cluster_config_file,
                 log_abs_path, stream_logs, logging_info)
 
-            if gang_failed or rc != 0:
-                if gang_failed:
-                    # There exist partial nodes (e.g., head node) so we must
-                    # down before moving on to other regions.
-                    # FIXME(zongheng): terminating a potentially live cluster
-                    # is scary.  Say: users have an existing cluster, do sky
-                    # launch, gang failed, then we are terminating it here.
-                    logger.error('*** Failed provisioning the cluster. ***')
-                    logger.error('====== stdout ======')
-                    logger.error(stdout)
-                    logger.error('====== stderr ======')
-                    logger.error(stderr)
-                    logger.error('*** Tearing down the failed cluster. ***')
-                    CloudVmRayBackend().teardown(handle, terminate=True)
-                    self._update_blocklist_on_error(
-                        to_provision.cloud,
-                        region,
-                        # Ignored and block region:
-                        zones=None,
-                        stdout=None,
-                        stderr=None)
-                else:
-                    self._update_blocklist_on_error(to_provision.cloud, region,
-                                                    zones, stdout, stderr)
+            if rc != 0:
+                # ray up failed.
+                self._update_blocklist_on_error(to_provision.cloud, region,
+                                                zones, stdout, stderr)
+            elif not gang_succeeded:
+                # gang scheduling failed.
+
+                # There exist partial nodes (e.g., head node) so we must
+                # down before moving on to other regions.
+                # FIXME(zongheng): terminating a potentially live cluster
+                # is scary.  Say: users have an existing cluster, do sky
+                # launch, gang failed, then we are terminating it here.
+                logger.error('*** Failed provisioning the cluster. ***')
+                # The stdout/stderr of ray up is not useful here, since
+                # head node is successfully provisioned.
+                logger.error('*** Tearing down the failed cluster. ***')
+                CloudVmRayBackend().teardown(handle, terminate=True)
+                self._update_blocklist_on_error(
+                    to_provision.cloud,
+                    region,
+                    # Ignored and block region:
+                    zones=None,
+                    stdout=None,
+                    stderr=None)
             else:
                 # Success.
 
@@ -772,7 +775,7 @@ def _gang_schedule_ray_up(self, to_provision_cloud: clouds.Cloud,
             would fail.)
 
         Returns:
-          (did gang scheduling failed; proc; stdout; stderr).
+          (did gang scheduling succeeded; returncode; stdout; stderr).
         """
         # FIXME(zhwu,zongheng): ray up on multiple nodes ups the head node then
         # waits for all workers; turn it into real gang scheduling.
@@ -826,7 +829,7 @@ def ray_up(start_streaming_at):
 
         # Only 1 node or head node provisioning failure.
         if num_nodes == 1 or returncode != 0:
-            return False, returncode, stdout, stderr
+            return True, returncode, stdout, stderr
 
         logger.info(f'{style.BRIGHT}Successfully provisioned or found'
                     f' existing head VM. Waiting for workers.{style.RESET_ALL}')
@@ -834,16 +837,14 @@ def ray_up(start_streaming_at):
         # FIXME(zongheng): the below requires ray processes are up on head. To
         # repro it failing: launch a 2-node cluster, log into head and ray
         # stop, then launch again.
-        # TODO: if requesting a large amount (say 32) of expensive VMs, this
-        # may loop for a long time.  Use timeouts and treat as gang_failed.
         cluster_ready = backend_utils.wait_until_ray_cluster_ready(
-            to_provision_cloud, cluster_config_file, num_nodes)
-        gang_failed = not cluster_ready
-        if gang_failed:
-            # Head OK; gang scheduling failure.
-            return gang_failed, returncode, stdout, stderr
-
-        return False, returncode, stdout, stderr
+            cluster_config_file,
+            num_nodes,
+            log_path=log_abs_path,
+            per_node_timeout=_CLUSTER_PER_NODE_PROVISION_TIMEOUT)
+        # Do not need rc/stdout/stderr if gang scheduling failed.
+        # gang_succeeded = False, if head OK, but workers failed.
+        return cluster_ready, returncode, None, None
 
     def _ensure_cluster_ray_started(self,
                                     handle: 'CloudVmRayBackend.ResourceHandle',
@@ -1038,8 +1039,8 @@ def _check_existing_cluster(
     def _set_tpu_name(self, cluster_config_file: str, num_nodes: int,
                       tpu_name: str) -> None:
         """Sets TPU_NAME on all nodes."""
-        ip_list = self._get_node_ips(cluster_config_file, num_nodes)
-        ssh_user, ssh_private_key = self._get_ssh_credential(
+        ip_list = backend_utils.get_node_ips(cluster_config_file, num_nodes)
+        ssh_user, ssh_key = backend_utils.ssh_credential_from_yaml(
             cluster_config_file)
 
         for ip in ip_list:
@@ -1049,7 +1050,7 @@ def _set_tpu_name(self, cluster_config_file: str, num_nodes: int,
                 ip,
                 cmd,
                 ssh_user=ssh_user,
-                ssh_private_key=ssh_private_key,
+                ssh_private_key=ssh_key,
                 log_path=os.path.join(self.log_dir, 'tpu_setup.log'))
             backend_utils.handle_returncode(returncode, cmd,
                                             'Failed to set TPU_NAME on node.')
@@ -1102,11 +1103,15 @@ def provision(self,
         if tpu_name is not None:
             self._set_tpu_name(cluster_config_file, launched_nodes, tpu_name)
 
+        head_ip = backend_utils.get_head_ip(
+            cluster_config_file,
+            use_cached_head_ip=False,
+            retry_count=backend_utils.WAIT_HEAD_NODE_IP_RETRY_COUNT)
         handle = self.ResourceHandle(
             cluster_name=cluster_name,
             cluster_yaml=cluster_config_file,
             # Cache head ip in the handle to speed up ssh operations.
-            head_ip=self._get_node_ips(cluster_config_file, launched_nodes)[0],
+            head_ip=head_ip,
             launched_nodes=launched_nodes,
             launched_resources=provisioned_resources,
             # TPU.
@@ -1125,7 +1130,8 @@ def sync_workdir(self, handle: ResourceHandle, workdir: Path) -> None:
         # e.g., in CLI.  So we should rerun rsync_up.
         fore = colorama.Fore
         style = colorama.Style
-        ip_list = self._get_node_ips(handle.cluster_yaml, handle.launched_nodes)
+        ip_list = backend_utils.get_node_ips(handle.cluster_yaml,
+                                             handle.launched_nodes)
         full_workdir = os.path.abspath(os.path.expanduser(workdir))
 
         # These asserts have been validated at Task construction time.
@@ -1187,8 +1193,9 @@ def sync_file_mounts(
         style = colorama.Style
         logger.info(f'{fore.CYAN}Processing file mounts.{style.RESET_ALL}')
 
-        ip_list = self._get_node_ips(handle.cluster_yaml, handle.launched_nodes)
-        ssh_user, ssh_private_key = self._get_ssh_credential(
+        ip_list = backend_utils.get_node_ips(handle.cluster_yaml,
+                                             handle.launched_nodes)
+        ssh_user, ssh_key = backend_utils.ssh_credential_from_yaml(
             handle.cluster_yaml)
 
         log_path = os.path.join(self.log_dir, 'file_mounts.log')
@@ -1211,7 +1218,7 @@ def _sync_node(ip):
                         ip,
                         command,
                         ssh_user=ssh_user,
-                        ssh_private_key=ssh_private_key,
+                        ssh_private_key=ssh_key,
                         log_path=log_path,
                         stream_logs=False,
                         ssh_control_name=self._ssh_control_name(handle))
@@ -1323,7 +1330,7 @@ def _symlink_node(ip):
                 ip,
                 symlink_command,
                 ssh_user=ssh_user,
-                ssh_private_key=ssh_private_key,
+                ssh_private_key=ssh_key,
                 log_path=log_path,
                 ssh_control_name=self._ssh_control_name(handle))
             backend_utils.handle_returncode(
@@ -1353,9 +1360,9 @@ def setup(self, handle: ResourceHandle, task: Task) -> None:
             setup_sh_path = f.name
             setup_file = os.path.basename(setup_sh_path)
             # Sync the setup script up and run it.
-            ip_list = self._get_node_ips(handle.cluster_yaml,
-                                         handle.launched_nodes)
-            ssh_user, ssh_private_key = self._get_ssh_credential(
+            ip_list = backend_utils.get_node_ips(handle.cluster_yaml,
+                                                 handle.launched_nodes)
+            ssh_user, ssh_key = backend_utils.ssh_credential_from_yaml(
                 handle.cluster_yaml)
 
             def _setup_node(ip: int) -> int:
@@ -1370,7 +1377,7 @@ def _setup_node(ip: int) -> int:
                     ip,
                     cmd,
                     ssh_user=ssh_user,
-                    ssh_private_key=ssh_private_key,
+                    ssh_private_key=ssh_key,
                     log_path=os.path.join(self.log_dir, 'setup.log'),
                     ssh_control_name=self._ssh_control_name(handle))
                 backend_utils.handle_returncode(
@@ -1409,7 +1416,8 @@ def sync_down_logs(self, handle: ResourceHandle, job_id: int) -> None:
         logger.info(f'{fore.CYAN}Logs Directory: '
                     f'{style.BRIGHT}{local_log_dir}{style.RESET_ALL}')
 
-        ips = self._get_node_ips(handle.cluster_yaml, handle.launched_nodes)
+        ips = backend_utils.get_node_ips(handle.cluster_yaml,
+                                         handle.launched_nodes)
 
         def rsync_down(ip: str) -> None:
             from ray.autoscaler import sdk  # pylint: disable=import-outside-toplevel
@@ -1816,33 +1824,6 @@ def teardown(self, handle: ResourceHandle, terminate: bool) -> None:
                 os.remove(handle.tpu_create_script)
                 os.remove(handle.tpu_delete_script)
 
-    def _get_node_ips(self,
-                      cluster_yaml: str,
-                      expected_num_nodes: int,
-                      return_private_ips: bool = False) -> List[str]:
-        """Returns the IPs of all nodes in the cluster."""
-        yaml_handle = cluster_yaml
-        if return_private_ips:
-            config = backend_utils.read_yaml(yaml_handle)
-            # Add this field to a temp file to get private ips.
-            config['provider']['use_internal_ips'] = True
-            yaml_handle = cluster_yaml + '.tmp'
-            backend_utils.dump_yaml(yaml_handle, config)
-
-        out = backend_utils.run(f'ray get-head-ip {yaml_handle}',
-                                stdout=subprocess.PIPE).stdout.decode().strip()
-        head_ip = re.findall(backend_utils.IP_ADDR_REGEX, out)
-        assert 1 == len(head_ip), out
-
-        out = backend_utils.run(f'ray get-worker-ips {yaml_handle}',
-                                stdout=subprocess.PIPE).stdout.decode()
-        worker_ips = re.findall(backend_utils.IP_ADDR_REGEX, out)
-        assert expected_num_nodes - 1 == len(worker_ips), (expected_num_nodes -
-                                                           1, out)
-        if return_private_ips:
-            os.remove(yaml_handle)
-        return head_ip + worker_ips
-
     def _rsync_up(
         self,
         handle: ResourceHandle,
@@ -1863,7 +1844,7 @@ def _rsync_up(
                 f'The cluster "{handle.cluster_name}" appears to be down. '
                 'Run a re-provisioning command (e.g., sky launch) and retry.')
 
-        ssh_user, ssh_private_key = self._get_ssh_credential(
+        ssh_user, ssh_key = backend_utils.ssh_credential_from_yaml(
             handle.cluster_yaml)
         # Build command.
         # TODO(zhwu): This will print a per-file progress bar (with -P),
@@ -1873,7 +1854,7 @@ def _rsync_up(
         rsync_command = ['rsync', '-Pavz']
         rsync_command.append('--filter=\':- .gitignore\'')
         ssh_options = ' '.join(
-            backend_utils.ssh_options_list(ssh_private_key,
+            backend_utils.ssh_options_list(ssh_key,
                                            self._ssh_control_name(handle)))
         rsync_command.append(f'-e "ssh {ssh_options}"')
         rsync_command.extend([
@@ -1895,33 +1876,6 @@ def _rsync_up(
     def _ssh_control_name(self, handle: ResourceHandle) -> str:
         return f'{hashlib.md5(handle.cluster_yaml.encode()).hexdigest()[:10]}'
 
-    def _get_head_ip(
-        self,
-        handle: ResourceHandle,
-        use_cached_head_ip: bool = True,
-    ) -> str:
-        """Returns the ip of the head node"""
-        if use_cached_head_ip:
-            if handle.head_ip is None:
-                # This happens for INIT clusters (e.g., exit 1 in setup).
-                raise ValueError(
-                    'Cluster\'s head IP not found; is it up? To fix: '
-                    'run a successful launch first (`sky launch`) to ensure'
-                    ' the cluster status is UP (`sky status`).')
-            head_ip = handle.head_ip
-        else:
-            head_ip = self._get_node_ips(handle.cluster_yaml,
-                                         handle.launched_nodes)[0]
-        return head_ip
-
-    def _get_ssh_credential(self, cluster_yaml: str) -> str:
-        """Returns ssh_user and ssh_private_key."""
-        config = backend_utils.read_yaml(cluster_yaml)
-        auth = config['auth']
-        ssh_user = auth['ssh_user'].strip()
-        ssh_private_key = auth.get('ssh_private_key')
-        return ssh_user, ssh_private_key
-
     # TODO(zhwu): Refactor this to a CommandRunner class, so different backends
     # can support its own command runner.
     def run_on_head(
@@ -1939,8 +1893,8 @@ def run_on_head(
         require_outputs: bool = False,
     ) -> Union[int, Tuple[int, str, str]]:
         """Runs 'cmd' on the cluster's head node."""
-        head_ip = self._get_head_ip(handle, use_cached_head_ip)
-        ssh_user, ssh_private_key = self._get_ssh_credential(
+        head_ip = backend_utils.get_head_ip(handle, use_cached_head_ip)
+        ssh_user, ssh_private_key = backend_utils.ssh_credential_from_yaml(
             handle.cluster_yaml)
         if under_remote_workdir:
             cmd = f'cd {SKY_REMOTE_WORKDIR} && {cmd}'
diff --git a/sky/sky_logging.py b/sky/sky_logging.py
index e9c0828b1da..95b29a884dd 100644
--- a/sky/sky_logging.py
+++ b/sky/sky_logging.py
@@ -29,5 +29,5 @@ def init_logger(name: str):
 
     logger = logging.getLogger(name)
     logger.addHandler(h)
-    logger.setLevel(logging.INFO)
+    logger.setLevel(logging.DEBUG)
     return logger
diff --git a/sky/templates/aws-ray.yml.j2 b/sky/templates/aws-ray.yml.j2
index 46871d22a1b..d6c1fcc1068 100644
--- a/sky/templates/aws-ray.yml.j2
+++ b/sky/templates/aws-ray.yml.j2
@@ -2,6 +2,7 @@ cluster_name: {{cluster_name}}
 
 # The maximum number of workers nodes to launch in addition to the head node.
 max_workers: {{num_nodes - 1}}
+upscaling_speed: {{num_nodes - 1}}
 idle_timeout_minutes: 60
 
 provider:
diff --git a/sky/templates/azure-ray.yml.j2 b/sky/templates/azure-ray.yml.j2
index 01c86401f3e..e3b08f1009b 100644
--- a/sky/templates/azure-ray.yml.j2
+++ b/sky/templates/azure-ray.yml.j2
@@ -2,6 +2,7 @@ cluster_name: {{cluster_name}}
 
 # The maximum number of workers nodes to launch in addition to the head node.
 max_workers: {{num_nodes - 1}}
+upscaling_speed: {{num_nodes - 1}}
 idle_timeout_minutes: 60
 
 provider:
diff --git a/sky/templates/gcp-ray.yml.j2 b/sky/templates/gcp-ray.yml.j2
index 2a0036b5e3d..437faa19d76 100644
--- a/sky/templates/gcp-ray.yml.j2
+++ b/sky/templates/gcp-ray.yml.j2
@@ -5,6 +5,7 @@ cluster_name: {{cluster_name}}
 
 # The maximum number of workers nodes to launch in addition to the head node.
 max_workers: {{num_nodes - 1}}
+upscaling_speed: {{num_nodes - 1}}
 idle_timeout_minutes: 60
 
 provider:


Code review comments:
diff_hunk: @@ -773,6 +840,68 @@ def generate_cluster_name():
     return f'sky-{uuid.uuid4().hex[:4]}-{getpass.getuser()}'
 
 
+def get_node_ips(cluster_yaml: str,
+                 expected_num_nodes: int,
+                 return_private_ips: bool = False) -> List[str]:
+    """Returns the IPs of all nodes in the cluster."""
+    yaml_handle = cluster_yaml
+    if return_private_ips:
+        config = read_yaml(yaml_handle)
+        # Add this field to a temp file to get private ips.
+        config['provider']['use_internal_ips'] = True
+        yaml_handle = cluster_yaml + '.tmp'
+        dump_yaml(yaml_handle, config)
+
+    out = run(f'ray get-head-ip {yaml_handle}',
+              stdout=subprocess.PIPE).stdout.decode().strip()
+    head_ip = re.findall(IP_ADDR_REGEX, out)
+    assert 1 == len(head_ip), out
+
+    out = run(f'ray get-worker-ips {yaml_handle}',
+              stdout=subprocess.PIPE).stdout.decode()
+    worker_ips = re.findall(IP_ADDR_REGEX, out)
+    assert expected_num_nodes - 1 == len(worker_ips), (expected_num_nodes - 1,
+                                                       out)
+    if return_private_ips:
+        os.remove(yaml_handle)
+    return head_ip + worker_ips
+
+
+def get_head_ip(
+    handle: Union[backends.Backend.ResourceHandle, str],
+    use_cached_head_ip: bool = True,
+    retry_count: int = 1,
+) -> str:
+    """Returns the ip of the head node"""
+    assert not use_cached_head_ip or retry_count == 1, (
+        'Cannot use cached_head_ip when retry_count is not 1')
+    assert not use_cached_head_ip or isinstance(
+        handle, str), ('Cannot use cached_head_ip when handle is None')
path: sky/backends/backend_utils.py
text: @user1:
The parentheses for the message seem unnecessary

@author:
Fixed the whole function. Thanks!