Repository: skypilot-org/skypilot
Pull request: [LLM] Fix mixtral example for Azure
<!-- Describe the changes in this PR -->

Fixes #2905
Fix mixtral example for Azure by removing the problemetic nccl config file on Azure.

With this PR, after #2434 is merged, we should be able to change the examples to `disk_tier: best`, so that Azure is allowed to be part of the candidates.

<!-- Describe the tests ran -->
<!-- Unit tests (tests/test_*.py) are part of GitHub CI; below are tests that launch on the cloud. -->

Tested (run the relevant ones):

- [ ] Code formatting: `bash format.sh`
- [x] Any manual or new tests for this PR (please specify below)
  - [x] `sky launch --disk-tier none -c test-mixtral --cloud azure llm/mixtral/serve.yaml`
  - [x] `sky launch -c test-gcp-mixtral --cloud gcp --use-spot llm/mixtral/serve.yaml`
- [ ] All smoke tests: `pytest tests/test_smoke.py` 
- [ ] Relevant individual smoke tests: `pytest tests/test_smoke.py::test_fill_in_the_name` 
- [ ] Backward compatibility tests: `bash tests/backward_comaptibility_tests.sh`


Problem statement:
[Example] Mixtral example fail to work on Azure VM due to NCCL error
`sky launch --disk-tier none -c test-mixtral --cloud azure llm/mixtral/serve.yaml` 

```
I 12-27 11:40:38 log_lib.py:431] Start streaming logs for job 1.
INFO: Tip: use Ctrl-C to exit log streaming (task will not be killed).
INFO: Waiting for task resources on 1 node. This will block if the cluster is full.
INFO: All task resources reserved.
INFO: Reserved IPs: ['10.130.0.4']
(task, pid=9688) INFO 12-27 11:40:41 api_server.py:727] args: Namespace(host='0.0.0.0', port=8000, allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], served_model_name=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, model='mistralai/Mixtral-8x7B-Instruct-v0.1', tokenizer=None, revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', dtype='auto', max_model_len=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=2, max_parallel_loading_workers=None, block_size=16, seed=0, swap_space=4, gpu_memory_utilization=0.9, max_num_batched_tokens=None, max_num_seqs=256, max_paddings=256, disable_log_stats=False, quantization=None, enforce_eager=False, max_context_len_to_capture=8192, engine_use_ray=False, disable_log_requests=False, max_log_len=None)
config.json: 100%|██████████| 720/720 [00:00<00:00, 4.91MB/s]
(task, pid=9688) 2023-12-27 11:40:43,005        INFO worker.py:1724 -- Started a local Ray instance.
(task, pid=9688) INFO 12-27 11:40:43 llm_engine.py:73] Initializing an LLM engine with config: model='mistralai/Mixtral-8x7B-Instruct-v0.1', tokenizer='mistralai/Mixtral-8x7B-Instruct-v0.1', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=2, quantization=None, enforce_eager=False, seed=0)
tokenizer_config.json: 100%|██████████| 1.46k/1.46k [00:00<00:00, 13.0MB/s]
tokenizer.model: 100%|██████████| 493k/493k [00:00<00:00, 96.4MB/s]
tokenizer.json: 100%|██████████| 1.80M/1.80M [00:00<00:00, 104MB/s]
special_tokens_map.json: 100%|██████████| 72.0/72.0 [00:00<00:00, 805kB/s]
(task, pid=9688) Traceback (most recent call last):
(task, pid=9688)   File "/home/azureuser/miniconda3/envs/mixtral/lib/python3.10/runpy.py", line 196, in _run_module_as_main
(task, pid=9688)     return _run_code(code, main_globals, None,
(task, pid=9688)   File "/home/azureuser/miniconda3/envs/mixtral/lib/python3.10/runpy.py", line 86, in _run_code
(task, pid=9688)     exec(code, run_globals)
(task, pid=9688)   File "/home/azureuser/miniconda3/envs/mixtral/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 737, in <module>
(task, pid=9688)     engine = AsyncLLMEngine.from_engine_args(engine_args)
(task, pid=9688)   File "/home/azureuser/miniconda3/envs/mixtral/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 494, in from_engine_args
(task, pid=9688)     engine = cls(parallel_config.worker_use_ray,
(task, pid=9688)   File "/home/azureuser/miniconda3/envs/mixtral/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 267, in __init__
(task, pid=9688)     self.engine = self._init_engine(*args, **kwargs)
(task, pid=9688)   File "/home/azureuser/miniconda3/envs/mixtral/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 312, in _init_engine
(task, pid=9688)     return engine_class(*args, **kwargs)
(task, pid=9688)   File "/home/azureuser/miniconda3/envs/mixtral/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 112, in __init__
(task, pid=9688)     self._init_workers_ray(placement_group)
(task, pid=9688)   File "/home/azureuser/miniconda3/envs/mixtral/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 195, in _init_workers_ray
(task, pid=9688)     self._run_workers(
(task, pid=9688)   File "/home/azureuser/miniconda3/envs/mixtral/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 755, in _run_workers
(task, pid=9688)     self._run_workers_in_batch(workers, method, *args, **kwargs))
(task, pid=9688)   File "/home/azureuser/miniconda3/envs/mixtral/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 732, in _run_workers_in_batch
(task, pid=9688)     all_outputs = ray.get(all_outputs)
(task, pid=9688)   File "/home/azureuser/miniconda3/envs/mixtral/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
(task, pid=9688)     return fn(*args, **kwargs)
(task, pid=9688)   File "/home/azureuser/miniconda3/envs/mixtral/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
(task, pid=9688)     return func(*args, **kwargs)
(task, pid=9688)   File "/home/azureuser/miniconda3/envs/mixtral/lib/python3.10/site-packages/ray/_private/worker.py", line 2624, in get
(task, pid=9688)     raise value.as_instanceof_cause()
(task, pid=9688) ray.exceptions.RayTaskError(DistBackendError): ray::RayWorkerVllm.execute_method() (pid=12792, ip=10.130.0.4, actor_id=9241629b5329c36992ed7e6301000000, repr=<vllm.engine.ray_utils.RayWorkerVllm object at 0x15334ad95f90>)
(task, pid=9688)   File "/home/azureuser/miniconda3/envs/mixtral/lib/python3.10/site-packages/vllm/engine/ray_utils.py", line 31, in execute_method
(task, pid=9688)     return executor(*args, **kwargs)
(task, pid=9688)   File "/home/azureuser/miniconda3/envs/mixtral/lib/python3.10/site-packages/vllm/worker/worker.py", line 72, in init_model
(task, pid=9688)     _init_distributed_environment(self.parallel_config, self.rank,
(task, pid=9688)   File "/home/azureuser/miniconda3/envs/mixtral/lib/python3.10/site-packages/vllm/worker/worker.py", line 190, in _init_distributed_environment
(task, pid=9688)     torch.distributed.all_reduce(torch.zeros(1).cuda())
(task, pid=9688)   File "/home/azureuser/miniconda3/envs/mixtral/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
(task, pid=9688)     return func(*args, **kwargs)
(task, pid=9688)   File "/home/azureuser/miniconda3/envs/mixtral/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2050, in all_reduce
(task, pid=9688)     work = group.allreduce([tensor], opts)
(task, pid=9688) torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1333, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
(task, pid=9688) ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. 
(task, pid=9688) Last error:
(task, pid=9688) XML Import Channel : dev 2 not found.
INFO: Job finished (status: SUCCEEDED).
```

Hints from issue discussion:
@Michaelvll  have you resolved it?
Thanks for pinning this @yangyingxiang! Currently we did not figure out the reason for this yet. It seems to be an issue with Azure's GPU setup, and we found people also experiencing this issue: https://github.com/NVIDIA/nccl/issues/1101. 

We will keep an eye on this issue, and see if there is some possible reason. Please let us know if you are able to find some solution. : )

Commit message: Add considered resources

Patch to review:
diff --git a/llm/mixtral/README.md b/llm/mixtral/README.md
index 68cc97820d5..8bef6e70db4 100644
--- a/llm/mixtral/README.md
+++ b/llm/mixtral/README.md
@@ -19,6 +19,30 @@ resources:
   accelerators: {A100:4, A100:8, A100-80GB:2, A100-80GB:4, A100-80GB:8}
 ```
 
+The following is the example output of the optimizer:
+
+```
+Considered resources (1 node):
+----------------------------------------------------------------------------------------------------------
+ CLOUD   INSTANCE                    vCPUs   Mem(GB)   ACCELERATORS   REGION/ZONE     COST ($)   CHOSEN   
+----------------------------------------------------------------------------------------------------------
+ Azure   Standard_NC48ads_A100_v4    48      440       A100-80GB:2    eastus          7.35          ✔     
+ GCP     g2-standard-96              96      384       L4:8           us-east4-a      7.98                
+ GCP     a2-ultragpu-2g              24      340       A100-80GB:2    us-central1-a   10.06               
+ Azure   Standard_NC96ads_A100_v4    96      880       A100-80GB:4    eastus          14.69               
+ GCP     a2-highgpu-4g               48      340       A100:4         us-central1-a   14.69               
+ AWS     g5.48xlarge                 192     768       A10G:8         us-east-1       16.29               
+ GCP     a2-ultragpu-4g              48      680       A100-80GB:4    us-central1-a   20.11               
+ Azure   Standard_ND96asr_v4         96      900       A100:8         eastus          27.20               
+ GCP     a2-highgpu-8g               96      680       A100:8         us-central1-a   29.39               
+ Azure   Standard_ND96amsr_A100_v4   96      1924      A100-80GB:8    eastus          32.77               
+ AWS     p4d.24xlarge                96      1152      A100:8         us-east-1       32.77               
+ GCP     a2-ultragpu-8g              96      1360      A100-80GB:8    us-central1-a   40.22               
+ AWS     p4de.24xlarge               96      1152      A100-80GB:8    us-east-1       40.97               
+----------------------------------------------------------------------------------------------------------
+```
+
+
 ### Accessing the model
 
 We can now access the model through the OpenAI API with the IP and port:
diff --git a/llm/mixtral/serve.yaml b/llm/mixtral/serve.yaml
index 1116c617524..d67f6ef5de7 100644
--- a/llm/mixtral/serve.yaml
+++ b/llm/mixtral/serve.yaml
@@ -34,6 +34,9 @@ setup: |
   pip list | grep megablocks || pip install megablocks
 
 run: |
+  # Remove the default nccl.conf which causes failure on Azure
+  sudo mv /etc/nccl.conf /etc/nccl.conf.bak || true
+
   conda activate mixtral
   export PATH=$PATH:/sbin
   python -u -m vllm.entrypoints.openai.api_server \
diff --git a/sky/templates/gcp-ray.yml.j2 b/sky/templates/gcp-ray.yml.j2
index d8c02cb4f9c..5268b4cbf77 100644
--- a/sky/templates/gcp-ray.yml.j2
+++ b/sky/templates/gcp-ray.yml.j2
@@ -192,7 +192,7 @@ setup_commands:
     grep "export TPU_NAME=" ~/.bashrc && echo "TPU_NAME already set" || echo "export TPU_NAME={{tpu_node_name}}" >> ~/.bashrc;
   {%- endif %}
     mkdir -p ~/sky_workdir && mkdir -p ~/.sky/sky_app;
-    (pip3 list | grep "ray " | grep {{ray_version}} 2>&1 > /dev/null || pip3 install --exists-action w -U ray[default]=={{ray_version}});
+    pip3 list | grep "ray " | grep {{ray_version}} 2>&1 > /dev/null || pip3 install --exists-action w -U ray[default]=={{ray_version}};
     (pip3 list | grep "skypilot " && [ "$(cat {{sky_remote_path}}/current_sky_wheel_hash)" == "{{sky_wheel_hash}}" ]) || (pip3 uninstall skypilot -y; pip3 install "$(echo {{sky_remote_path}}/{{sky_wheel_hash}}/skypilot-{{sky_version}}*.whl)[gcp, remote]" && echo "{{sky_wheel_hash}}" > {{sky_remote_path}}/current_sky_wheel_hash || exit 1);
     sudo bash -c 'rm -rf /etc/security/limits.d; echo "* soft nofile 1048576" >> /etc/security/limits.conf; echo "* hard nofile 1048576" >> /etc/security/limits.conf';
     {%- if docker_image is none %}


Code review comments:
diff_hunk: @@ -34,6 +34,9 @@ setup: |
   pip list | grep megablocks || pip install megablocks
 
 run: |
+  # Remove the default nccl.conf which causes failure on Azure
+  sudo mv /etc/nccl.conf /etc/nccl.conf.bak || true
path: llm/mixtral/serve.yaml
text: @user1:
What is this file for? Will this have any side-effects?

@user1:
Also is it a common issue for other workloads? Should we add it in `azure-ray.yml.j2` instead?

@author:
This is a good point! This file is wrongly configured on Azure with multiple GPUs, causing the issue with the distributed workload using nccl. We should include this in the `azure-ray.yml.j2`.

@author:
Fixed. PTAL.