Repository: modal-labs/modal-examples

Issue:
"Fast inference with vLLM (Llama 2 13B)" example is broken
As per https://modal.com/docs/guide/ex/vllm_inference, I ran:
```
git clone https://github.com/modal-labs/modal-examples
cd modal-examples
modal run 06_gpu_and_ml/vllm_inference.py
```

Here's the error that I got:

```
Downloading ray-2.7.1-cp38-cp38-manylinux2014_x86_64.whl (62.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.5/62.5 MB 178.5 MB/s eta 0:00:00
Downloading transformers-4.34.0-py3-none-any.whl (7.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.7/7.7 MB 163.4 MB/s eta 0:00:00
Downloading xformers-0.0.22-cp38-cp38-manylinux2014_x86_64.whl (211.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 211.6/211.6 MB 165.8 MB/s eta 0:00:00
Downloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 307.2/307.2 kB 204.4 MB/s eta 0:00:00
Downloading uvicorn-0.23.2-py3-none-any.whl (59 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.5/59.5 kB 177.2 MB/s eta 0:00:00
Downloading safetensors-0.4.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 197.8 MB/s eta 0:00:00
Downloading tokenizers-0.14.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.8/3.8 MB 191.2 MB/s eta 0:00:00
Downloading huggingface_hub-0.17.3-py3-none-any.whl (295 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 295.0/295.0 kB 199.8 MB/s eta 0:00:00
Building wheels for collected packages: vllm
  Building wheel for vllm (pyproject.toml): started
  Building wheel for vllm (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Building wheel for vllm (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [140 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-38
      creating build/lib.linux-x86_64-cpython-38/vllm
      copying vllm/sequence.py -> build/lib.linux-x86_64-cpython-38/vllm
      copying vllm/block.py -> build/lib.linux-x86_64-cpython-38/vllm
      copying vllm/outputs.py -> build/lib.linux-x86_64-cpython-38/vllm
      copying vllm/sampling_params.py -> build/lib.linux-x86_64-cpython-38/vllm
      copying vllm/logger.py -> build/lib.linux-x86_64-cpython-38/vllm
      copying vllm/utils.py -> build/lib.linux-x86_64-cpython-38/vllm
      copying vllm/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm
      copying vllm/config.py -> build/lib.linux-x86_64-cpython-38/vllm
      creating build/lib.linux-x86_64-cpython-38/vllm/transformers_utils
      copying vllm/transformers_utils/tokenizer.py -> build/lib.linux-x86_64-cpython-38/vllm/transformers_utils
      copying vllm/transformers_utils/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/transformers_utils
      copying vllm/transformers_utils/config.py -> build/lib.linux-x86_64-cpython-38/vllm/transformers_utils
      creating build/lib.linux-x86_64-cpython-38/vllm/model_executor
      copying vllm/model_executor/input_metadata.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor
      copying vllm/model_executor/weight_utils.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor
      copying vllm/model_executor/model_loader.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor
      copying vllm/model_executor/utils.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor
      copying vllm/model_executor/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor
      creating build/lib.linux-x86_64-cpython-38/vllm/engine
      copying vllm/engine/arg_utils.py -> build/lib.linux-x86_64-cpython-38/vllm/engine
      copying vllm/engine/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/engine
      copying vllm/engine/llm_engine.py -> build/lib.linux-x86_64-cpython-38/vllm/engine
      copying vllm/engine/async_llm_engine.py -> build/lib.linux-x86_64-cpython-38/vllm/engine
      copying vllm/engine/ray_utils.py -> build/lib.linux-x86_64-cpython-38/vllm/engine
      creating build/lib.linux-x86_64-cpython-38/vllm/worker
      copying vllm/worker/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/worker
      copying vllm/worker/worker.py -> build/lib.linux-x86_64-cpython-38/vllm/worker
      copying vllm/worker/cache_engine.py -> build/lib.linux-x86_64-cpython-38/vllm/worker
      creating build/lib.linux-x86_64-cpython-38/vllm/core
      copying vllm/core/scheduler.py -> build/lib.linux-x86_64-cpython-38/vllm/core
      copying vllm/core/policy.py -> build/lib.linux-x86_64-cpython-38/vllm/core
      copying vllm/core/block_manager.py -> build/lib.linux-x86_64-cpython-38/vllm/core
      copying vllm/core/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/core
      creating build/lib.linux-x86_64-cpython-38/vllm/entrypoints
      copying vllm/entrypoints/api_server.py -> build/lib.linux-x86_64-cpython-38/vllm/entrypoints
      copying vllm/entrypoints/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/entrypoints
      copying vllm/entrypoints/llm.py -> build/lib.linux-x86_64-cpython-38/vllm/entrypoints
      creating build/lib.linux-x86_64-cpython-38/vllm/transformers_utils/configs
      copying vllm/transformers_utils/configs/mpt.py -> build/lib.linux-x86_64-cpython-38/vllm/transformers_utils/configs
      copying vllm/transformers_utils/configs/baichuan.py -> build/lib.linux-x86_64-cpython-38/vllm/transformers_utils/configs
      copying vllm/transformers_utils/configs/qwen.py -> build/lib.linux-x86_64-cpython-38/vllm/transformers_utils/configs
      copying vllm/transformers_utils/configs/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/transformers_utils/configs
      copying vllm/transformers_utils/configs/falcon.py -> build/lib.linux-x86_64-cpython-38/vllm/transformers_utils/configs
      creating build/lib.linux-x86_64-cpython-38/vllm/model_executor/layers
      copying vllm/model_executor/layers/attention.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/layers
      copying vllm/model_executor/layers/sampler.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/layers
      copying vllm/model_executor/layers/activation.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/layers
      copying vllm/model_executor/layers/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/layers
      copying vllm/model_executor/layers/layernorm.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/layers
      creating build/lib.linux-x86_64-cpython-38/vllm/model_executor/parallel_utils
      copying vllm/model_executor/parallel_utils/parallel_state.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/parallel_utils
      copying vllm/model_executor/parallel_utils/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/parallel_utils
      creating build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/mpt.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/opt.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/internlm.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/gpt2.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/baichuan.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/gpt_j.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/gpt_neox.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/llama.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/qwen.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/bloom.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/falcon.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      copying vllm/model_executor/models/gpt_bigcode.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/models
      creating build/lib.linux-x86_64-cpython-38/vllm/model_executor/parallel_utils/tensor_parallel
      copying vllm/model_executor/parallel_utils/tensor_parallel/random.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/parallel_utils/tensor_parallel
      copying vllm/model_executor/parallel_utils/tensor_parallel/mappings.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/parallel_utils/tensor_parallel
      copying vllm/model_executor/parallel_utils/tensor_parallel/layers.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/parallel_utils/tensor_parallel
      copying vllm/model_executor/parallel_utils/tensor_parallel/utils.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/parallel_utils/tensor_parallel
      copying vllm/model_executor/parallel_utils/tensor_parallel/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/model_executor/parallel_utils/tensor_parallel
      creating build/lib.linux-x86_64-cpython-38/vllm/entrypoints/openai
      copying vllm/entrypoints/openai/protocol.py -> build/lib.linux-x86_64-cpython-38/vllm/entrypoints/openai
      copying vllm/entrypoints/openai/api_server.py -> build/lib.linux-x86_64-cpython-38/vllm/entrypoints/openai
      copying vllm/entrypoints/openai/__init__.py -> build/lib.linux-x86_64-cpython-38/vllm/entrypoints/openai
      running build_ext
      /tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: numpy.core.multiarray failed to import (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
        device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),
      No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
      Traceback (most recent call last):
        File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 434, in build_wheel
          return self._build_with_temp_dir(
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 419, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 145, in <module>
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 364, in run
          self.run_command("build")
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 88, in run
          _build_ext.run(self)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 525, in build_extensions
          _check_cuda_version(compiler_name, compiler_version)
        File "/tmp/pip-build-env-u_96mrb3/overlay/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 413, in _check_cuda_version
          raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
      RuntimeError:
      The detected CUDA version (11.8) mismatches the version that was used to compile
      PyTorch (12.1). Please make sure to use the same CUDA versions.
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for vllm
Failed to build vllm
ERROR: Could not build wheels for vllm, which is required to install pyproject.toml-based projects
Terminating task due to error: failed to run builder command "python -m pip install typing-extensions==4.5.0 'vllm @ git+https://github.com/vllm-project/vllm.git@805de738f618f8b47ab0d450423d23db1e636fa2' "

Caused by:
    container exit status: 1
Runner failed with exception: task exited with failure, status = exit status: 1
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/alexkim/mambaforge/bin/modal:8 in <module>                                                │
│                                                                                                  │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(main())                                                                         │
│   9                                                                                              │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/__main__.py:9 in main               │
│                                                                                                  │
│    8 │   setup_rich_traceback()                                                                  │
│ ❱  9 │   entrypoint_cli()                                                                        │
│   10                                                                                             │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/click/core.py:1157 in __call__            │
│                                                                                                  │
│   1156 │   │   """Alias for :meth:`main`."""                                                     │
│ ❱ 1157 │   │   return self.main(*args, **kwargs)                                                 │
│   1158                                                                                           │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/typer/core.py:778 in main                 │
│                                                                                                  │
│   777 │   ) -> Any:                                                                              │
│ ❱ 778 │   │   return _main(                                                                      │
│   779 │   │   │   self,                                                                          │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/typer/core.py:216 in _main                │
│                                                                                                  │
│   215 │   │   │   with self.make_context(prog_name, args, **extra) as ctx:                       │
│ ❱ 216 │   │   │   │   rv = self.invoke(ctx)                                                      │
│   217 │   │   │   │   if not standalone_mode:                                                    │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/click/core.py:1688 in invoke              │
│                                                                                                  │
│   1687 │   │   │   │   with sub_ctx:                                                             │
│ ❱ 1688 │   │   │   │   │   return _process_result(sub_ctx.command.invoke(sub_ctx))               │
│   1689                                                                                           │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/click/core.py:1688 in invoke              │
│                                                                                                  │
│   1687 │   │   │   │   with sub_ctx:                                                             │
│ ❱ 1688 │   │   │   │   │   return _process_result(sub_ctx.command.invoke(sub_ctx))               │
│   1689                                                                                           │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/click/core.py:1434 in invoke              │
│                                                                                                  │
│   1433 │   │   if self.callback is not None:                                                     │
│ ❱ 1434 │   │   │   return ctx.invoke(self.callback, **ctx.params)                                │
│   1435                                                                                           │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/click/core.py:783 in invoke               │
│                                                                                                  │
│    782 │   │   │   with ctx:                                                                     │
│ ❱  783 │   │   │   │   return __callback(*args, **kwargs)                                        │
│    784                                                                                           │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/click/decorators.py:33 in new_func        │
│                                                                                                  │
│    32 │   def new_func(*args: "P.args", **kwargs: "P.kwargs") -> "R":                            │
│ ❱  33 │   │   return f(get_current_context(), *args, **kwargs)                                   │
│    34                                                                                            │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/cli/run.py:145 in f                 │
│                                                                                                  │
│   144 │   │                                                                                      │
│ ❱ 145 │   │   with run_stub(                                                                     │
│   146 │   │   │   stub,                                                                          │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/synchronicity/synchronizer.py:497 in      │
│ proxy_method                                                                                     │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/synchronicity/combined_types.py:26 in     │
│ __call__                                                                                         │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/contextlib.py:199 in __aenter__                         │
│                                                                                                  │
│   198 │   │   try:                                                                               │
│ ❱ 199 │   │   │   return await anext(self.gen)                                                   │
│   200 │   │   except StopAsyncIteration:                                                         │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/runner.py:88 in _run_stub           │
│                                                                                                  │
│    87 │   │   │   # Create all members                                                           │
│ ❱  88 │   │   │   await app._create_all_objects(                                                 │
│    89 │   │   │   │   stub._blueprint, post_init_state, environment_name, shell=shell, output_   │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/app.py:103 in _create_all_objects   │
│                                                                                                  │
│   102 │   │   │   │   existing_object_id = tag_to_object_id.get(tag)                             │
│ ❱ 103 │   │   │   │   await resolver.load(obj, existing_object_id)                               │
│   104 │   │   │   │   self._tag_to_object_id[tag] = obj.object_id                                │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/_resolver.py:126 in load            │
│                                                                                                  │
│   125 │   │                                                                                      │
│ ❱ 126 │   │   return await cached_future                                                         │
│   127                                                                                            │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/_resolver.py:102 in loader          │
│                                                                                                  │
│   101 │   │   │   async def loader():                                                            │
│ ❱ 102 │   │   │   │   await obj._load(obj, self, existing_object_id)                             │
│   103 │   │   │   │   if existing_object_id is not None and obj.object_id != existing_object_i   │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/image.py:176 in _load               │
│                                                                                                  │
│    175 │   │   │   for image in base_images.values():                                            │
│ ❱  176 │   │   │   │   base_image_ids.append((await resolver.load(image)).object_id)             │
│    177 │   │   │   base_images_pb2s = [                                                          │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/_resolver.py:126 in load            │
│                                                                                                  │
│   125 │   │                                                                                      │
│ ❱ 126 │   │   return await cached_future                                                         │
│   127                                                                                            │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/_resolver.py:102 in loader          │
│                                                                                                  │
│   101 │   │   │   async def loader():                                                            │
│ ❱ 102 │   │   │   │   await obj._load(obj, self, existing_object_id)                             │
│   103 │   │   │   │   if existing_object_id is not None and obj.object_id != existing_object_i   │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/image.py:176 in _load               │
│                                                                                                  │
│    175 │   │   │   for image in base_images.values():                                            │
│ ❱  176 │   │   │   │   base_image_ids.append((await resolver.load(image)).object_id)             │
│    177 │   │   │   base_images_pb2s = [                                                          │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/_resolver.py:126 in load            │
│                                                                                                  │
│   125 │   │                                                                                      │
│ ❱ 126 │   │   return await cached_future                                                         │
│   127                                                                                            │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/_resolver.py:102 in loader          │
│                                                                                                  │
│   101 │   │   │   async def loader():                                                            │
│ ❱ 102 │   │   │   │   await obj._load(obj, self, existing_object_id)                             │
│   103 │   │   │   │   if existing_object_id is not None and obj.object_id != existing_object_i   │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/image.py:176 in _load               │
│                                                                                                  │
│    175 │   │   │   for image in base_images.values():                                            │
│ ❱  176 │   │   │   │   base_image_ids.append((await resolver.load(image)).object_id)             │
│    177 │   │   │   base_images_pb2s = [                                                          │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/_resolver.py:126 in load            │
│                                                                                                  │
│   125 │   │                                                                                      │
│ ❱ 126 │   │   return await cached_future                                                         │
│   127                                                                                            │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/_resolver.py:102 in loader          │
│                                                                                                  │
│   101 │   │   │   async def loader():                                                            │
│ ❱ 102 │   │   │   │   await obj._load(obj, self, existing_object_id)                             │
│   103 │   │   │   │   if existing_object_id is not None and obj.object_id != existing_object_i   │
│                                                                                                  │
│ /Users/alexkim/mambaforge/lib/python3.10/site-packages/modal/image.py:296 in _load               │
│                                                                                                  │
│    295 │   │   │   if result.status == api_pb2.GenericResult.GENERIC_STATUS_FAILURE:             │
│ ❱  296 │   │   │   │   raise RemoteError(f"Image build for {image_id} failed with the exception  │
│    297 │   │   │   elif result.status == api_pb2.GenericResult.GENERIC_STATUS_TERMINATED:        │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RemoteError: Image build for im-bvJc9XyO2U9rSetK1p4yUT failed with the exception:
task exited with failure, status = exit status: 1
```



Discussion:

Resolving patch:
diff --git a/06_gpu_and_ml/vllm_inference.py b/06_gpu_and_ml/vllm_inference.py
--- a/06_gpu_and_ml/vllm_inference.py
+++ b/06_gpu_and_ml/vllm_inference.py
@@ -58,18 +58,23 @@ def download_model_to_folder():
 
 # ### Image definition
 # We’ll start from a Dockerhub image recommended by `vLLM`, upgrade the older
-# version of `torch` to a new one specifically built for CUDA 11.8. Next, we install `vLLM` from source to get the latest updates.
-# Finally, we’ll use run_function to run the function defined above to ensure the weights of the model
-# are saved within the container image.
-#
+# version of `torch` (from 1.14) to a new one specifically built for CUDA 11.8.
+# Next, we install `vLLM` from source to get the latest updates. Finally, we’ll
+# use run_function to run the function defined above to ensure the weights of
+# the model are saved within the container image.
 image = (
     Image.from_registry("nvcr.io/nvidia/pytorch:22.12-py3")
     .pip_install(
-        "torch==2.0.1", index_url="https://download.pytorch.org/whl/cu118"
+        "torch==2.0.1+cu118", index_url="https://download.pytorch.org/whl/cu118"
     )
-    # Pinned to 08/15/2023
+    # Pinned to 10/10/2023.
     .pip_install(
-        "vllm @ git+https://github.com/vllm-project/vllm.git@805de738f618f8b47ab0d450423d23db1e636fa2",
+        # TODO: Point back upstream once
+        # https://github.com/vllm-project/vllm/pull/1239 is merged. We need it
+        # when installing from a SHA directly. We also need to install from a
+        # SHA directly to pick up https://github.com/vllm-project/vllm/pull/1290,
+        # which locks torch==2.0.1 (torch==2.1.0 is built using CUDA 12.1).
+        "vllm @ git+https://github.com/modal-labs/vllm.git@eed12117603bcece41d7ac0f10bcf7ece0fde2fc",
         "typing-extensions==4.5.0",  # >=4.6 causes typing issues
     )
     # Use the barebones hf-transfer package for maximum download speeds. No progress bar, but expect 700MB/s.


Test patch: