Repository: ray-project/ray
Pull request: Remove `botocore` dependency in Ray Serve LLM
<!-- Thank you for your contribution! Please review https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR. If you don't have the access to it, we will shortly find a reviewer and assign them to your PR. -->

## Why are these changes needed?

Superfluous `botocore` dependency; the default value for `get_lora_model_func`, `get_lora_model_ids`, swallows all exceptions and returns an empty list. I.e., `ClientError` is never seen.

## Related issue number

Closes #53052

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [x] I've included any doc changes needed for https://docs.ray.io/en/master/.
    - [x] I've added any new APIs to the API Reference. For example, if I added a
           method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [x] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [x] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(


Problem statement:
[Serve, LLM] missing botocore dependency!
### What happened + What you expected to happen

There seems to be a missing (or unhandled import of) botocore dependency when importing ray.serve.llm stuff. `ModuleNotFoundError: No module named 'botocore'`


### Versions / Dependencies

```
underwent 19s  uv pip list
Package                                  Version
---------------------------------------- -------------
aiohappyeyeballs                         2.6.1
aiohttp                                  3.11.18
aiohttp-cors                             0.8.1
aiosignal                                1.3.2
airportsdata                             20250224
annotated-types                          0.7.0
anyio                                    4.9.0
astor                                    0.8.1
attrs                                    25.3.0
blake3                                   1.0.4
cachetools                               5.5.2
certifi                                  2025.4.26
charset-normalizer                       3.4.2
click                                    8.2.0
cloudpickle                              3.1.1
colorful                                 0.5.6
compressed-tensors                       0.9.3
deprecated                               1.2.18
depyf                                    0.18.0
dill                                     0.4.0
diskcache                                5.6.3
distlib                                  0.3.9
distro                                   1.9.0
dnspython                                2.7.0
einops                                   0.8.1
email-validator                          2.2.0
fastapi                                  0.115.12
fastapi-cli                              0.0.7
filelock                                 3.18.0
frozenlist                               1.6.0
fsspec                                   2025.3.2
gguf                                     0.16.3
google-api-core                          2.24.2
google-auth                              2.40.1
googleapis-common-protos                 1.70.0
grpcio                                   1.71.0
h11                                      0.16.0
hf-xet                                   1.1.1
httpcore                                 1.0.9
httptools                                0.6.4
httpx                                    0.28.1
huggingface-hub                          0.31.2
idna                                     3.10
importlib-metadata                       8.0.0
interegular                              0.3.3
jinja2                                   3.1.6
jiter                                    0.9.0
jsonref                                  1.1.0
jsonschema                               4.23.0
jsonschema-specifications                2025.4.1
lark                                     1.2.2
llguidance                               0.7.19
llvmlite                                 0.44.0
lm-format-enforcer                       0.10.11
markdown-it-py                           3.0.0
markupsafe                               3.0.2
mdurl                                    0.1.2
mistral-common                           1.5.4
mpmath                                   1.3.0
msgpack                                  1.1.0
msgspec                                  0.19.0
multidict                                6.4.3
nest-asyncio                             1.6.0
networkx                                 3.4.2
ninja                                    1.11.1.4
numba                                    0.61.2
numpy                                    2.2.5
openai                                   1.78.1
opencensus                               0.11.4
opencensus-context                       0.1.3
opencv-python-headless                   4.11.0.86
opentelemetry-api                        1.26.0
opentelemetry-exporter-otlp              1.26.0
opentelemetry-exporter-otlp-proto-common 1.26.0
opentelemetry-exporter-otlp-proto-grpc   1.26.0
opentelemetry-exporter-otlp-proto-http   1.26.0
opentelemetry-proto                      1.26.0
opentelemetry-sdk                        1.26.0
opentelemetry-semantic-conventions       0.47b0
opentelemetry-semantic-conventions-ai    0.4.8
outlines                                 0.1.11
outlines-core                            0.1.26
packaging                                25.0
pandas                                   2.2.3
partial-json-parser                      0.2.1.1.post5
pillow                                   11.2.1
platformdirs                             4.3.8
prometheus-client                        0.21.1
prometheus-fastapi-instrumentator        7.1.0
propcache                                0.3.1
proto-plus                               1.26.1
protobuf                                 4.25.7
psutil                                   7.0.0
py-cpuinfo                               9.0.0
py-spy                                   0.4.0
pyarrow                                  17.0.0
pyasn1                                   0.6.1
pyasn1-modules                           0.4.2
pycountry                                24.6.1
pydantic                                 2.11.4
pydantic-core                            2.33.2
pygments                                 2.19.1
python-dateutil                          2.9.0.post0
python-dotenv                            1.1.0
python-json-logger                       3.3.0
python-multipart                         0.0.20
pytz                                     2025.2
pyyaml                                   6.0.2
pyzmq                                    26.4.0
ray                                      2.46.0
referencing                              0.36.2
regex                                    2024.11.6
requests                                 2.32.3
rich                                     14.0.0
rich-toolkit                             0.14.6
rpds-py                                  0.25.0
rsa                                      4.9.1
safetensors                              0.5.3
scipy                                    1.15.3
sentencepiece                            0.2.0
setuptools                               80.7.1
shellingham                              1.5.4
six                                      1.17.0
smart-open                               7.1.0
sniffio                                  1.3.1
starlette                                0.46.2
sympy                                    1.13.1
tiktoken                                 0.9.0
tokenizers                               0.21.1
torch                                    2.6.0
torchaudio                               2.6.0
torchvision                              0.21.0
tqdm                                     4.67.1
transformers                             4.51.3
typer                                    0.15.3
typing-extensions                        4.13.2
typing-inspection                        0.4.0
tzdata                                   2025.2
urllib3                                  2.4.0
uvicorn                                  0.34.2
uvloop                                   0.21.0
virtualenv                               20.31.2
vllm                                     0.8.5.post1
watchfiles                               1.0.5
websockets                               15.0.1
wrapt                                    1.17.2
yarl                                     1.20.0
zipp                                     3.21.0
```

### Reproduction script

```bash
mkdir bug
cd bug
uv init
uv add ray --active --extra data --extra serve --extra llm --extra default
echo 'from ray.serve.llm import LLMConfig, LLMRouter, LLMServer, ModelLoadingConfig' > main.py
uv run main.py
```
>> ModuleNotFoundError: No module named 'botocore'

### Issue Severity

High: It blocks me from completing my task.

Hints from issue discussion:

Commit message: CloudFileSystem.list_subfolders already swallows all Exceptions

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Patch to review:
diff --git a/python/ray/llm/_internal/serve/observability/usage_telemetry/usage.py b/python/ray/llm/_internal/serve/observability/usage_telemetry/usage.py
index 36ccc928848e3..ed9439bfc2afc 100644
--- a/python/ray/llm/_internal/serve/observability/usage_telemetry/usage.py
+++ b/python/ray/llm/_internal/serve/observability/usage_telemetry/usage.py
@@ -1,9 +1,6 @@
 from enum import Enum
 from typing import TYPE_CHECKING, Callable, Dict, List, Optional, Sequence
 
-# TODO (genesu): remove dependency on botocore
-from botocore.exceptions import ClientError
-
 import ray
 from ray import serve
 from ray._private.usage.usage_lib import (
@@ -254,19 +251,12 @@ def push_telemetry_report_for_all_models(
         )
         initial_num_lora_adapters = 0
         if use_lora:
-            # This try-except block is used to handle the case where the Lora model IDs
-            # cannot be fetched. In such cases, the telemetry report will be pushed with
-            # 0 initial Lora adapters.
-            try:
-                lora_model_ids = get_lora_model_func(
-                    dynamic_lora_loading_path=model.lora_config.dynamic_lora_loading_path,
-                    base_model_id=model.model_id,
-                )
-                initial_num_lora_adapters = len(lora_model_ids)
-            except ClientError as e:
-                logger.error(
-                    f"Failed to get Lora model IDs for model {model.model_id}: {e}"
-                )
+            lora_model_ids = get_lora_model_func(
+                dynamic_lora_loading_path=model.lora_config.dynamic_lora_loading_path,
+                base_model_id=model.model_id,
+            )
+            initial_num_lora_adapters = len(lora_model_ids)
+
         use_autoscaling = model.deployment_config.get("autoscaling_config") is not None
         num_replicas, min_replicas, max_replicas = 1, 1, 1
         if use_autoscaling:


Code review comments:
diff_hunk: @@ -361,6 +361,19 @@ def test_list_subfolders(self, mock_gcsfs):
         folders = CloudFileSystem.list_subfolders("gs://bucket/parent")
         assert sorted(folders) == ["dir1", "dir2"]
 
+    @patch("ray.llm._internal.common.utils.cloud_utils.CloudFileSystem.get_fs_and_path")
+    def test_list_subfolders_exception_handling(self, mock_get_fs_and_path):
+        """Test that list_subfolders returns empty list when get_fs_and_path raises exception."""
+        # Make get_fs_and_path raise an exception
+        mock_get_fs_and_path.side_effect = ValueError("Example exception")
+        
+        # Test that list_subfolders handles the exception gracefully
+        folders = CloudFileSystem.list_subfolders("gs://bucket/parent")
+        assert folders == []
+        
+        # Verify get_fs_and_path was called
+        mock_get_fs_and_path.assert_called_once_with("gs://bucket/parent/")
path: python/ray/llm/tests/common/utils/test_cloud_utils.py
text: @user1:
[nitpick] Asserting the exact trailing slash can be brittle. Consider normalizing the path or using a substring or `endswith` check to accept both with and without a trailing slash.
```suggestion
        mock_get_fs_and_path.assert_called_once()
        assert mock_get_fs_and_path.call_args[0][0].endswith("gs://bucket/parent/")
```

@author:
I don't expect `get_fs_and_path` to change much. This test is pretty specific/am not too concerned about it being brittle.