Repository: skypilot-org/skypilot
Pull request: [k8s] Fix recursive decoration of kubernetes API methods
Closes #3774. This was a regression in https://github.com/skypilot-org/skypilot/pull/3674 which decorated the kubernetes API methods multiple times, blowing up the stack depth and causing the recursion error.

This PR fixes it by decorating methods only once.

Tested (run the relevant ones):

- [x] Code formatting: `bash format.sh`
- [x] Manual repro by terminating SkyServe replica and watching controller logs
- [x] `pytest tests/test_smoke.py::test_skyserve_update_autoscale --kubernetes`


Problem statement:
[k8s][Serve] Replica prober failing with RecursionError on k8s
Repro: `pytest tests/test_smoke.py::test_skyserve_update_autoscale --cloud kubernetes` stays stuck.

When inspected with `sky serve logs t-ss-update-auto-7q-4975-e2 --controller`:

```
I 07-23 05:22:14 httptools_impl.py:466] 127.0.0.1:51340 - "POST /controller/load_balancer_sync HTTP/1.1" 500
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/serve/controller.py", line 102, in load_balancer_sync
    self._replica_manager.get_active_replica_urls()
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/serve/replica_managers.py", line 1159, in get_active_replica_urls
    assert info.url is not None, info
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/serve/replica_managers.py", line 434, in url
    endpoint_dict = core.endpoints(handle.cluster_name,
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/core.py", line 132, in endpoints
    return backend_utils.get_endpoints(cluster=cluster, port=port)
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/backends/backend_utils.py", line 2777, in get_endpoints
    cloud.check_features_are_supported(
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/clouds/cloud.py", line 543, in check_features_are_supported
    unsupported_features2reason = cls._unsupported_features_for_resources(
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/clouds/kubernetes.py", line 102, in _unsupported_features_for_resources
    spot_label_key, _ = kubernetes_utils.get_spot_label()
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/provision/kubernetes/utils.py", line 1622, in get_spot_label
    for node in get_kubernetes_nodes():
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/provision/kubernetes/utils.py", line 348, in get_kubernetes_nodes
    nodes = kubernetes.core_api().list_node(
  File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  [Previous line repeated 577 more times]
  File "/home/sky/.local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py", line 17005, in list_node
    return self.list_node_with_http_info(**kwargs)  # noqa: E501
  File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  [Previous line repeated 371 more times]
  File "/opt/conda/lib/python3.10/contextlib.py", line 78, in inner
    with self._recreate_cm():
  File "/opt/conda/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/sky_logging.py", line 100, in set_logging_level
    logger = logging.getLogger(logger)
  File "/opt/conda/lib/python3.10/logging/__init__.py", line 2079, in getLogger
    return Logger.manager.getLogger(name)
  File "/opt/conda/lib/python3.10/logging/__init__.py", line 1329, in getLogger
    _acquireLock()
  File "/opt/conda/lib/python3.10/logging/__init__.py", line 226, in _acquireLock
    _lock.acquire()
RecursionError: maximum recursion depth exceeded while calling a Python object
D 07-23 05:22:17 replica_managers.py:1128] Running replica prober.
E 07-23 05:22:17 replica_managers.py:1142] Error in replica prober: RecursionError: maximum recursion depth exceeded while calling a Python object
E 07-23 05:22:17 replica_managers.py:1145]   Traceback: Traceback (most recent call last):
E 07-23 05:22:17 replica_managers.py:1145]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/serve/replica_managers.py", line 1130, in _replica_prober
E 07-23 05:22:17 replica_managers.py:1145]     self._probe_all_replicas()
E 07-23 05:22:17 replica_managers.py:1145]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/serve/replica_managers.py", line 203, in wrapper
E 07-23 05:22:17 replica_managers.py:1145]     return func(self, *args, **kwargs)
E 07-23 05:22:17 replica_managers.py:1145]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/serve/replica_managers.py", line 1038, in _probe_all_replicas
E 07-23 05:22:17 replica_managers.py:1145]     f'replica_{info.replica_id}(url={info.url})')
E 07-23 05:22:17 replica_managers.py:1145]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/serve/replica_managers.py", line 434, in url
E 07-23 05:22:17 replica_managers.py:1145]     endpoint_dict = core.endpoints(handle.cluster_name,
E 07-23 05:22:17 replica_managers.py:1145]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/core.py", line 132, in endpoints
E 07-23 05:22:17 replica_managers.py:1145]     return backend_utils.get_endpoints(cluster=cluster, port=port)
E 07-23 05:22:17 replica_managers.py:1145]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/backends/backend_utils.py", line 2777, in get_endpoints
E 07-23 05:22:17 replica_managers.py:1145]     cloud.check_features_are_supported(
E 07-23 05:22:17 replica_managers.py:1145]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/clouds/cloud.py", line 543, in check_features_are_supported
E 07-23 05:22:17 replica_managers.py:1145]     unsupported_features2reason = cls._unsupported_features_for_resources(
E 07-23 05:22:17 replica_managers.py:1145]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/clouds/kubernetes.py", line 102, in _unsupported_features_for_resources
E 07-23 05:22:17 replica_managers.py:1145]     spot_label_key, _ = kubernetes_utils.get_spot_label()
E 07-23 05:22:17 replica_managers.py:1145]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/provision/kubernetes/utils.py", line 1622, in get_spot_label
E 07-23 05:22:17 replica_managers.py:1145]     for node in get_kubernetes_nodes():
E 07-23 05:22:17 replica_managers.py:1145]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/provision/kubernetes/utils.py", line 348, in get_kubernetes_nodes
E 07-23 05:22:17 replica_managers.py:1145]     nodes = kubernetes.core_api().list_node(
E 07-23 05:22:17 replica_managers.py:1145]   File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
E 07-23 05:22:17 replica_managers.py:1145]     return func(*args, **kwds)
E 07-23 05:22:17 replica_managers.py:1145]   File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
E 07-23 05:22:17 replica_managers.py:1145]     return func(*args, **kwds)
E 07-23 05:22:17 replica_managers.py:1145]   File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
E 07-23 05:22:17 replica_managers.py:1145]     return func(*args, **kwds)
E 07-23 05:22:17 replica_managers.py:1145]   [Previous line repeated 578 more times]
E 07-23 05:22:17 replica_managers.py:1145]   File "/home/sky/.local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py", line 17005, in list_node
E 07-23 05:22:17 replica_managers.py:1145]     return self.list_node_with_http_info(**kwargs)  # noqa: E501
E 07-23 05:22:17 replica_managers.py:1145]   File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
E 07-23 05:22:17 replica_managers.py:1145]     return func(*args, **kwds)
E 07-23 05:22:17 replica_managers.py:1145]   File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
E 07-23 05:22:17 replica_managers.py:1145]     return func(*args, **kwds)
E 07-23 05:22:17 replica_managers.py:1145]   File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
E 07-23 05:22:17 replica_managers.py:1145]     return func(*args, **kwds)
E 07-23 05:22:17 replica_managers.py:1145]   [Previous line repeated 395 more times]
E 07-23 05:22:17 replica_managers.py:1145]   File "/opt/conda/lib/python3.10/contextlib.py", line 78, in inner
E 07-23 05:22:17 replica_managers.py:1145]     with self._recreate_cm():
E 07-23 05:22:17 replica_managers.py:1145]   File "/opt/conda/lib/python3.10/contextlib.py", line 135, in __enter__
E 07-23 05:22:17 replica_managers.py:1145]     return next(self.gen)
E 07-23 05:22:17 replica_managers.py:1145]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/sky_logging.py", line 100, in set_logging_level
E 07-23 05:22:17 replica_managers.py:1145]     logger = logging.getLogger(logger)
E 07-23 05:22:17 replica_managers.py:1145]   File "/opt/conda/lib/python3.10/logging/__init__.py", line 2079, in getLogger
E 07-23 05:22:17 replica_managers.py:1145]     return Logger.manager.getLogger(name)
E 07-23 05:22:17 replica_managers.py:1145]   File "/opt/conda/lib/python3.10/logging/__init__.py", line 1329, in getLogger
E 07-23 05:22:17 replica_managers.py:1145]     _acquireLock()
E 07-23 05:22:17 replica_managers.py:1145]   File "/opt/conda/lib/python3.10/logging/__init__.py", line 226, in _acquireLock
E 07-23 05:22:17 replica_managers.py:1145]     _lock.acquire()
E 07-23 05:22:17 replica_managers.py:1145] RecursionError: maximum recursion depth exceeded while calling a Python object
E 07-23 05:22:17 replica_managers.py:1145]
D 07-23 05:22:25 replica_managers.py:945] Refreshing process pool.
E 07-23 05:22:26 controller.py:83] Error in autoscaler: RecursionError: maximum recursion depth exceeded while calling a Python object
E 07-23 05:22:26 controller.py:86]   Traceback: Traceback (most recent call last):
E 07-23 05:22:26 controller.py:86]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/serve/controller.py", line 61, in _run_autoscaler
E 07-23 05:22:26 controller.py:86]     logger.info(f'All replica info: {replica_infos}')
E 07-23 05:22:26 controller.py:86]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/serve/replica_managers.py", line 473, in __repr__
E 07-23 05:22:26 controller.py:86]     info_dict = self.to_info_dict(
E 07-23 05:22:26 controller.py:86]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/serve/replica_managers.py", line 463, in to_info_dict
E 07-23 05:22:26 controller.py:86]     'endpoint': self.url,
E 07-23 05:22:26 controller.py:86]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/serve/replica_managers.py", line 434, in url
E 07-23 05:22:26 controller.py:86]     endpoint_dict = core.endpoints(handle.cluster_name,
E 07-23 05:22:26 controller.py:86]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/core.py", line 132, in endpoints
E 07-23 05:22:26 controller.py:86]     return backend_utils.get_endpoints(cluster=cluster, port=port)
E 07-23 05:22:26 controller.py:86]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/backends/backend_utils.py", line 2777, in get_endpoints
E 07-23 05:22:26 controller.py:86]     cloud.check_features_are_supported(
E 07-23 05:22:26 controller.py:86]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/clouds/cloud.py", line 543, in check_features_are_supported
E 07-23 05:22:26 controller.py:86]     unsupported_features2reason = cls._unsupported_features_for_resources(
E 07-23 05:22:26 controller.py:86]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/clouds/kubernetes.py", line 102, in _unsupported_features_for_resources
E 07-23 05:22:26 controller.py:86]     spot_label_key, _ = kubernetes_utils.get_spot_label()
E 07-23 05:22:26 controller.py:86]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/provision/kubernetes/utils.py", line 1622, in get_spot_label
E 07-23 05:22:26 controller.py:86]     for node in get_kubernetes_nodes():
E 07-23 05:22:26 controller.py:86]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/provision/kubernetes/utils.py", line 348, in get_kubernetes_nodes
E 07-23 05:22:26 controller.py:86]     nodes = kubernetes.core_api().list_node(
E 07-23 05:22:26 controller.py:86]   File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
E 07-23 05:22:26 controller.py:86]     return func(*args, **kwds)
E 07-23 05:22:26 controller.py:86]   File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
E 07-23 05:22:26 controller.py:86]     return func(*args, **kwds)
E 07-23 05:22:26 controller.py:86]   File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
E 07-23 05:22:26 controller.py:86]     return func(*args, **kwds)
E 07-23 05:22:26 controller.py:86]   [Previous line repeated 579 more times]
E 07-23 05:22:26 controller.py:86]   File "/home/sky/.local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py", line 17005, in list_node
E 07-23 05:22:26 controller.py:86]     return self.list_node_with_http_info(**kwargs)  # noqa: E501
E 07-23 05:22:26 controller.py:86]   File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
E 07-23 05:22:26 controller.py:86]     return func(*args, **kwds)
E 07-23 05:22:26 controller.py:86]   File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
E 07-23 05:22:26 controller.py:86]     return func(*args, **kwds)
E 07-23 05:22:26 controller.py:86]   File "/opt/conda/lib/python3.10/contextlib.py", line 79, in inner
E 07-23 05:22:26 controller.py:86]     return func(*args, **kwds)
E 07-23 05:22:26 controller.py:86]   [Previous line repeated 391 more times]
E 07-23 05:22:26 controller.py:86]   File "/opt/conda/lib/python3.10/contextlib.py", line 78, in inner
E 07-23 05:22:26 controller.py:86]     with self._recreate_cm():
E 07-23 05:22:26 controller.py:86]   File "/opt/conda/lib/python3.10/contextlib.py", line 135, in __enter__
E 07-23 05:22:26 controller.py:86]     return next(self.gen)
E 07-23 05:22:26 controller.py:86]   File "/home/sky/skypilot-runtime/lib/python3.10/site-packages/sky/sky_logging.py", line 100, in set_logging_level
E 07-23 05:22:26 controller.py:86]     logger = logging.getLogger(logger)
E 07-23 05:22:26 controller.py:86]   File "/opt/conda/lib/python3.10/logging/__init__.py", line 2079, in getLogger
E 07-23 05:22:26 controller.py:86]     return Logger.manager.getLogger(name)
E 07-23 05:22:26 controller.py:86]   File "/opt/conda/lib/python3.10/logging/__init__.py", line 1329, in getLogger
E 07-23 05:22:26 controller.py:86]     _acquireLock()
E 07-23 05:22:26 controller.py:86]   File "/opt/conda/lib/python3.10/logging/__init__.py", line 226, in _acquireLock
E 07-23 05:22:26 controller.py:86]     _lock.acquire()
E 07-23 05:22:26 controller.py:86] RecursionError: maximum recursion depth exceeded while calling a Python object
E 07-23 05:22:26 controller.py:86]
D 07-23 05:22:27 replica_managers.py:1008] Refreshing job status.
D 07-23 05:22:27 replica_managers.py:1128] Running replica prober.
```

Hints from issue discussion:
Oh look at that, I'm seeing this too
FYI: I switched down to commit aea732214ed7997b8c15f93f9b764f9175a6d5db and this problem isn't triggering
Nevermind, it does start triggering at some point.
Possibly related to https://github.com/skypilot-org/skypilot/pull/3674. Reverting that change fixes this. Looks like adding the logging decorator to _all_ methods in the kubernetes python API blows up the stack depth, causing this recursion errror.

Commit message: lint

Patch to review:
diff --git a/sky/adaptors/kubernetes.py b/sky/adaptors/kubernetes.py
index 7f52a099f56..e75149836ae 100644
--- a/sky/adaptors/kubernetes.py
+++ b/sky/adaptors/kubernetes.py
@@ -33,8 +33,11 @@
 def _decorate_methods(obj, decorator):
     for attr_name in dir(obj):
         attr = getattr(obj, attr_name)
-        if callable(attr) and not attr_name.startswith('__'):
-            setattr(obj, attr_name, decorator(attr))
+        if callable(attr) and not attr_name.startswith('__') and not getattr(
+                attr, '_is_sky_decorated', False):
+            decorated_attr = decorator(attr)
+            decorated_attr._is_sky_decorated = True  # pylint: disable=protected-access
+            setattr(obj, attr_name, decorated_attr)
     return obj
 
 


Code review comments:
diff_hunk: @@ -33,8 +33,11 @@
 def _decorate_methods(obj, decorator):
     for attr_name in dir(obj):
         attr = getattr(obj, attr_name)
-        if callable(attr) and not attr_name.startswith('__'):
-            setattr(obj, attr_name, decorator(attr))
+        if callable(attr) and not attr_name.startswith('__') and not getattr(
+                attr, '_is_sky_decorated', False):
+            decorated_attr = decorator(attr)
+            decorated_attr._is_sky_decorated = True  # pylint: disable=protected-access
path: sky/adaptors/kubernetes.py
text: @user1:
It is a bit weird to skip the decoration if it is ever decorated with this method, and `_is_sky_decorated` feels a bit too general. Is it possible to make it `_sky_decorator_types: Set[str]` which stores the purpose for the decoration eg `api_log`.

Also, can we add some comments for why we use `startswith('__')` in the condition?