Repository: skypilot-org/skypilot
Pull request: [k8s] suppress connection error warnings when disconnected from k8s

<!-- Describe the changes in this PR -->

suppresses `urllib3` warnings for `sky show-gpus` if k8s cluster gets disconnected. Closes #3591

before changes
```
(base) gcpuser@k3s-ebd1-head-82q3gnw3-compute:~/skypilot$ sky show-gpus --cloud kubernetes
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f75a56eeb60>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/default/pods
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f75a56ef3a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/default/pods
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f75a56ef6a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/default/pods
No GPUs found in Kubernetes cluster. If your cluster contains GPUs, make sure nvidia.com/gpu resource is available on the nodes and the node labels for identifying GPUs (e.g., skypilot.co/accelerator) are setup correctly. To further debug, run: sky check
```

after changes 
```
(base) gcpuser@k3s-ebd1-head-82q3gnw3-compute:~/skypilot$ sky show-gpus --cloud kubernetes
No GPUs found in Kubernetes cluster. If your cluster contains GPUs, make sure nvidia.com/gpu resource is available on the nodes and the node labels for identifying GPUs (e.g., skypilot.co/accelerator) are setup correctly. To further debug, run: sky check 
```

<!-- Describe the tests ran -->
<!-- Unit tests (tests/test_*.py) are part of GitHub CI; below are tests that launch on the cloud. -->

Tested (run the relevant ones):

- [x] Code formatting: `bash format.sh`
- [x] Any manual or new tests for this PR (please specify below)

To create a disconnected cluster I (un)installed via k3s on a GCE VM
```
sky launch -c k3s --cloud gcp
ssh k3s
curl -sfL https://get.k3s.io | sh -
mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown -R $USER ~/.kube
sky check
sky show-gpus --cloud kubernetes
bash /usr/local/bin/k3s-uninstall.sh
sky show-gpus --cloud kubernetes
```


Problem statement:
[k8s] Suppress warnings when kubernetes cluster is not reachable
We should consider suppressing warnings when the Kubernetes cluster is no longer reachable (e.g., after GKE cluster has been deleted or docker runtime is not running for `sky local up`). Currently, we get warnings like:
```
sky show-gpus --cloud kubernetes
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x1075c3af0>, 'Connection to 146.148.51.187 timed out. (connect timeout=5)')': /api/v1/namespaces/default/pods
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x1075c3c10>, 'Connection to 146.148.51.187 timed out. (connect timeout=5)')': /api/v1/namespaces/default/pods
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x1075c3730>, 'Connection to 146.148.51.187 timed out. (connect timeout=5)')': /api/v1/namespaces/default/pods
No GPUs found in Kubernetes cluster. If your cluster contains GPUs, make sure nvidia.com/gpu resource is available on the nodes and the node labels for identifying GPUs (e.g., skypilot.co/accelerator) are setup correctly. To further debug, run: sky check.
```

Hints from issue discussion:

Commit message: suppress connection error warnings when disconnected from k8s

format

set urllib3 log level

set to ERROR level

Patch to review:
diff --git a/sky/provision/kubernetes/utils.py b/sky/provision/kubernetes/utils.py
index 4301176204b..2bc82cde7e2 100644
--- a/sky/provision/kubernetes/utils.py
+++ b/sky/provision/kubernetes/utils.py
@@ -1,5 +1,6 @@
 """Kubernetes utilities for SkyPilot."""
 import json
+import logging
 import math
 import os
 import re
@@ -16,6 +17,7 @@
 from sky import skypilot_config
 from sky.adaptors import kubernetes
 from sky.provision.kubernetes import network_utils
+from sky.provision.logging import set_logging_level
 from sky.utils import common_utils
 from sky.utils import env_options
 from sky.utils import kubernetes_enums
@@ -601,9 +603,12 @@ def check_credentials(timeout: int = kubernetes.API_TIMEOUT) -> \
         bool: True if credentials are valid, False otherwise
         str: Error message if credentials are invalid, None otherwise
     """
+
     try:
         ns = get_current_kube_config_context_namespace()
-        kubernetes.core_api().list_namespaced_pod(ns, _request_timeout=timeout)
+        with set_logging_level('urllib3', logging.ERROR):
+            kubernetes.core_api().list_namespaced_pod(ns,
+                                                      _request_timeout=timeout)
     except ImportError:
         # TODO(romilb): Update these error strs to also include link to docs
         #  when docs are ready.
diff --git a/sky/provision/logging.py b/sky/provision/logging.py
index 2ea3967d15c..aa246e4cf36 100644
--- a/sky/provision/logging.py
+++ b/sky/provision/logging.py
@@ -58,5 +58,15 @@ def setup_provision_logging(log_dir: str):
         fh.close()
 
 
+@contextlib.contextmanager
+def set_logging_level(logger, level):
+    original_level = logger.level
+    logger.setLevel(level)
+    try:
+        yield
+    finally:
+        logger.setLevel(original_level)
+
+
 def get_log_path() -> pathlib.Path:
     return config.log_path


Code review comments:
diff_hunk: @@ -16,6 +17,7 @@
 from sky import skypilot_config
 from sky.adaptors import kubernetes
 from sky.provision.kubernetes import network_utils
+from sky.provision.logging import set_logging_level
path: sky/provision/kubernetes/utils.py
text: @user1:
According to the style guide, we would like to import modules instead of functions directly: https://google.github.io/styleguide/pyguide.html#22-imports
```suggestion
from sky.provision import logging
```