Repository: skypilot-org/skypilot
Pull request: [API server] Set default min resources and warning on low resources
<!-- Describe the changes in this PR -->

close https://github.com/skypilot-org/skypilot/issues/5093

<!-- Describe the tests ran -->
<!-- Unit tests (tests/test_*.py) are part of GitHub CI; below are tests that launch on the cloud. -->

Tested (run the relevant ones):

- [x] Code formatting: install pre-commit (auto-check on commit) or `bash format.sh`
- [x] Any manual or new tests for this PR (please specify below)
- [x] Normal case, without specifying `.apiService.resources.requests`, succeed with no warning
- [x]  Setting cpu=1, memory=3GiB, installation failed
```bash
helm upgrade --install skypilot-test ./charts/skypilot -f values.yaml --set ingress.nodePortEnabled=true

Release "skypilot-test" does not exist. Installing it now.
Error: execution error at (skypilot/templates/NOTES.txt:2:4): Error
API server requires at least 4 CPU cores and 8 GiB memory. You can either:
1. increase .apiService.resources.requests.* to meet the requirements or unset them to use defaults
2. add `--set apiService.skipResourceCheck=true` in command args to bypass this check (not recommended for production)
```
- [x] Follow the hint and specify `--set apiService.skipResourceCheck=true`, succeed
```
helm upgrade --install skypilot-test ./charts/skypilot -f values.yaml --set ingress.nodePortEnabled=true --set apiService.skipResourceCheck=true
Release "skypilot-test" does not exist. Installing it now.
NAME: skypilot-test
LAST DEPLOYED: Thu Apr  3 12:47:41 2025
NAMESPACE: skypilot
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
```
- [ ] All smoke tests: `/smoke-test` (CI) or `pytest tests/test_smoke.py` (local)
- [ ] Relevant individual tests: `/smoke-test -k test_name` (CI) or `pytest tests/test_smoke.py::test_name` (local)
- [ ] Backward compatibility: `/quicktest-core` (CI) or `pytest tests/smoke_tests/test_backward_compat.py` (local)

<!-- CI commands (/-prefixed) can only be triggered by repo members -->


Problem statement:
[API server] Set default resources requests in helm deployment
One of our user use kata-container as container runtime, which set a default tiny VM size if the Pod has no resources specified. We should set a default resources request in our chart.

Hints from issue discussion:

Commit message: Update charts/skypilot/templates/_helpers.tpl

Co-authored-by: Zhanghao Wu <zhanghao.wu@outlook.com>

Patch to review:
diff --git a/charts/skypilot/templates/NOTES.txt b/charts/skypilot/templates/NOTES.txt
new file mode 100644
index 00000000000..85105a04239
--- /dev/null
+++ b/charts/skypilot/templates/NOTES.txt
@@ -0,0 +1,3 @@
+{{- if not .Values.apiService.skipResourceCheck }}
+{{- include "skypilot.checkResources" . }}
+{{- end }}
diff --git a/charts/skypilot/templates/_helpers.tpl b/charts/skypilot/templates/_helpers.tpl
new file mode 100644
index 00000000000..8dfd55655c5
--- /dev/null
+++ b/charts/skypilot/templates/_helpers.tpl
@@ -0,0 +1,34 @@
+{{- define "skypilot.checkResources" -}}
+{{- $cpu := .Values.apiService.resources.requests.cpu | default "0" -}}
+{{- $memory := .Values.apiService.resources.requests.memory | default "0" -}}
+
+{{- /* Convert CPU to numeric value */ -}}
+{{- $cpuNum := 0.0 -}}
+{{- if kindIs "string" $cpu -}}
+  {{- if hasSuffix "m" $cpu -}}
+    {{- $cpuNum = float64 (divf (trimSuffix "m" $cpu | atoi) 1000) -}}
+  {{- else -}}
+    {{- $cpuNum = float64 ($cpu | atoi) -}}
+  {{- end -}}
+{{- else -}}
+  {{- $cpuNum = float64 $cpu -}}
+{{- end -}}
+
+{{- /* Convert memory to Gi */ -}}
+{{- $memNum := 0.0 -}}
+{{- if hasSuffix "Gi" $memory -}}
+  {{- $memNum = float64 (trimSuffix "Gi" $memory | atoi) -}}
+{{- else if hasSuffix "Mi" $memory -}}
+  {{- $memNum = float64 (divf (trimSuffix "Mi" $memory | atoi) 1024) -}}
+{{- else if hasSuffix "G" $memory -}}
+  {{- $memNum = float64 ($memory | trimSuffix "G" | atoi) -}}
+{{- else if hasSuffix "M" $memory -}}
+  {{- $memNum = float64 (divf (trimSuffix "M" $memory | atoi) 1024) -}}
+{{- end -}}
+
+{{- if or (lt $cpuNum 4.0) (lt $memNum 8.0) -}}
+{{/* TODO(aylei): add a reference to the tuning guide once complete */}}
+  {{- fail "Error\nDeploying a SkyPilot API server requires at least 4 CPU cores and 8 GiB memory. You can either:\n1. Change `--set apiService.resources.requests.cpu` and `--set apiService.resources.requests.memory` to meet the requirements or unset them to use defaults\n2. add `--set apiService.skipResourceCheck=true` in command args to bypass this check (not recommended for production)\nto resolve this issue and then try again." -}}
+{{- end -}}
+
+{{- end -}} 
\ No newline at end of file
diff --git a/charts/skypilot/values.yaml b/charts/skypilot/values.yaml
index 57844f5ce9f..35a8054bc9d 100644
--- a/charts/skypilot/values.yaml
+++ b/charts/skypilot/values.yaml
@@ -33,18 +33,21 @@ apiService:
   #     - aws
   #     - kubernetes
 
+  # Skip resource check for the API server, not recommended for production deployment
+  skipResourceCheck: false
   # Set resource requests and limits for the API server
+  resources:
+    # Request a moderate amount of resources for an remote API server by default, which meets the
+    # basic requirements for team usage at medium scale.
+    requests:
+      cpu: "4"
+      memory: "8Gi"
+      # Additional resource types like ephemeral-storage can be specified here
   # TODO: Setting limits causes issues with the API server. For now we advise
   # not to set limits.
-  resources: null
-  # resources:
-  #   requests:
-  #     cpu: "1"
-  #     memory: "2Gi"
-  #   limits:
-  #     cpu: "2"
-  #     memory: "4Gi"
-  #     # Additional resource types like ephemeral-storage can be specified here
+  # limits:
+  #  cpu: "4"
+  #  memory: "8Gi"
   
   # [Internal] Enable developer mode for SkyPilot
   skypilotDev: false
diff --git a/docs/source/reference/api-server/api-server-admin-deploy.rst b/docs/source/reference/api-server/api-server-admin-deploy.rst
index 272090fada1..4675a4faea7 100644
--- a/docs/source/reference/api-server/api-server-admin-deploy.rst
+++ b/docs/source/reference/api-server/api-server-admin-deploy.rst
@@ -197,6 +197,14 @@ If you configured any cloud credentials in the previous step, make sure to enabl
 
     The secret must be in the same namespace as the API server and must contain a key named ``auth`` with the basic auth credentials in htpasswd format.
 
+After the API server is deployed, you can inspect the API server pod status with:
+
+.. code-block:: bash
+
+    kubectl get pods -n $NAMESPACE -l app=${RELEASE_NAME}-api --watch
+
+You should see the pod is initializing and finally becomes running and ready. If not, refer to :ref:`sky-api-server-troubleshooting-helm` to diagnose the issue.
+
 .. _sky-get-api-server-url:
 
 Step 4: Get the API server URL
diff --git a/docs/source/reference/api-server/api-server-troubleshooting.rst b/docs/source/reference/api-server/api-server-troubleshooting.rst
new file mode 100644
index 00000000000..3848a11d124
--- /dev/null
+++ b/docs/source/reference/api-server/api-server-troubleshooting.rst
@@ -0,0 +1,42 @@
+.. _sky-api-server-troubleshooting:
+
+Troubleshooting SkyPilot API Server
+===================================
+
+If you're unable to run SkyPilot API server, this guide will help you debug common issues.
+
+If this guide does not help resolve your issue, please reach out to us on `Slack <https://slack.skypilot.co>`_ or `GitHub <http://www.github.com/skypilot-org/skypilot>`_.
+
+.. _sky-api-server-troubleshooting-helm:
+
+Helm deployment troubleshooting
+-------------------------------
+
+.. _sky-api-server-troubleshooting-pod-pending:
+
+API server pod is pending
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If the API server pod is pending, you can inspect the pending reason with:
+
+.. code-block:: bash
+
+    kubectl describe pod -n $NAMESPACE -l app=${RELEASE_NAME}-api
+
+If the pending reason is ``FailedScheduling`` and the information indicates there is insufficient cpu/memory, you can either:
+
+- Adding more resources to the Kubernetes cluster, or
+- Using a smaller API server resources request, for example (change the cpu and memory to your desired values):
+
+.. code-block:: bash
+
+    # Update the resources requests while keeping existing values set in the previous commands
+    helm upgrade --install $RELEASE_NAME skypilot/skypilot-nightly \
+    --namespace $NAMESPACE \
+    --reuse-values \
+    --set apiService.resources.requests.cpu=4 \
+    --set apiService.resources.requests.memory=8Gi
+
+.. note::
+
+    API server requires at least 4 CPU cores and 8 GiB memory, setting lower values may cause degraded performance.
diff --git a/docs/source/reference/api-server/api-server.rst b/docs/source/reference/api-server/api-server.rst
index df974d5754a..4ef601ea9dd 100644
--- a/docs/source/reference/api-server/api-server.rst
+++ b/docs/source/reference/api-server/api-server.rst
@@ -112,3 +112,4 @@ To verify that the API server is working, run ``sky api info``:
    :hidden:
 
    Deploying API Server <api-server-admin-deploy>
+   Troubleshooting <api-server-troubleshooting>


Code review comments:
diff_hunk: @@ -0,0 +1,42 @@
+.. _sky-api-server-troubleshooting:
+
+Troubleshooting SkyPilot API Server
+===================================
+
+If you're unable to run SkyPilot API server, this guide will help you debug common issues.
+
+If this guide does not help resolve your issue, please reach out to us on `Slack <https://slack.skypilot.co>`_ or `GitHub <http://www.github.com/skypilot-org/skypilot>`_.
+
+.. _sky-api-server-troubleshooting-helm:
+
+Helm deployment troubleshooting
+-------------------------------
+
+.. _sky-api-server-troubleshooting-pod-pending:
+
+API server pod is pending
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If the API server pod is pending, you can inspect the pending reason with:
+
+.. code-block:: bash
+
+    kubectl describe pod -n $NAMESPACE -l app=${RELEASE_NAME}-api
+
+If the pending reason is ``FailedScheduling`` and the information indicates there is insufficient cpu/memory, you can either:
+
+- Adding more resources to the Kubernetes cluster, or
+- Using a smaller API server resources request, for example (change the cpu and memory to your desired values):
path: docs/source/reference/api-server/api-server-troubleshooting.rst
text: @user1:
```suggestion
- Add more resources to the Kubernetes cluster, or
- Use a smaller API server resources request; for example (change the cpu and memory to your desired values):
```